You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Labslogbot
(switched restbase1009 to node 4.2 for testing, and restarted restbase; see https://phabricator.wikimedia.org/T107762 (gwicke))
imported>Stashbot
(_joe_: temporarily raised replicas for shellbox to 24)
 
Line 1: Line 1:
== 2016-01-06 ==
== 2022-07-03 ==
* 23:03 gwicke: switched restbase1009 to node 4.2 for testing, and restarted restbase; see https://phabricator.wikimedia.org/T107762
* 11:36 _joe_: temporarily raised replicas for shellbox to 24
* 02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan  6 02:34:38 UTC 2016 (duration 6m 53s)
* 11:35 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 30s)
* 11:35 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/shellbox: apply


== 2016-01-05 ==
== 2022-07-02 ==
* 22:38 logmsgbot: aaron@tin Synchronized rpc: 830e1ed8d80295710dc02f18102b4fadae7fca86 (duration: 00m 55s)
* 05:36 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 18:34 logmsgbot: jzerebecki@tin scap aborted: deploy-log (duration: 00m 04s)
* 05:36 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 18:34 logmsgbot: jzerebecki@tin Started scap: deploy-log
* 05:24 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 15:47 ottomata: transitioned analytics1001 to active namenode
* 05:23 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 03:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/includes/specials/SpecialJavaScriptTest.php: Idaacf71870 (duration: 00m 30s)
* 05:21 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 03:50 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.special/: Idaacf71870 (duration: 00m 30s)
* 05:20 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 03:49 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/resources/Resources.php: Idaacf71870 (duration: 00m 36s)
* 05:11 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan  5 02:31:46 UTC 2016 (duration 6m 54s)
* 05:11 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 13s)
* 04:49 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 04:49 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 04:48 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 04:48 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 03:59 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 03:59 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 03:57 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 03:57 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 03:56 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 03:56 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 02:49 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 02:49 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 01:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 01:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 00:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 00:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance


== 2016-01-04 ==
== 2022-07-01 ==
* 20:50 mutante: ms-be1011 - powercycled, was frozen
* 23:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
* 20:43 mutante: ms-be2007 - System halted!Error: Integrated RAID
* 23:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
* 20:42 mutante: ms-be2007 - powercycle (was status: on but all frozen) (i assume xfs like be2006 appears in SAL recently)
* 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30753 and previous config saved to /var/cache/conftool/dbconfig/20220701-235524-ladsgroup.json
* 20:36 mutante: mw2019 - puppet run (icinga claimed it failed but just here)
* 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P30752 and previous config saved to /var/cache/conftool/dbconfig/20220701-234019-ladsgroup.json
* 20:19 mutante: rutherfordium - attempt to restart with gnt-instance
* 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P30751 and previous config saved to /var/cache/conftool/dbconfig/20220701-232514-ladsgroup.json
* 20:12 mutante: rutherfordium (people.wm) was down for days per icinga - then magically fixes itself when i connect to console but before even loggin in (ganeti VM)
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30750 and previous config saved to /var/cache/conftool/dbconfig/20220701-231009-ladsgroup.json
* 20:00 mutante: mw1123 - start HHVM (was 503 and service stopped)
* 23:02 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1012.eqiad.wmnet with OS bullseye
* 19:28 mutante: elastic1006 - out of disk - gzip eqiad_index_search_slowlog.log files
* 22:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1012.eqiad.wmnet with reason: host reimage
* 17:37 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext - gerrit 262357 (duration: 00m 33s)
* 22:43 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1012.eqiad.wmnet with reason: host reimage
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jan  4 02:32:10 UTC 2016 (duration 6m 53s)
* 22:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1015.eqiad.wmnet with OS bullseye
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 05s)
* 22:31 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1012.eqiad.wmnet with OS bullseye
* 22:22 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1012.eqiad.wmnet with OS bullseye
* 22:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1015.eqiad.wmnet with reason: host reimage
* 22:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30749 and previous config saved to /var/cache/conftool/dbconfig/20220701-221438-ladsgroup.json
* 22:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1118.eqiad.wmnet with reason: Maintenance
* 22:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1015.eqiad.wmnet with reason: host reimage
* 22:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1118.eqiad.wmnet with reason: Maintenance
* 22:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30748 and previous config saved to /var/cache/conftool/dbconfig/20220701-221418-ladsgroup.json
* 22:12 mutante: restbase2018 - attempting power cycle via mgmt - /admin1-> racadm serveraction powercycle  ([[phab:T311890|T311890]])
* 22:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1014.eqiad.wmnet with OS bullseye
* 22:05 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1013.eqiad.wmnet with OS bullseye
* 22:05 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1008.eqiad.wmnet with OS bullseye
* 22:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1010.eqiad.wmnet with OS bullseye
* 22:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1012.eqiad.wmnet with OS bullseye
* 22:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1015.eqiad.wmnet with OS bullseye
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P30747 and previous config saved to /var/cache/conftool/dbconfig/20220701-215913-ladsgroup.json
* 21:57 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1009.eqiad.wmnet with OS bullseye
* 21:57 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1011.eqiad.wmnet with OS bullseye
* 21:57 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1007.eqiad.wmnet with OS bullseye
* 21:52 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1015.eqiad.wmnet with OS bullseye
* 21:51 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1012.eqiad.wmnet with OS bullseye
* 21:51 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage
* 21:51 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1014.eqiad.wmnet with reason: host reimage
* 21:51 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1010.eqiad.wmnet with reason: host reimage
* 21:50 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1008.eqiad.wmnet with reason: host reimage
* 21:50 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1013.eqiad.wmnet with reason: host reimage
* 21:50 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage
* 21:50 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on an-presto1009.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1009.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1008.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1013.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1010.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage
* 21:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1014.eqiad.wmnet with reason: host reimage
* 21:48 mutante: https://doc.wikimedia.org switched to doc1002 backend on buster [[phab:T247653|T247653]]
* 21:48 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1009.eqiad.wmnet with OS bullseye
* 21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P30746 and previous config saved to /var/cache/conftool/dbconfig/20220701-214408-ladsgroup.json
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1015.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1011.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1013.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1007.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bullseye
* 21:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1012.eqiad.wmnet with OS bullseye
* 21:36 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1014.eqiad.wmnet with OS bullseye
* 21:34 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1006.eqiad.wmnet with OS bullseye
* 21:33 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage
* 21:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage
* 21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30745 and previous config saved to /var/cache/conftool/dbconfig/20220701-212903-ladsgroup.json
* 21:20 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage
* 21:18 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host stat1009.eqiad.wmnet with OS bullseye
* 21:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage
* 21:09 mutante: https://doc.wikimedia.org - scheduled maintenance period - switching to buster backend doc1002 ([[phab:T247653|T247653]])
* 21:04 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-presto1006.eqiad.wmnet with OS bullseye
* 20:33 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30744 and previous config saved to /var/cache/conftool/dbconfig/20220701-203251-ladsgroup.json
* 20:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 20:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30743 and previous config saved to /var/cache/conftool/dbconfig/20220701-203231-ladsgroup.json
* 20:29 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 20:22 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:19 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P30742 and previous config saved to /var/cache/conftool/dbconfig/20220701-201726-ladsgroup.json
* 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P30741 and previous config saved to /var/cache/conftool/dbconfig/20220701-200221-ladsgroup.json
* 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30740 and previous config saved to /var/cache/conftool/dbconfig/20220701-194716-ladsgroup.json
* 18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30739 and previous config saved to /var/cache/conftool/dbconfig/20220701-183504-ladsgroup.json
* 18:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30738 and previous config saved to /var/cache/conftool/dbconfig/20220701-183444-ladsgroup.json
* 18:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P30737 and previous config saved to /var/cache/conftool/dbconfig/20220701-181939-ladsgroup.json
* 18:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P30736 and previous config saved to /var/cache/conftool/dbconfig/20220701-180434-ladsgroup.json
* 17:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30735 and previous config saved to /var/cache/conftool/dbconfig/20220701-174929-ladsgroup.json
* 17:47 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 17:47 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30734 and previous config saved to /var/cache/conftool/dbconfig/20220701-165407-ladsgroup.json
* 16:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30733 and previous config saved to /var/cache/conftool/dbconfig/20220701-165347-ladsgroup.json
* 16:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P30732 and previous config saved to /var/cache/conftool/dbconfig/20220701-163842-ladsgroup.json
* 16:30 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2168.codfw.wmnet with OS bullseye
* 16:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P30731 and previous config saved to /var/cache/conftool/dbconfig/20220701-162337-ladsgroup.json
* 16:16 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 16:13 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 16:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30730 and previous config saved to /var/cache/conftool/dbconfig/20220701-160831-ladsgroup.json
* 15:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2168.codfw.wmnet with OS bullseye
* 15:22 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS bullseye
* 15:16 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS bullseye
* 15:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage
* 15:04 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage
* 15:02 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage
* 15:02 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 15:02 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 15:01 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudstore[1008-1009]
* 14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30729 and previous config saved to /var/cache/conftool/dbconfig/20220701-145937-ladsgroup.json
* 14:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 14:59 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage
* 14:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 14:55 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:48 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 14:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS bullseye
* 14:40 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS bullseye
* 14:39 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudstore[1008-1009]
* 14:05 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 14:04 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30728 and previous config saved to /var/cache/conftool/dbconfig/20220701-135831-ladsgroup.json
* 13:50 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 13:50 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:43 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 13:43 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 07s)
* 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P30727 and previous config saved to /var/cache/conftool/dbconfig/20220701-134326-ladsgroup.json
* 13:43 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:36 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 13:36 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P30726 and previous config saved to /var/cache/conftool/dbconfig/20220701-132821-ladsgroup.json
* 13:23 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 13:23 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:19 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 13:19 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30725 and previous config saved to /var/cache/conftool/dbconfig/20220701-131316-ladsgroup.json
* 13:12 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 13:12 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:08 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 13:08 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Add db2155 to s4 [[phab:T311493|T311493]]', diff saved to https://phabricator.wikimedia.org/P30724 and previous config saved to /var/cache/conftool/dbconfig/20220701-130106-marostegui.json
* 12:38 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 12:38 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 12:37 moritzm: uploaded rsyslog 8.2102.0-2+deb11u1+wmf2 to component/rsyslog-k8s (backport of latest security fixes on top of the rsyslog with mmkubernetes plugin)
* 12:09 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 12:09 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30723 and previous config saved to /var/cache/conftool/dbconfig/20220701-120657-ladsgroup.json
* 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 12:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30722 and previous config saved to /var/cache/conftool/dbconfig/20220701-120636-ladsgroup.json
* 12:02 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 12:02 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30721 and previous config saved to /var/cache/conftool/dbconfig/20220701-115414-ladsgroup.json
* 11:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P30720 and previous config saved to /var/cache/conftool/dbconfig/20220701-115131-ladsgroup.json
* 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P30719 and previous config saved to /var/cache/conftool/dbconfig/20220701-113909-ladsgroup.json
* 11:38 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 11:38 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 11:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P30718 and previous config saved to /var/cache/conftool/dbconfig/20220701-113626-ladsgroup.json
* 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P30717 and previous config saved to /var/cache/conftool/dbconfig/20220701-112404-ladsgroup.json
* 11:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30716 and previous config saved to /var/cache/conftool/dbconfig/20220701-112121-ladsgroup.json
* 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30715 and previous config saved to /var/cache/conftool/dbconfig/20220701-110859-ladsgroup.json
* 11:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1172 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30714 and previous config saved to /var/cache/conftool/dbconfig/20220701-110204-ladsgroup.json
* 11:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance
* 11:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance
* 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30713 and previous config saved to /var/cache/conftool/dbconfig/20220701-110117-ladsgroup.json
* 10:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P30712 and previous config saved to /var/cache/conftool/dbconfig/20220701-104612-ladsgroup.json
* 10:45 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s)
* 10:45 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 10:44 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
* 10:44 bmansurov@deploy1002: Started deploy [airflow-dags/research@b3fe77c]: (no justification provided)
* 10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P30711 and previous config saved to /var/cache/conftool/dbconfig/20220701-103107-ladsgroup.json
* 10:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30710 and previous config saved to /var/cache/conftool/dbconfig/20220701-102810-ladsgroup.json
* 10:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 10:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30709 and previous config saved to /var/cache/conftool/dbconfig/20220701-101602-ladsgroup.json
* 09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T309311|T309311]])', diff saved to https://phabricator.wikimedia.org/P30708 and previous config saved to /var/cache/conftool/dbconfig/20220701-094927-ladsgroup.json
* 09:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 09:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 09:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 13 hosts with reason: Maintenance
* 09:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 13 hosts with reason: Maintenance
* 09:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 09:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 08:35 marostegui: Stop mysql on db2073 for cloning db2155
* 07:47 mmandere: kubemaster2001, restart rsyslog
* 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Add db2154 to s8 [[phab:T311493|T311493]]', diff saved to https://phabricator.wikimedia.org/P30705 and previous config saved to /var/cache/conftool/dbconfig/20220701-074607-marostegui.json
* 07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Add db2153 to s1 [[phab:T311493|T311493]]', diff saved to https://phabricator.wikimedia.org/P30704 and previous config saved to /var/cache/conftool/dbconfig/20220701-073512-marostegui.json
* 06:00 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db2091 from dbctl [[phab:T311803|T311803]]', diff saved to https://phabricator.wikimedia.org/P30703 and previous config saved to /var/cache/conftool/dbconfig/20220701-060000-marostegui.json
* 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db2092 from dbctl [[phab:T311802|T311802]]', diff saved to https://phabricator.wikimedia.org/P30701 and previous config saved to /var/cache/conftool/dbconfig/20220701-054102-marostegui.json
* 02:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS bullseye
* 02:16 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage
* 02:13 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage
* 02:06 krinkle@deploy1002: Synchronized wmf-config/: {{Gerrit|I60edfb0f60}} (3/3) (duration: 03m 31s)
* 02:01 krinkle@deploy1002: Synchronized multiversion/: {{Gerrit|I60edfb0f60}} (2/3) (duration: 03m 34s)
* 01:54 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS bullseye
* 01:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS bullseye
* 01:39 krinkle@deploy1002: Synchronized tests/: {{Gerrit|I60edfb0f60}} (1/3) (duration: 03m 32s)
* 01:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage
* 01:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 01:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 01:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 01:31 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage
* 01:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 01:30 krinkle@deploy1002: Synchronized src/: {{Gerrit|I796f38d0f04600c}} (3/3) (duration: 03m 24s)
* 01:26 krinkle@deploy1002: Synchronized multiversion/: {{Gerrit|I796f38d0f04600c}} (2/3) (duration: 03m 32s)
* 01:23 krinkle@deploy1002: Synchronized tests/: {{Gerrit|I796f38d0f04600c}} (1/3) (duration: 03m 41s)
* 01:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 01:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 01:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 01:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 01:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS bullseye
* 01:12 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS bullseye
* 01:02 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage
* 01:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS bullseye
* 00:57 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: host reimage
* 00:53 ejegg: updated payments-wiki from {{Gerrit|ef53c82e}} to {{Gerrit|78dee85e}}
* 00:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2168.mgmt.codfw.wmnet with reboot policy FORCED
* 00:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2167.mgmt.codfw.wmnet with reboot policy FORCED
* 00:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage
* 00:42 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage
* 00:37 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2162.codfw.wmnet with OS bullseye
* 00:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2168.mgmt.codfw.wmnet with reboot policy FORCED
* 00:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2167.mgmt.codfw.wmnet with reboot policy FORCED
* 00:27 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2166.mgmt.codfw.wmnet with reboot policy FORCED
* 00:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2165.mgmt.codfw.wmnet with reboot policy FORCED
* 00:23 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS bullseye
* 00:05 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2166.mgmt.codfw.wmnet with reboot policy FORCED
* 00:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2163.mgmt.codfw.wmnet with reboot policy FORCED
* 00:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2165.mgmt.codfw.wmnet with reboot policy FORCED


== 2016-01-03 ==
==Archives==
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jan  3 02:31:58 UTC 2016 (duration 6m 52s)
See [[Server Admin Log/Archives]].
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 22s)
<noinclude>
 
[[Category:SAL]]
== 2016-01-02 ==
[[Category:Operations]]
* 03:34 twentyafterfour: deploying https://gerrit.wikimedia.org/r/261725, restarted apache2 on iridium
</noinclude>
* 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan  2 02:31:28 UTC 2016 (duration 6m 58s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 09s)
* 01:04 YuviPanda: imported vagrant 1.8.1 for jessie per bd808
* 00:04 ori: (at 23:46 UTC) restarted nova-compute on labvirt1002
 
== 2016-01-01 ==
* 23:50 legoktm: restarted nodepool on labnodepool1001
* 23:37 ori: restarting nodepool on labnodepool1001.eqiad.wmnet (T122731)
* 19:41 bd808: Updated scholarships.wikimedia.org with latest translation data from translatewiki
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan  1 02:30:27 UTC 2016 (duration 6m 47s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 58s)
 
== 2015-12-31 ==
* 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 31 02:31:43 UTC 2015 (duration 6m 52s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 25s)
 
== 2015-12-30 ==
* 15:05 andrewbogott: restarting puppet on seaborgium; openldap will restart but config should not change
* 14:57 andrewbogott: restarting puppet on serpens; openldap will restart but config should not change
* 13:14 mark: labstore1001: apt-get install irqbalance
* 08:01 jynus: setting dbstore1001 to read_only, converting ruwiki.recentchanges back to InnoDB
* 05:27 ejegg|away: set drupal variable wmf_common_requeue_max back to 10
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 30 02:32:15 UTC 2015 (duration 6m 55s)
* 02:26 gwicke: restbase: finished full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 50s)
* 02:17 gwicke: restbase: starting full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
* 02:14 gwicke: restbase: canary deploy of 7db8e216 (small bug fix & a security fix) to restbase1001
* 00:08 andrewbogott: restarting nova-compute on labvirt1002
 
== 2015-12-29 ==
* 23:56 andrewbogott: restarting nodepool on labnodepool1001
* 23:00 ejegg: re-enabled fundraising banner campaigns
* 22:50 ejegg: restarted fundraising scheduled jobs
* 22:14 ejegg: shut down campaigns and scheduled jobs due to dead ActiveMQ
* 22:13 andrewbogott: restarting slapd on seaborgium and serpens
* 22:11 andrewbogott: disabling puppet on seaborgium and serpens
* 16:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off AB test for search lang detect via accept-language (duration: 00m 29s)
* 16:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable cross-wiki upload A/B test [[gerrit:261371]] (duration: 00m 31s)
* 15:06 apergos: labs salt instances salt update in progress.  It's slow and tedious and automated.  A few hundred instances already done, the rest are going one at a time. Only instances that use the labcontrol salt master will be affected.
* 13:29 apergos: salt wm2 packages now installed on all production hosts except for: mw1041.eqiad.wmnet, technetium.eqiad.wmnet, mw1228.eqiad.wmnet, ms-be1011.eqiad.wmnet
* 13:08 apergos: labcontrol*, neodymium and palladium updated to latest salt packages (wm2), rest of prod to follow
* 12:24 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Attempt to fix math related fatal (duration: 00m 33s)
* 09:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Setting weight values for s6 to original production values (duration: 00m 35s)
* 09:37 jynus: changing the mysql master of db2028, from db1030 to db1050
* 02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 29 02:41:49 UTC 2015 (duration 6m 38s)
* 02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 15m 38s)
* 01:02 logmsgbot: awight@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice: Update CentralNotice: T122251 (duration: 00m 34s)
 
== 2015-12-28 ==
* 21:45 gwicke: restbase: rolling restart to apply https://gerrit.wikimedia.org/r/261206
* 21:26 mutante: tin & mira: started salt minions that were in status stop/waiting
* 21:25 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 30s)
* 21:20 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: $wmfSwiftConfig convenience variable (duration: 00m 30s)
* 20:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1050 & db1022 after emergency fix (duration: 00m 31s)
* 20:52 mutante: cygnus - starting salt-minion
* 20:35 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/261200/ (duration: 00m 31s)
* 19:22 ejegg: updated DjangoBannerStats from 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd to a64fe0e373a978d3df0b7f1dd74ac4cc5c78d34e
* 18:46 jynus: importing wikishared from x1-master into analytics-slave and setting up replication
* 18:28 jynus: restarting and upgrading db1050, using the fact that it is depooled
* 17:16 paravoid: disabled varnish TBF and force-ran puppet on all cp* hosts (I12ea52165e125aaf4ed779399f34cff16d5cd140)
* 16:38 jynus: applying production-side replication filters for wikimania2017wiki on labs
* 16:24 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 30s)
* 16:14 logmsgbot: krenair@tin Synchronized dblists: (no message) (duration: 00m 29s)
* 16:13 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
* 16:12 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/wikimania2017wiki.png: (no message) (duration: 00m 31s)
* 16:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/260521/ (duration: 00m 30s)
* 14:55 jynus: cloning db1050's mysql data to db1022
* 14:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1050 (duration: 00m 31s)
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 28 02:30:47 UTC 2015 (duration 6m 58s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 55s)
 
== 2015-12-27 ==
* 03:05 YuviPanda: run nodetool clearsnapshot -- v3 and nodetool clearsnapshot -- v1 on maps-test2001
* 02:45 YuviPanda: run drop keyspace v3; on csql on maps-test1001 for yurik
* 02:42 YuviPanda: run drop keyspace v1; on csql on maps-test1001 for yurik
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 27 02:30:29 UTC 2015 (duration 6m 59s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 35s)
 
== 2015-12-26 ==
* 19:12 paravoid: restarting varnish-frontend on cp3042
* 19:06 jynus: setting db1030 as the new master of db2028
* 18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1022 (duration: 00m 30s)
* 18:28 jynus: disabling lag notifications for codfw (s6)
* 18:01 paravoid: cp3048: cleaned up /run/vmod_tbf/tbf.db/, kept a backup copy under ~faidon
* 17:55 paravoid: cp3048: service varnish-frontend stop (sending 429 to lots of people, T122453)
* 05:38 paravoid: rolling restart of hhvm jobrunners (T122069)
* 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 26 02:31:39 UTC 2015 (duration 6m 54s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s)
 
== 2015-12-25 ==
* 15:50 jynus: testing new mariadb packages on db2070
* 13:19 jynus: setting db2018's binlog_format as MIXED
* 10:51 jynus: powercycle cp3010
* 09:17 jynus: powercycling cp4007 (unresponsive to ssh, ping, serial console)
* 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 25 02:29:49 UTC 2015 (duration 6m 56s)
* 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)
 
== 2015-12-24 ==
* 23:28 mutante: powercycled mw1114
* 23:27 mutante: i just reset dra on mw1114 because it said it was in use and i didnt see a log yet :;p
* 23:26 robh: mw1114 spammed all icinga errors, system is outputting endless scroll of login prompt, not halting for input (like anohter session or crash cart is sending it, or an error)
* 17:31 gwicke: aqs: tweaked table properties for local_group_default_T_pageviews_per_article_flat: 2 months max DTCS window size, deflate compression
* 17:20 jynus: restarting and reconfiguring mysql at db2066
* 16:45 jynus: restart and reconfigure mysql at db2059
* 16:11 jynus: restart and mysql reconfguration of db2052
* 15:02 jynus: restarting and reconfiguring mysql at db2045
* 14:11 paravoid: powercycling mw1012, OOM'ed/stuck
* 14:09 paravoid: rolling restart of hhvm jobrunners (T122069)
* 14:06 jynus: restart and reconfigure mysql at db2038
* 12:32 jynus: restart and reconfigure mysql at db2065
* 12:12 jynus: restart and reconfiguring mysql for db2058
* 11:50 jynus: restarting and reconfiguring mysql at db2051
* 11:28 jynus: restarting and reconfiguring mysql at db2044
* 10:33 jynus: restarting 's2' replication on dbstore200[12] after cloning
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 24 02:30:40 UTC 2015 (duration 6m 52s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 03s)
* 01:05 awight: update payments from bae4d02afd8cfe1f8b8617c2f74bb36e420d281d to a7785baa7b40b442ecf0b60d47572502d0759780
* 00:38 gwicke: restbase1003: starting `nodetool cleanup`
 
== 2015-12-23 ==
* 23:31 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/ve-graph: https://gerrit.wikimedia.org/r/#/c/260868/ (duration: 00m 31s)
* 19:59 mutante: restbase1004 - puppet stopped and host key changed, what's up?
* 19:42 mutante: ran puppet on mw2112
* 19:41 mutante: logstash1002 - started logstash service
* 19:22 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Remove unused $wgMaxSquidPurgeTitles setting (duration: 00m 30s)
* 18:55 ejegg: updated fundraising tools from ebed29c0eccf38c812b20a957b3487a15bfa9cbc to 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973
* 18:51 ejegg: updated fundraising dashboard from 59e51c4ff74c3c584daf6c5de3bb66daa764cd28 to af8a493ab9ac5431e0d294e5019ac4e426ac6e08
* 18:38 jynus: restart and reconfigure mysql at db2037
* 18:17 mutante: bohrium - finish install, signing puppet certs
* 17:11 jynus: cloning s2 databases from dbstore2001 to dbstore2002 (s2 replication disabled on both)
* 14:05 jynus: restart and reconfigure mysql at db2064
* 13:26 jynus: restart and reconfigure mysql at db2063
* 12:41 jynus: reenabling event scheduler on db1046 (eventlogging m4-master)
* 12:30 jynus: restart and reconfigure mysql at db2056
* 12:14 godog: upgrade cassandra on aqs1003
* 11:24 jynus: reloading and reconfiguring mysql on db2049
* 11:11 godog: roll-upgrade cassandra to 2.1.12 on aqs100[123]
* 10:55 jynus: rebooting and reconfiguring mysql on db2041
* 10:11 jynus: restarting and reconfiguring mysql at db2035
* 09:52 gwicke: rebuilding restbase1004
* 09:51 gwicke: wiped & started boostrap on restbase1008
* 09:18 gwicke: nodetool removenode e2813bb9-f1f2-4d21-ac19-95a7a35b4513 in preparation for adding 1004 to the cluster without bootstrap
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 23 02:30:25 UTC 2015 (duration 7m 1s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 18s)
* 00:40 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/260696 & https://gerrit.wikimedia.org/r/260699 (duration: 05m 28s)
* 00:37 mutante: mw1133 - powercycle
* 00:36 legoktm: manually fixed up stuck global rename of "RCJU-ArCJ" -> "Archives cantonales jurassiennes"
* 00:31 matt_flaschen: Ran UPDATE flow_workflow SET workflow_page_id = 41854369 WHERE workflow_wiki = 'enwiki' AND workflow_namespace = 5 AND workflow_title_text = 'Flow/Developer_test_page' AND workflow_page_id = 48099373; to work around DB inconsistency (T117812)
 
== 2015-12-22 ==
* 21:46 gwicke: restbase1004: tune2fs -m 0 /dev/mapper/restbase1004--vg-srv
* 21:45 gwicke: restbase1004: restarted bootstrap
* 21:22 gwicke: restbase1003: restarting cassandra to clear up disk space from old stream
* 21:11 gwicke: restbase1008: restarting cassandra to clear up disk space from old stream
* 18:36 robh: silver returned to normal service, wikitech.w.o certificate renewed.
* 18:26 robh: silver puppet staying stalled during toollabs issue (we dont want to rehup silver web serivce)
* 18:17 robh: puppet disabled on silver, going to update wikitech.wikimedia.org certificate
* 18:10 jynus: disabling event scheduling on db1046
* 18:03 jynus: rolling schema change (ALTER TABLE ENGINE=TokuDB) on m4-master (db1046) log (eventlogging)
* 16:44 godog: bounce cassandra on restbase1004, restart bootstrap
* 16:42 mutante: powercycling crashed mw1144
* 16:41 jynus: converting dbstore2001 (delayed slave) into an actual delayed slave, adding redundancy to dbstore1002
* 16:40 godog: bounce cassandra on restbase1003
* 16:15 akosiaris: upgrade cassandra on maps-test2001
* 16:15 akosiaris: upgrade cassandra on maps-test2002
* 15:53 mutante: kafka1001,1002 - crit - eventlogging not running (?)
* 15:52 mutante: restbase1003 - disk space, restbase1008 - disk space, restbase1004 - cassandra cql refused
* 15:23 akosiaris: upgrade cassandra on maps-test2003
* 15:06 jynus: restarting and reconfiguring mysql at dbstore2001
* 15:06 mutante: labtestcontrol2001 - puppet had not been running for a while, a bunch of changes have been applied incl. keys and passwords
* 15:04 mutante: enabling puppet on labtestcontrol2001
* 15:04 akosiaris: upgraded cassandra on maps-test2004
* 11:54 apergos: salt packages with wmf packages precise running on ms-{bf}e* in esams; trusty running on analytics103* in eqiad; jessie running on restbase2* in codfw
* 11:43 godog: restart cassandra bootstrap on restbase1004
* 10:09 jynus: online resizing /srv/postgres on labsdb1006 +100GB
* 10:06 hashar: Restarting Jenkins
* 09:54 apergos: precise and trusty salt packages with wmf patches deployed manually on dataset1001 and analytics1001, seem to work fine
* 08:42 jynus: restarting and reconfiguring mysql at db2036
* 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 22 02:30:28 UTC 2015 (duration 6m 54s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)
* 00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/260492/ (duration: 00m 32s)
* 00:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightDialogTool.js: https://gerrit.wikimedia.org/r/#/c/260429/ (duration: 00m 30s)
 
== 2015-12-21 ==
* 22:41 cwd: updated paymentswiki from a1be1ad134d06464e98de180227554fceddc91d4 to bae4d02afd8cfe1f8b8617c2f74bb36e420d281d
* 20:49 godog: restbase1004 bootstrap failed, restbase1007-a is down java.lang.RuntimeException: A node required to move the data consistently is down (/10.64.0.230).
* 19:27 legoktm: running checkLocalUser.php --delete=1 for real this time on terbium
* 19:22 godog: reimage restbase1004
* 19:14 paravoid: powercycling mw1011
* 19:11 paravoid: rolling restart of hhvm on the eqiad jobrunners
* 18:47 jynus: common-sync: Copying to mw1016.eqiad.wmnet from tin.eqiad.wmnet
* 18:35 ori: correction: previous log message was for mw1015, not mw1017
* 18:27 ori: mw1017: enabled jemalloc profiling, restarted hhvm, now running hhvm-collect-heaps
* 17:48 akosiaris: restarted hhvm on mw1012.eqiad.wmnet
* 16:57 thcipriani: timeout on sync-file to mw1016.eqiad.wmnet
* 16:56 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/Popups/Popups.hooks.php: SWAT: Use ExtensionRegistry to determine whether TextExtracts is installed [[gerrit:260346]] (duration: 02m 48s)
* 16:34 jynus: sync-common to mw1085
* 16:26 jynus: powercycling mw1085.eqiad.wmnet
* 16:22 thcipriani: mw1085.eqiad.wmnet times out on SSH connection
* 16:19 godog: reboot restbase1007, load through the roof
* 16:18 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice/resources/subscribing/ext.centralNotice.geoIP.js: SWAT: Update CentralNotice [[gerrit:260316]] (duration: 03m 03s)
* 16:08 godog: depool restbase1007
* 16:01 apergos: jessie packages for salt with local patches deployed on restbase1001, looks fine but just in case.
* 15:44 godog: adding new 1TB disk to restbase1007
* 14:22 andrewbogott: disabling puppet on labnet1002 for dnsmasq tests
* 14:07 MaxSem: me and yurik are nuking old maps data and reimporting planet
* 13:46 jynus: extending online s2-master data disk by +100GB
* 13:15 akosiaris: disabled puppet on maps-test2001 and commented out osmupdater crontab entry until we fix the sync process
* 11:02 jynus: emergency restart of db1047's mysql
* 09:54 jynus: reenabling semisync replication on s3
* 09:07 godog: stop cassandra on restbase1004, decomissioned
* 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 21 02:29:51 UTC 2015 (duration 6m 47s)
* 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 45s)
* 02:20 andrewbogott: disabling puppet on labnet1002 to mess with dnsmasq
* 01:44 andrewbogott: disabled puppet on holmium and labservices1001 to control roll-out of https://gerrit.wikimedia.org/r/#/c/260037/
 
== 2015-12-20 ==
* 23:24 Reedy: Katie and Jeff paged about bellatrix
* 18:46 andrewbogott: graceful restart of zuul as per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
* 18:31 andrewbogott: restarting stuck Jenkins
* 17:47 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.6
* 17:11 godog: depool mw1228, reported ro fs
* 15:53 logmsgbot: reedy@tin Synchronized README: noop (duration: 00m 32s)
* 15:50 Reedy: reedy@tin Purged l10n cache for 1.27.0-wmf.6 (hanging due to mw1228 issue)
* 15:42 Reedy: mw1228 reporting readonly fs
* 15:41 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.7
* 09:00 godog: powercycle ms-be2019, xfs lockup
* 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 20 02:28:49 UTC 2015 (duration 6m 54s)
* 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 59s)
 
== 2015-12-19 ==
* 21:55 _joe_: restarted zotero on sca1001, various OOM messages
* 20:48 gwicke: restbase1004: `systemctl mask cassandra` in preparation for the decommission finishing
* 19:49 akosiaris: killed gmond on db2036. it was clearly misbehaving and running since Jan 02. db2036 was not listed on the ganglia web interface. killing the orphaned process and restarting seems to have fixed it
* 18:54 akosiaris: scheduled maintenance of s3 slave lag on db2036, db2043, db2050, db2057 (all of db2018's family that pages) to effectively silence pages while debugging. Check is flapping since 15:00 UTC today
* 15:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/259611/ - noop for prod, other than making icinga stop complaining (duration: 00m 31s)
* 10:07 hashar: CI jobs for MediaWiki were broken because of cssjanus dependency. Should be fixed once mw/core https://gerrit.wikimedia.org/r/#/c/260169/ lands
* 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 19 02:28:56 UTC 2015 (duration 6m 53s)
* 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 53s)
* 01:01 gwicke: entire restbase cluster: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase$NODE--vg-{srv,var}
* 00:49 gwicke: restbase1008: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase1008--vg-srv
 
== 2015-12-18 ==
* 22:57 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki/mediawiki.searchSuggest.js: allow override of suggestion type reported in event loggin (duration: 00m 29s)
* 22:56 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
* 22:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 30s)
* 22:30 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
* 22:20 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregator.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 31s)
* 19:26 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust queue "maxPartitionsTry" and timeouts (duration: 00m 30s)
* 18:49 mutante: disregard that, apache config only is enough
* 18:47 mutante: gerrit will restart in a moment and be right back
* 18:44 ori: ditto
* 18:43 Krinkle: Created account "Krinkle" on collabwiki
* 16:28 twentyafterfour: restarted apache on iridium to deploy redirect script changes
* 16:20 jynus: restarting and reconfiguring mysql on db1047
* 14:57 godog: stop compactions on restbase1008
* 14:55 jynus: SET GLOBAL query_cache_type = 0; on db1025
* 14:54 hashar: gallium: restarted apache2 , was deadlocked/unresponsive somehow
* 14:44 godog: update privatesettings with swift codfw configuration
* 14:43 godog: set temp-url-key for mw:media account in swift codfw
* 12:19 paravoid: upgrading tor on radium, rebooting for kernel upgrade
* 12:18 _joe_: disabled puppet on all lvs hosts for a potentially harmful change (should be a noop)
* 11:47 _joe_: restarted hhvm on mw1107, stuck at startup
* 11:40 hashar: logstash: reorganized list of dashboards per sections  https://logstash.wikimedia.org/#/dashboard/elasticsearch/default
* 09:43 akosiaris: rebooting planet1001, memory exhaustion, OOM showed up
* 09:20 hashar: Killed Zuul entirely, the queues were full / deadlocked.  Patches need to be retriggered
* 06:47 gwicke: restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space
* 03:07 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s)
* 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s)
* 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s)
* 01:52 ori: re-enabled puppet on rdb* / mc*
* 01:25 ori: in preparation for Iaefb2d191e, disabling puppet on mc* and rdb*
* 01:21 logmsgbot: krinkle@tin Synchronized docroot and w: (no message) (duration: 00m 32s)
* 00:53 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: Revert Nuke-Flow integration, doesn't work (duration: 00m 32s)
* 00:42 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 3 (duration: 00m 32s)
* 00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add completion suggester to BetaFeatures whitelist (duration: 00m 30s)
* 00:26 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: grumble grumble touch InitialiseSettings grumble (duration: 00m 30s)
* 00:25 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 2 (duration: 00m 32s)
* 00:23 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 30s)
* 00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 29s)
* 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Nuke/: SWAT: Nuke support in Flow, part 1 (duration: 00m 30s)
* 00:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.messagePoster/mediawiki.messagePoster.factory.js: SWAT: fix error in messagePoster (duration: 00m 29s)
* 00:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/MobileFrontend: SWAT: Schema:MobileWebSectionUsage: always log the isTestA field (duration: 00m 31s)
* 00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: cleanup (duration: 00m 30s)
* 00:00 ori: restarted mathoid on sca1001
 
== 2015-12-17 ==
* 23:42 mobrovac: mathoid deploying 8d2295
* 22:47 hashar: ssh to tin is back https://gerrit.wikimedia.org/r/#/c/259876/
* 22:39 hashar: Only tin lost SSH user keys  apparently due to https://gerrit.wikimedia.org/r/#/c/253465/  overriding the admin::groups to simply "eventlogging-admins"
* 22:34 hashar: Ssh User keys are gone on deployment servers ( tin / mira )
* 22:18 eileen1: update CiviCRM from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
* 22:10 eileen1: Updating civicrm from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
* 21:33 hashar: gallium: upgrading Zuul from 2.1.0-60-g1cc37f7-wmf2precise1 .. 2.1.0-60-g1cc37f7-wmf4precise1 .  Should be noop, only change zuul-cloner which is not used there
* 21:29 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4jessie1 to jessie-wikimedia repo
* 21:28 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4trusty1 to trusty-wikimedia repo
* 21:22 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4precise1 to precise-wikimedia APT
* 19:42 mobrovac: mathoid deploying a2187a6
* 19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.9
* 19:05 godog: disable puppet on graphite2001, brief testing cluster aggregations
* 19:01 thcipriani: starting update of all wikis to 1.27.0-wmf.9
* 18:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool x1-slave (db1031), increase db1041 load to 100% (duration: 00m 30s)
* 18:06 gwicke: running `nodetool cleanup` on restbase1001 and restbase1005
* 18:04 robh: calcium is supposed to be down, reclaiming to spares, ignore any irc alerts (its in maint mode in icinga)
* 17:55 gwicke: running `nodetool cleanup` on restbase1002
* 17:18 jynus: setting mysql db1031 as db2009's master
* 17:05 jynus: restarting and reconfiguring mysql at db2009
* 16:49 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Math: SWAT: Make math usable without RESTbase [[gerrit:259734]] (duration: 00m 30s)
* 16:41 jynus: problems with corruption on x1-slave for cebwiki. Fixed them. Will leave db1031 depooled for a while to check they are gone.
* 16:39 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/includes/Translation.php: SWAT: Fix Undefined index: targetRevisionId in ContentTranslation [[gerrit:259649]] (duration: 00m 29s)
* 16:23 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: Actually define tags for cross-wiki upload A/B test [[gerrit:259729]] (duration: 00m 31s)
* 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test in additional languages [[gerrit:259665]] (duration: 00m 30s)
* 16:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Fix array-to-string conversion [[gerrit:259633]] (duration: 00m 30s)
* 15:57 moritzm: stopping opendj LDAP servers on nembus/neptunium (read-only since about days now due to migration to openldap)
* 15:56 akosiaris: depool sca1001, playing with cxserver config
* 15:24 jynus: reinstall, reboot and reconfigure mysql at db1031
* 15:03 moritzm: installing git security updates
* 14:23 _joe_: restarting HHVM on the first jobrunners
* 14:11 jynus: soft-rebooting mw1004, responsive to ping, but not to salt, ssh
* 14:06 godog: nodetool stop -- CLEANUP on restbase1002
* 13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1031 (x1-slave) for maintenance (duration: 07m 30s)
* 13:11 moritzm: starting to restart hhvm on application servers (to effect security updates for libxml2, openssl and others)
* 13:01 akosiaris: moved old cruft /srv/deployment/cxserver/deploy/src/config.js out of the way
* 12:51 akosiaris: repooled sca1002 for cxserver
* 12:51 akosiaris: restarted cxserver on sca1002
* 12:33 akosiaris: depooling sca1002 for cxserver
* 12:33 akosiaris: repooling sca1001 for cxserver
* 12:24 kart_: Updated cxserver on sca1002
* 12:08 akosiaris: disable puppet, stop salt-minion on sca1002
* 12:08 akosiaris: depool sca1001 from cxserver service.
* 11:56 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1041 with low weight (duration: 00m 37s)
* 11:33 jynus: performing schema change on officewiki
* 11:31 jynus: performing schema change on x1-master flowdb
* 11:20 jynus: performing schema change on x1-master wikishared.cx_translations
* 10:47 jynus: performing schema change on s7-master metawiki.oauth_registered_consumer
* 10:30 godog: nodetool stop -- CLEANUP restbase1004
* 02:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 17 02:53:11 UTC 2015 (duration 7m 12s)
* 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 22s)
* 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 10m 32s)
* 00:28 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: I552cf6b0420: Upgrade some ApiStashEdit logging calls to info() (duration: 00m 30s)
* 00:05 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for sysadmin group (duration: 00m 29s)
 
== 2015-12-16 ==
* 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (take 2) (duration: 00m 30s)
* 23:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (duration: 00m 30s)
* 23:27 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext to master - protocol issue (duration: 00m 32s)
* 23:19 ejegg: updated DjangoBannerStats from 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77 to 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd
* 23:10 legoktm: running foreachwiki extensions/CentralAuth/maintenance/checkLocalUser.php --verbose=1 --delete=1 (T119736)
* 23:07 robh: fermium returned to normal service with new/renewed ssl cert for lists.w.o
* 22:58 robh: disabled puppet on fermium for the lists.w.o cert update
* 22:58 robh: librenms returned to normal service (puppet renabled on netmon1001)
* 22:48 robh: puppet disabled on netmon1001 for librenms certificate update
* 22:17 ejegg: updated DjangoBannerStats from 26b48add5c6f70b395a3195c401a12612c626796 to 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77
* 21:31 subbu: finished deploying parsoid 64029e12
* 21:29 bearND: mobileapps deployed sha1 9f91ad5
* 21:23 subbu: restarted parsoid on wtp1005 as a canary
* 21:19 subbu: starting parsoid deploy
* 21:14 bearND: starting mobileapps deploy
* 20:18 ejegg: updated SmashPig from b3ee85aab38fdeef3f045ba1733f582e9a5006b2 to 072c7ec6ed94e7074ba35b7986d5dde94866fe2f
* 19:46 ostriches: iridium: repacking MW / OPUP repositories as phd user. This needs to be a cron.
* 19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.9
* 19:00 thcipriani: updating group1 wikis to 1.27.0-wmf.9
* 18:39 jynus: performing schema change on ruwiki.geo_tags on db2046
* 17:46 jynus: setting mysql db1041 as db2029's master
* 17:21 jynus: restarting and configuring mysql on db2029 (there will be an increase of errors- from pings, not real traffic)
* 17:08 jynus: setting mysql db1022 as db2028's master
* 16:50 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: graph namespace removal/cleanup (duration: 00m 31s)
* 16:41 gwicke: cleared snapshots on cassandra cluster
* 16:32 gwicke: restarted `nodetool decommission` on restbase1004
* 16:31 jynus: restarting and reconfiguring mysql on db1041
* 15:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022, depool db1041 (duration: 00m 30s)
* 15:53 godog: cassandra on restbase1004 couldn't finish decomissioning, out of disk space, running nodetool clean
* 15:28 jynus: restarting and reconfiguring mysql on db2028
* 15:17 yurik: updated graphoid service
* 15:16 moritzm: removed ecryptfs-utils across the cluster
* 15:09 godog: roll-restart swift daemons on ms-be1*
* 15:04 jynus: setting db2023 master to be now db1049 instead of m5-master
* 14:56 mobrovac: restbase roll-restarting restbase
* 14:52 godog: reenable and run puppet on restbase* and aqs*
* 14:49 mobrovac: restbase restart rb1001
* 14:46 urandom: starting `nodetool cleanup' on restbase1009-a.eqiad (https://phabricator.wikimedia.org/T121535)
* 14:41 urandom: starting `nodetool cleanup' on restbase1006.eqiad (https://phabricator.wikimedia.org/T121535)
* 14:17 mobrovac: restbase reenabling and running puppet on canary rb1001
* 14:08 godog: reenable puppet on restbase test cluster
* 14:06 akosiaris: delete empty panel in https://grafana-admin.wikimedia.org/dashboard/db/graphoid
* 14:04 akosiaris: repool scb1001 for mobileapps
* 13:59 akosiaris: depool scb1001 for mobileapps (oid)
* 13:58 akosiaris: repool sca1002 for citoid, mathoid, graphoid, cxserver
* 13:55 godog: enable puppet on cerium and bounce restbase
* 13:55 moritzm: restarting HHVM on canary appservers to effect various security updates (libxml, openssl, gs, freetype)
* 13:50 godog: enable puppet on restbase-test2001 and bounce restbase
* 13:47 godog: (retroactive) enable puppet on sca1001
* 13:46 akosiaris: depooling sca1002, scb1002 for citoid, cxserver, graphoid, mathoid
* 13:35 godog: disable puppet on deployment_target:restbase/deploy and sca/scb
* 12:29 jynus: restarting and reconfiguring mysql at db1022
* 12:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1022 for maintenance (duration: 00m 37s)
* 11:49 jynus: restarting and reconfiguring mysql on db2023
* 11:34 akosiaris: merged YuviPanda: labstore: Run sync-exports in start-nfs too (d5273d9) and YuviPanda: labstore: Activate volumes before mounting them (dff32de) on palladium
* 08:54 YuviPanda: start nfs-kernel-server on labstore1001
* 08:53 YuviPanda: stopped nfs-kernel-server on labstore1001
* 08:36 mark: Rebooting labstore1001
* 03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 16m 00s)
* 02:54 papaul: kafka200[1-2] signing puppet certs, salt key initial run
* 02:42 papaul: installation complete on kafka200[1-2]
* 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 13m 37s)
* 02:18 papaul: installing OS on kafka200[1-2]
* 00:40 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on all wikis (duration: 00m 30s)
* 00:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on mediawikiwiki (duration: 00m 30s)
* 00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: SWAT: update Vega (duration: 00m 30s)
* 00:12 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for staff, take 3; fix $wgExtractsRemoveClasses (duration: 00m 31s)
 
== 2015-12-15 ==
* 23:43 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to b307d744def9289a7f86cb02bc6e1a00225e474d
* 23:31 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to 49e949c303466db9f96cb38b718190c887f7ed89
* 23:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
* 23:25 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enabling Wikidata data access for meta-wiki (duration: 00m 29s)
* 23:24 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
* 23:13 Krinkle: Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/extension.json 'T121308 - unbreak VE'
* 22:14 gwicke: restbase: finished full deploy of 3b7ae07e to restbase prod cluster
* 22:13 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9 mira test
* 22:07 gwicke: restbase: starting full deploy of 3b7ae07e to restbase prod cluster
* 22:05 gwicke: restbase: canary deploy of 3b7ae07e to restbase1001
* 21:53 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9
* 21:39 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache (duration: 29m 58s)
* 21:09 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache
* 20:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: ad2e2aeedc: Make edit stashing use named DB locks (duration: 01m 46s)
* 20:21 thcipriani: 1.27.0-wmf.9 branching complete 1h 18m 2s, checking out to tin
* 20:00 subbu: restarted parsoid on all nodes to kill stuck processes (thanks to T104523)
* 19:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1049 and db1042 (duration: 00m 29s)
* 19:57 subbu: restarted parsoid on wtp1019 to kill stuck processes
* 19:42 mutante: mw1146: hhvm restart
* 19:39 mutante: ms-fe1001 thru ms-fe1003:  swift-proxy-server stop/start
* 19:37 mutante: ms-fe1004, swift-proxy-server stop/start
* 19:23 mutante: powercycling mw1129
* 19:08 ottomata: merged change to allow longer programnames in remote rsyslog config. 
* 19:00 thcipriani: starting branch cut for 1.27.0-wmf.9
* 18:57 bblack: repooling cp1053 (eqiad text cache)
* 18:37 ori: Depooled and drained mw1161-1169 app servers, now re-purposing as job runners, per T121549
* 18:19 jynus: changing db2019 master to be db1042 instead of m4-master
* 17:55 bblack: manually enabled ipsec rules in iptables on kafka10xx - puppet disabled for now until I can fix the puppetization of it...
* 17:48 urandom: beginning `nodetool cleanup' on restbase1003.eqiad (https://phabricator.wikimedia.org/T121535)
* 17:48 jynus: restart and reconfigure mysql at db1049
* 17:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, Depool db1049 (duration: 00m 30s)
* 17:33 urandom: beginning `nodetool cleanup' on restbase1005.eqiad (https://phabricator.wikimedia.org/T121535)
* 17:30 gwicke: aborted large compaction on restbase1004 with `nodetool stop -- COMPACTION` to free disk space
* 17:29 urandom: beginning `nodetool cleanup' on restbase1002.eqiad (https://phabricator.wikimedia.org/T121535)
* 17:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Update cirrus avro schema to 111448028943 (duration: 00m 29s)
* 17:00 jynus: restarting and reconfiguring mysql at db2018
* 16:34 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bump portals to master (duration: 00m 29s)
* 16:30 jynus: restarting and reconfiguring mysql on db1042
* 16:26 godog: decommission restbase1004
* 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Centralise feedback from test2wiki to MediaWiki.org [[gerrit:259041]] (duration: 00m 30s)
* 16:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Enable single edit tab mode on test2wiki [[gerrit:259039]] (duration: 00m 29s)
* 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: In VisualEditor on single edit tab wikis, set the default editor appropriately [[gerrit:258403]] (duration: 00m 28s)
* 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: BetaFeatures: Update language and dates of "retirement" [[gerrit:258409]] (duration: 00m 29s)
* 16:14 jynus: switch db2018's master from s3-master to db1027
* 16:09 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test on English-language wikis [[gerrit:259258]] (duration: 00m 29s)
* 16:08 logmsgbot: thcipriani@tin Synchronized wmf-config/db-eqiad.php: SWATish: Depool db1042 for maintenance (duration: 00m 29s)
* 15:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1042 for maintenance (duration: 00m 29s)
* 15:18 jynus: restarting db2018 for upgrade and configuration change
* 14:41 jynus: restarting mysql on db1027 to apply new configuration
* 13:53 bblack: disabling puppet on neon to avoid race-condition ipsec alert spam
* 13:45 hashar: reverted composer upgrade on CI with https://gerrit.wikimedia.org/r/#/c/259241/
* 13:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 after maintenance; depool db1027 for maintenance (duration: 00m 29s)
* 13:37 hashar: bumping composer on CI to  1.0.0-alpha11 https://gerrit.wikimedia.org/r/#/c/258933/
* 12:48 mobrovac: restbase deploy end of 844a41d
* 12:39 mobrovac: restbase deploy start of 844a41d
* 11:46 jynus: restarting and upgrading dbstore2002
* 11:33 paravoid: force-rebooting stat1002, kernel borked because of fuse
* 11:25 paravoid: rebooting lvs4003/lvs4004 for kernel upgrade
* 11:21 paravoid: rebooting lvs3004 for kernel upgrade
* 11:01 jynus: reloading haproxy on dbproxy1002
* 10:20 jynus: stopped eventlogging on dbstore1002 and db1047
* 10:09 jynus: enabling ferm, and restarting mysql at db1046 (m4-master, eventlogging)
* 10:02 jynus: stopping eventlogging mysql consumers
* 09:32 moritzm: umounted/remounted hdfs mount on stat1002 (got stuck due to kernel bug, see T121492)
* 09:32 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings-labs.php: enable EventBus logging channel (currently only in beta) https://phabricator.wikimedia.org/T116786 (duration: 08m 57s)
* 08:47 hashar: restarted zuul-merger on gallium
* 08:29 hashar: stopping zuul-merger on gallium for maintenance
* 07:48 _joe_: rebooting pollux
* 07:18 _joe_: logged into console on pollux, that made it responsive
* 03:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/259011/ and https://gerrit.wikimedia.org/r/#/c/259010/1 (duration: 00m 30s)
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 15 02:32:25 UTC 2015 (duration 6m 59s)
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 11m 38s)
* 02:03 ori: deployed I6ebffe559 to job runners
* 01:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Revert cirrus avro schema to 101446746400 due to camus not picking up events from new schema (duration: 00m 30s)
* 00:34 Reedy: puppet disabled on silver since last Puppet run was at Fri Dec 11 15:27:28 UTC 2015 due to 'testing osm debugging'
* 00:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Graph config updates (duration: 00m 28s)
* 00:32 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Graph config updates (duration: 00m 29s)
* 00:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s)
* 00:17 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Use event-schemas repository for avro schemas (duration: 00m 29s)
* 00:10 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend/: SWAT: fix page invalidation in mobile API (duration: 00m 36s)
* 00:06 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Wikidata config changes (duration: 00m 28s)
 
== 2015-12-14 ==
* 23:47 gwicke: restbase: finished deploy of 9f31847ad to restbase prod cluster
* 23:39 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 82f4f9df64e42fd04bd32395 (duration: 00m 29s)
* 23:34 gwicke: restbase: starting deploy of 9f31847ad to restbase prod cluster
* 22:50 logmsgbot: mobrovac@tin Synchronized wmf-config/InitialiseSettings.php: Enable logging for the Math extension (duration: 00m 29s)
* 22:49 gwicke: restbase: canary deploy of 9f31847ad to restbase1001
* 22:16 robh: puppet enabled on uranium and resumed normal service, cert updated for ganglia
* 22:10 robh: puppet disabled on uranium for ganglia cert update
* 22:09 mobrovac: mathoid deploying 5b20fe1
* 21:34 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 29s)
* 21:15 subbu: finished deploy of parsoid sha df3171e6
* 21:11 subbu: restarted parsoid on wtp1004 (~4 mins back) as a canary
* 21:04 subbu: starting parsoid deploy
* 21:01 robh: neon returned to normal puppet updates.  icinga new cert is live.
* 20:57 mutante: mw1259 - enabled hyperthreading (logical processor)
* 20:54 mutante: rebooting mw1259 for BIOS settting
* 20:50 robh: puppet on neon disabled for certificate update
* 20:25 yurik: updated graphoid service
* 20:15 hashar: scandium restarted zuul-merger
* 20:00 jynus: changing master of db2017 to be now db1018, instead of s2-master
* 19:20 jynus: rolling restart s2 codfw mysql servers
* 19:02 mutante: reboot cygnus
* 18:37 jynus: restarting, upgrading and reconfiguring mysql on db1018
* 18:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1057 after maintenance; depool db1018 for maintenance (duration: 00m 29s)
* 17:20 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationToken.php: SWAT: Fix check for JWT [[gerrit:258776]] (duration: 00m 29s)
* 17:05 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails update [[gerrit:258767]] (duration: 00m 30s)
* 16:27 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% Part II [[gerrit:258985]] (duration: 00m 29s)
* 16:14 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% [[gerrit:258985]] (duration: 00m 30s)
* 16:13 jynus: performing schema change on testwiki.page (s3)
* 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: do not index User namespace on ko.wikipedia [[gerrit:258667]] (duration: 00m 29s)
* 15:01 bblack: "service ganglia-monitor restart" on cp*, because most had no varnish stats flowing into ganglia
* 14:47 hashar: Stopping zuul-merger daemon on scandium.  It lost its disk somehow earlier "DISK CRITICAL - /srv/ssd is not accessible: No such file or directory"
* 13:31 bblack: rebooting cp4005
* 13:30 bblack: issues on cp4005, investigating
* 13:29 mobrovac: restbase cassandra restarting cassandra on rb1004 due tolow disk space caused by compactions
* 13:22 godog: repool restbase1007
* 12:04 mobrovac: restbase deploy on rb1007 after re-imaging
* 11:31 godog: upgrade diamond to 3.5-5 in eqiad
* 11:13 dcausse: elastic in eqiad: resuming writes
* 10:50 jynus: performing schema change on wikishared.cx_translations (x1-master)
* 10:19 dcausse: elastic in eqiad: restarting elastic1016 (to release deleted filedesc)
* 10:11 jynus: restarting mysql at labsdb1004
* 10:11 dcausse: elastic in eqiad: freezing writes (to restart elastic1016)
* 09:46 godog: reimage restbase1007
* 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 14 02:29:39 UTC 2015 (duration 6m 57s)
* 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 00s)
 
== 2015-12-13 ==
* 11:10 dcausse: elastic in eqiad: restarting elastic1012 to release opened log filedesc
* 10:54 godog: extend fluorine's storage lv (94% util) lvresize --size +300G --resizefs  /dev/mapper/vg0-lv0
* 10:24 dcausse: elastic in eqiad: disabling TRACE indexing slowlog on jawiki_content and zhwiki_content
* 07:06 ori: elasticsearch1012 ditto.
* 07:05 ori: elasticsearch1016: Moving /var/log/elasticsearch/* to /var/lib/elasticsearch/logs to free up space.
* 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 13 02:28:52 UTC 2015 (duration 6m 49s)
* 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 08s)
 
== 2015-12-12 ==
* 22:17 ejegg: updated fundraising dashboard from 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859 to 59e51c4ff74c3c584daf6c5de3bb66daa764cd28
* 22:06 ejegg: updated fundraising dashboard from 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4 to 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859
* 21:26 legoktm: running fixDefaultJsonContentPages.php on all wikis (T108663)
* 21:02 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I53f13a159: Add rdb100[5-6] to job queue configuration (duration: 00m 31s)
* 19:30 hashar: Restarted Zuul
* 17:01 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/258669/ (duration: 00m 29s)
* 10:12 godog: bounce nslcd on tools-submit and stop puppet
* 10:01 dcausse: elastic in eqiad: disabling TRACE indexing slowlog for urwiki_content
* 09:44 jynus: recreated events on db1057 with sql_bin_log = 0 and restarted replication on db2016
* 09:15 godog: move old elasticsearch logs on elastic1026 out to /var/lib/elasticsearch/log (/ is full)
* 09:06 godog: move old elasticsearch logs on elastic1016 out to /var/lib/elasticsearch/log (/ is full)
* 09:02 godog: move old elasticsearch logs on elastic1012 out to /var/lib/elasticsearch/log (/ is full)
* 03:02 legoktm: ran fixDefaultJsonContentPages.php --wiki=thwiktionary for T108663
* 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 12 02:28:44 UTC 2015 (duration 6m 53s)
* 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 08m 31s)
* 00:30 paravoid: restarting slapd on seaborgium with manual hack
 
== 2015-12-11 ==
* 23:59 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CentralNotice: Improve impression diet state machine (duration: 00m 32s)
* 22:24 subbu: finished deploying parsoid sha ebd62ab5
* 22:09 subbu: restarted parsoid on wtp1002 as a canary
* 22:03 subbu: starting parsoid deploy
* 21:18 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.8/extensions/OpenStackManager/: deploying https://gerrit.wikimedia.org/r/#/c/258516/ (duration: 00m 28s)
* 20:55 jynus: stopping and reconfiguring replication on db2016 (only for some minutes)
* 20:48 jynus: setting db1057 as the parent mysql of db2016 for s1 replication
* 20:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Flow/maintenance/FlowPopulateRefId.php: Fix fatals due to missing wiki condition (T117786) (duration: 00m 29s)
* 20:06 jynus: restarting and reloading mysql for upgrade and reconfiguration at db1057
* 19:33 logmsgbot: demon@mira Synchronized README: testing (duration: 01m 35s)
* 19:26 legoktm: started foreachwiki checkLocalUser.php (CentralAuth) on terbium
* 19:14 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: CentralAuth logging for Lego (duration: 00m 28s)
* 19:13 mutante: creating cygnus.codfw.wmnet on ganeti2001
* 19:06 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: rm some old ldap stuff. yay hiera! (duration: 00m 28s)
* 18:54 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: rm some deprecated profiling config (duration: 00m 28s)
* 18:29 logmsgbot: demon@tin Synchronized docroot/: cleanup wgconf vhost config (the sequel) (duration: 00m 29s)
* 18:28 logmsgbot: demon@tin Synchronized wmf-config/: cleanup wgconf vhost config (duration: 00m 29s)
* 18:21 cwd: updated payments from 74143e43eb36f93be7881f626443182d3bb58cef to a1be1ad134d06464e98de180227554fceddc91d4
* 18:13 logmsgbot: demon@tin Synchronized wmf-config/: Remove ability to turn off Cirrus completely. It's a scary switch that would Break Everything. There's much better options to use if you need to tune its load or behavior. (duration: 00m 29s)
* 17:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1057 for maintenance (duration: 00m 35s)
* 17:44 jynus: rolling restart of codfw's S1 mysqls finished
* 16:36 mark: Installed bgp-med branch of operations/debs/pybal on pybal-test2001
* 16:23 mark: moved quagga install from pybal-test2001 to pybal-test2002
* 14:54 jynus: restarting and configuring S1:codfw mysqls (db2016,34,42,48,55,62,69,70)
* 14:40 mark: root@pybal-test2001:~# apt-get install quagga
* 14:21 andrewbogott: re-imaging promethium
* 10:57 _joe_: rolling restart of HHVM across the jobrunners to ease memory consumption (this will go on during the day, hhvm restarts will be distantiated by 2 hours each)
* 10:53 _joe_: restarting hhvm on mw1122 in order to collect heap dumps
* 10:35 jynus: restarting HHVM on mw1122, mw1145
* 09:50 jynus: importing several tables from master to dbstore1002 and labs, lag will will be slightly affected
* 09:26 _joe_: ganeti-rebooting serpens, cannot get into console
* 07:10 dcausse: deleting some indexing slow logs on elastic1012 and elastic1016
* 06:46 gwicke: deploy restbase c67a41e9d52: add an expensive title to re-render blacklist, per subbu's request
* 03:21 gwicke: cassandra: upgraded codfw production nodes to 2.1.12
* 02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 11 02:46:13 UTC 2015 (duration 6m 46s)
* 02:39 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 17m 01s)
* 02:34 gwicke: restarted nodetool decommission on restbase1007, as I had to restart one of the stream targets
* 02:15 bblack: unified cert upgrades complete
* 01:42 bblack: starting unified cert upgrade process
* 01:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/jobqueue: Ia44ec5ed4: Add per-partition JobQueueRedis aggregation + Fix bad regex in 6fe2f48df (duration: 00m 31s)
* 00:59 andrewbogott: rebooting promethium
* 00:58 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Add read-more to beta features whitelist (duration: 00m 30s)
* 00:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Search language detection A/B test and RelatedArticles on all Wikipedias in beta (duration: 00m 29s)
* 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch: SWAT (duration: 00m 30s)
* 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/RelatedArticles: SWAT (duration: 00m 29s)
* 00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/ImageMap: SWAT (duration: 00m 29s)
* 00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Graph: SWAT (duration: 00m 30s)
* 00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Staff password policy, take 2 (duration: 00m 28s)
* 00:06 logmsgbot: catrope@tin Synchronized portals/: SWAT (duration: 00m 30s)
 
== 2015-12-10 ==
* 23:55 gwicke: restbase: deploy 9657c4e
* 23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Math: Fix errors (duration: 00m 29s)
* 23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Echo: Fix errors (duration: 00m 29s)
* 23:27 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaEvents: Ia44ec5ed4: Updated mediawiki/core Project: mediawiki/extensions/WikimediaEvents  152ecb10311bb04f4f2f91775cf821aff14aa327 (duration: 00m 30s)
* 23:20 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 29s)
* 22:48 csteipp: Re-deployed a bunch of security patches for wmf8
* 22:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/VisualEditor/: SWAT (duration: 00m 47s)
* 21:56 gwicke: restbase: finished full deploy of 7398850fe9 to restbase cluster
* 21:49 gwicke: restbase: starting full deploy of 7398850fe9 to restbase cluster
* 21:49 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: VisualEditor config patches (duration: 00m 30s)
* 21:46 gwicke: restbase: canary deploy of 7398850fe9 to restbase1001
* 21:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: Fix MediaWiki:MediaWiki: (duration: 00m 29s)
* 21:35 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: New password policy for staff group (duration: 00m 28s)
* 21:28 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
* 21:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
* 21:20 Reedy: Changed email for mbeattie on otrs_wikiwiki per request of mdennis
* 20:32 subbu: finished parsoid deploy
* 20:14 subbu: restarted parsoid on wtp1003 as canary
* 20:10 subbu: starting parsoid deploy
* 19:11 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.8
* 18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1014 at 100% load and reduce es1019 load (duration: 00m 29s)
* 18:33 paravoid: reprepro: updating cassandra to latest upstream (2.1.12)
* 17:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 at 100% load, repool es1014 with low load (duration: 00m 29s)
* 17:06 bd808: Updated scholarships.wikimedia.org to a4e3dbf (Ensure that Auth\UserData is constructed with an array)
* 16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% [[gerrit:258159]] (duration: 00m 29s)
* 16:42 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% [[gerrit:258161]] (duration: 00m 29s)
* 16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add three new groups to pawiki, and allow sysops to add or remove users to them [[gerrit:257881]] (duration: 00m 29s)
* 16:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add a rollbacker group at wuuwiki [[gerrit:253084]] (duration: 00m 28s)
* 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable filemover group at ukwiki [[gerrit:255422]] (duration: 00m 29s)
* 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Nuke and unblockself only for bureaucrats on en.wikiversity [[gerrit:254029]] and Add new group "curator" to enwikiversity [[gerrit:252012]] (duration: 00m 36s)
* 16:00 jynus: restarting, upgrading and configuring es1014
* 15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 31s)
* 15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 28s)
* 15:22 jynus: perming es3 master switchover of es1014 -> es1019, no production impact is expected
* 15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 with lower weight after mantenance (duration: 00m 29s)
* 13:57 jynus: restarting, upgrading and reconfiguring mysql @ es1011
* 13:45 moritzm: stopping/starting slapd on seaborgium to update LDAP indices
* 13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover codfw es2 master es1011 -> es1015 (duration: 00m 29s)
* 13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es2 master es1011 -> es1015 (duration: 00m 37s)
* 13:30 paravoid: reenabled puppet on seaborgium & forced a puppet run
* 13:05 moritzm: restarted slapd on serpens
* 13:03 moritzm: stopping slapd on serpens to update LDAP indices
* 12:54 moritzm: re-enabled puppet on serpens / restarted slapd
* 12:53 jynus: performing switchover of es2 es1011-> es1015 for master maintenance, no production impact is expected
* 12:13 godog: powercycle mw1119, login on console sluggish and no ssh
* 12:11 jynus: codfw es2 server restart finished
* 09:04 jynus: rebooting, restarting and upgrading mysql on es2 (codfw external storage servers)
* 05:51 ejegg|away: updated fundraising dashdoard from 9f0a4d98586f1281716e39b34d924de524458177 to 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4
* 03:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 10 03:55:07 UTC 2015 (duration 54m 36s)
* 03:00 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 15m 54s)
* 02:28 gwicke: updated restbase1008 to cassandra 2.1.12
* 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 53s)
* 01:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 30s)
* 01:23 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 29s)
* 01:20 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Increase elasticsearch master timeout for maint actions to 2 minutes (duration: 00m 31s)
* 01:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
* 01:01 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
* 00:55 awight: update fundraising CRM from 49e949c303466db9f96cb38b718190c887f7ed89 to fa7124e1d8d92b576c7650030fe64d024c822088
* 00:54 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/: MessageBlobStore changes (duration: 02m 29s)
* 00:51 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/skins/Vector/: Make placeholder in logged-out personal bar greyed out (duration: 00m 29s)
* 00:26 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Thanks/: Fix bug with topic titles in Flow thank notifications (duration: 00m 28s)
* 00:18 logmsgbot: catrope@tin Synchronized wmf-config/mobile.php: Enable A/B test to measure impact of collapsing content (duration: 00m 29s)
* 00:06 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgExtDistGraphiteRenderApi (duration: 00m 28s)
* 00:02 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: Shut up unmerged change warning (duration: 00m 30s)
 
== 2015-12-09 ==
* 23:43 papaul: signing salt-key complete
* 23:28 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: Item/ Property id values need separate validators each (duration: 00m 39s)
* 23:26 papaul: auth2001 signing puppet certs
* 21:52 papaul: OS install complete on auth2001
* 21:34 subbu: finished deploying parsoid sha a0c626e4
* 21:30 hashar: Jenkins upgraded to 1.625.3 ( https://phabricator.wikimedia.org/T56599 )
* 21:22 subbu: restarted parsoid on wtp1003 as a canary
* 21:20 hashar: Jenkins going down right now.
* 21:18 subbu: starting parsoid deploy
* 21:11 hashar: Stopping Jenkins
* 19:56 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: hewiki and cawiki to php-1.27.0-wmf.8
* 19:46 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: Train: Add hewiki and cawiki to group1. Part II [[gerrit:257973]] (duration: 00m 29s)
* 19:45 logmsgbot: thcipriani@tin Synchronized dblists/group1-wikipedia.dblist: Train: Add hewiki and cawiki to group1. Part I [[gerrit:257973]] (duration: 00m 33s)
* 19:10 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.8
* 19:02 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/includes/Job/Job.php: Revert manual adjustment made to CirrusSearch Job.php  (duration: 00m 29s)
* 18:56 mutante: mw1003 - restarted hhvm (and apache a bit earlier too)
* 18:16 Jamesofur: inserted decryption key into database for enWP Arbcom case
* 17:31 akosiaris: restarted hhvm on mw1125, memory starvation
* 17:31 akosiaris: restarted hhvm on mw1125
* 17:17 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: SWAT: Update Wikibase [[gerrit:257901]] (duration: 00m 49s)
* 17:05 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 28s)
* 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add rights to CU+OS groups on en.wikipedia.org [[gerrit:255129]] (duration: 00m 29s)
* 16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off language detection user test [[gerrit:254071]] (duration: 00m 29s)
* 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set initial titlesuggest shard sizes [[gerrit:256974]] (duration: 00m 29s)
* 16:18 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: SWAT: Move Catalan and Hebrew Wikipedias to group1 [[gerrit:253371]] (duration: 00m 29s)
* 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable importupload and some import sources for officewiki [[gerrit:256881]] (duration: 00m 42s)
* 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part II [[gerrit:257800]] (duration: 00m 28s)
* 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part I [[gerrit:257800]] (duration: 00m 29s)
* 15:48 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-2
* 15:01 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1019 after maintenance (duration: 00m 33s)
* 14:09 godog: decommission restbase1007
* 13:06 hashar: restarting Jenkins (upgrading Gearman plugin)
* 11:51 akosiaris: reissue certificates for labs-ldap.{eqiad,codfw}.wikimedia.org with a 2048bit key size
* 11:29 kart_: Update cxserver to c88ef57
* 10:45 akosiaris: sudo gnt-instance modify -H cpu_type=SandyBridge seaborgium to get aesni and sse cpu flags. rebooting seaborgium to apply those
* 10:25 akosiaris: reboot serpens to add 3 more vCPUs
* 10:04 moritzm: restarting slapd on serpens/seaborgium to apply updated indices
* 10:03 ori: running hhvm-collect-heaps in screen on mw1001 with a 600s interval
* 10:00 akosiaris: stopped slapd on seaborgium, run sudo -u openldap slapindex to add the new indices, started slapd
* 09:58 akosiaris: stopped slapd on serpens, run sudo -u openldap slapindex to add the new indices, started slapd
* 08:15 ori: Memory utilization on the job runners began to climb around 2015-12-08T10:00:00+00:00 UTC. To investigate, enabled jemalloc profiling and restarted HHVM on mw1001.
* 07:40 YuviPanda: restarted create-dbusers on labstore1001
* 04:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec  9 04:44:57 UTC 2015 (duration 1h 50m 43s)
* 02:54 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 04m 34s)
* 02:34 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 27s)
* 02:23 logmsgbot: ori@tin Synchronized wmf-config/wikitech.php: I4ef826af47: Increase LDAP log verbosity (duration: 00m 44s)
* 02:16 logmsgbot: catrope@tin Finished scap: Graph cherry-picks for wmf.8, including i18n changes (duration: 26m 38s)
* 01:49 logmsgbot: catrope@tin Started scap: Graph cherry-picks for wmf.8, including i18n changes
* 01:45 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: SWAT: bump MediaWikiGadgetsDefinitionRepo cache version (duration: 00m 29s)
* 01:19 awight: Updating paymentswiki from 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a to 74143e43eb36f93be7881f626443182d3bb58cef
* 00:36 ejegg: updated paymentswiki from ff7a706219e72f7362f4e06386e6732933a40478 to 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a
 
== 2015-12-08 ==
* 23:47 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Train: group0 to 1.27.0-wmf.8
* 23:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend: I7b86a521: Ensure the parser cache varies on images disabled and 'light' images (duration: 00m 32s)
* 23:41 logmsgbot: thcipriani@tin Finished scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II (duration: 56m 13s)
* 22:45 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II
* 22:38 eileen: Updating civicrm from d385e34c775aec50a5c17357c8a11cb0244123ce to 291816c88ab07bb914d56c70ff3d58eb1aa0124d
* 22:37 eileen: Updating civicrm from xyz to abc
* 22:24 logmsgbot: thcipriani@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.d9wYd5Axk0" ' returned non-zero exit status 1 (duration: 01m 01s)
* 22:23 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache
* 22:16 yurik: deployed latest kartotherian & tilerator
* 21:09 thcipriani: checking out wmf/1.27.0-wmf.8 on tin for train deploy
* 20:51 moritzm: moved dns aliases ldap-eqiad.wikimedia.org and ldap-codfw.wikimedia.org to seaborgium/serpens (these are for compat reasons, using ldap-labs.[eqiad|codfw].wikimedia.org is preferred
* 20:50 gwicke: restbase: finished deploy of 49fdc615b to all nodes
* 20:37 gwicke: restbase: start deploy of 49fdc615b to all nodes
* 20:35 hashar: Jenkins: changing LDAP config from ldaps://ldap-eqiad.wikimedia.org:636 to ldaps://ldap-labs.eqiad.wikimedia.org:636
* 20:34 gwicke: restbase: canary deploy of 49fdc615b to restbase1001
* 20:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8a1fb69724b: Vary HTML output by NetSpeed designation (duration: 00m 28s)
* 19:52 akosiaris: fixed ldap logging on seaborgium/serpens
* 19:51 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Enable ldap log channel (duration: 00m 28s)
* 19:50 cmjohnson1: swapping failed disk db1019
* 18:53 akosiaris: running puppet on serpens/seaborgium
* 18:49 paravoid: enable ldaps:/// on seaborgium
* 18:49 akosiaris: enable ldaps:/// on serpens
* 18:36 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: Point to new ldap servers (duration: 00m 27s)
* 18:25 andrewbogott: running puppet on labcontrol1002
* 18:23 andrewbogott: running puppet on labcontrol1001
* 17:35 legoktm: deleted centralauth.localuser row for User:ThanduxoloKennethTwani5092/enwiki because the account doesn't exist (T120655)
* 17:13 moritzm: stopping slapd on seaborgium/serpens to refresh LDAP databases with latest export from nembus
* 17:00 andrewbogott: changing opendj writability-mode to ‘internal-only’ on nembus and neptunium
* 16:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT: Update CentralNotice [[gerrit:257615]] (duration: 00m 28s)
* 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in namespaces configuration [[gerrit:257487]] (duration: 00m 28s)
* 16:13 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Wikidata: set maxSerializedEntitySize to 2500 [[gerrit:257318]] (duration: 00m 27s)
* 16:07 godog: repool restbase1008
* 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: CX: Use ContentTranslationRESTBase [[gerrit:255102]] (duration: 00m 30s)
* 16:01 mobrovac: restbase end deploy of 28bf071
* 15:22 mobrovac: restbase enable puppet in prod
* 15:17 mobrovac: restbase start deploy of 28bf071
* 15:10 mobrovac: restbase canary deploy of 28bf071 onto restbase1001
* 14:19 bblack: upgrading openssl package on caches
* 14:16 mobrovac: restbase canary deploy of f47405a onto restbase1001
* 14:09 mobrovac: restbase running puppet on rb1001
* 11:02 godog: upgrade diamond to 3.5-5 in codfw
* 10:04 godog: start cassandra on restbase1008, bootstrapping
* 09:56 hashar: gallium: upgraded python-diamond and java. Restarting Jenkins
* 09:48 hashar: gallium: removing bunch of :i386 packages which were used for Android build but got removed ( https://gerrit.wikimedia.org/r/#/c/183790/3/modules/androidsdk/manifests/dependencies.pp,unified )
* 09:15 godog: reimage restbase1008
* 05:45 gwicke: restbase1008: disable cassandra with `systemctl mask cassandra`
* 04:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec  8 04:00:02 UTC 2015 (duration 1h 34m 44s)
* 03:12 gwicke: Keeping puppet on restbase production cluster disabled until Marko & Filippo can deploy tomorrow morning. PLEASE DO NOT ENABLE!
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 56s)
* 01:16 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Also blacklist Minerva for WPB (duration: 00m 28s)
* 00:59 gwicke: disabling puppet in production restbase cluster in preparation for testing https://gerrit.wikimedia.org/r/#/c/257408/ in staging
* 00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend/: SWAT (duration: 00m 28s)
* 00:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MwEmbedSupport/: SWAT (duration: 00m 28s)
* 00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/oojs-ui-local.js: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 28s)
* 00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/ResourcesOOUI.php: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 27s)
* 00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable banners on mobile web beta on enwiki; remove MobileFrontend Browse config (duration: 00m 27s)
* 00:31 logmsgbot: catrope@tin Synchronized wmf-config/squid.php: SWAT: stop using separate address for upload in wgHTCPRouting (duration: 00m 29s)
* 00:07 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in beta feature on bswiki and urwiki (duration: 00m 28s)
 
== 2015-12-07 ==
* 22:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8c94425693: Improve disableImages cookie code (T120151) (duration: 00m 30s)
* 22:22 logmsgbot: ebernhardson@tin Synchronized portals/prod: Deploy updated portals (duration: 00m 27s)
* 22:01 mdholloway: mobileapps deployed 984f14b
* 21:31 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata/: (no message) (duration: 00m 37s)
* 21:15 subbu: finished deploying parsoid sha 4a7df427 (cf0b9ef + cherry-pick of d65debd)
* 21:09 subbu: restart parsoid on wtp1002 as a canary
* 21:05 subbu: starting parsoid deploy
* 20:33 andrewbogott: enabling puppet on labcontrol1002, testing over
* 20:07 andrewbogott: disabling puppet on labcontrol1002 for an ldap/pdns test
* 19:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1015 at 100% load (duration: 00m 27s)
* 19:52 andrewbogott: restarting nova-compute on all labvirt1X nodes
* 19:20 andrewbogott: restarted pdns on labcontrol1001 and 1002 to catch up with recent opendj restarts
* 19:11 andrewbogott: restarting ldap on neptunium and nembus due to replication warnings in the logs
* 18:36 jynus: es1019 and its management interface became unresponsive
* 18:16 mutante: deleting pmtpa host groups from nagios/icinga
* 17:04 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 10% [[gerrit:254045]] (duration: 00m 29s)
* 17:02 thcipriani: reset uncommited change to includes/jobqueue/JobRunner.php in /srv/mediawiki-staging/php-1.27.0-wmf.7
* 17:00 hashar: Restarted Jenkins to start the ZeroMQ publisher https://phabricator.wikimedia.org/T120668
* 16:56 _joe_: upgrading pybal on primaries in esams, ulsfo, codfw
* 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for en.wiktionary [[gerrit:254476]] and Enable subpages on custom aliases from 112 to 119 [[gerrit:254475]] (duration: 00m 27s)
* 16:43 _joe_: installing pybal on eqiad backups
* 16:43 jynus: restarting mysql, upgrading and rebooting mysql on es1019
* 16:36 _joe_: installing the new pybal version on the backup lvss in esams and ulsfo
* 16:31 _joe_: upgrading pybal on all the backup LVSs in codfw
* 16:28 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia [[gerrit:254486]] (duration: 00m 29s)
* 16:28 _joe_: upgrading pybal on lvs1007-12
* 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia [[gerrit:254665]] (duration: 00m 30s)
* 16:18 _joe_: uploaded pybal 1.13.2 to reprepro
* 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki [[gerrit:255954]] (duration: 00m 28s)
* 15:39 moritzm: uploaded openssl 1.0.2e-1~wmf1 to carbon
* 15:37 yurik: deployed kartotherian
* 15:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s)
* 14:56 yurik: deployed latest tilerator
* 14:46 _joe_: also restarted pybal on lvs3003
* 14:42 _joe_: restarted pybal on lvs1006
* 14:39 hashar: restarting Jenkins
* 14:32 hashar: Jenkins lost a bunch of executors :/
* 14:30 hashar: CI / Zuul stalled somehow
* 12:56 jynus: rolling restart, configuration upgrade of es1015
* 12:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s)
* 11:08 YuviPanda: restarting pdns on holmium
* 10:52 jynus: database and system maintenance to es1017
* 10:43 hashar: CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs
* 10:20 YuviPanda: restarted nova-conductor and scheduler on labcontrol1001
* 10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s)
* 10:05 hashar: stopped Nodepool.  Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 )
* 09:46 hashar: restarting Nodepool on labnodepool1001.eqiad.wment
* 09:40 hashar: CI / Zuul stalled. Nodepool can no more spawn instances :-/
* 09:27 godog: nodetool decommission restbase1008
* 09:13 jynus: es1013 maintenance (mysql restart, upgrade, possible reboot)
* 08:27 _joe_: uploaded etcd 2.2 package from stretch to jessie-wikimedia
* 03:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec  7 03:56:49 UTC 2015 (duration 1h 32m 22s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)
 
== 2015-12-06 ==
* 21:48 ori: krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting.
* 18:49 legoktm: reset auth token for User:QuimGil
* 05:50 mutante: silver gzip /var/log/nutcracker.log.1
* 05:40 mutante: silver: apt-get clean for disk space
* 03:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec  6 03:57:02 UTC 2015 (duration 1h 31m 41s)
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s)
 
== 2015-12-05 ==
* 18:30 gwicke: started nodetool decommission on restbase1008
* 11:35 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s)
* 11:23 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.5
* 11:22 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s)
* 11:13 logmsgbot: reedy@tin Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s)
* 08:59 bblack: offlined db1019 megacli disk 32:11
* 06:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec  5 06:09:07 UTC 2015 (duration 3h 44m 18s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)
 
== 2015-12-04 ==
* 22:20 ejegg: updated dash from bed9895563fc0f5d3b96791fae6ed104b7778f0d to 9f0a4d98586f1281716e39b34d924de524458177
* 21:44 andrewbogott: disabling puppet on labcontrol1002 for ldap testing
* 21:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/Hooks.php: Iba0138a: Don't install a custom error handler for hooks (T117553) (duration: 00m 28s)
* 21:26 awight: Update fundraising CRM from 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a to f0ef0dab1fa7a85426a000c35750093d541fe4f5
* 20:28 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: Idee6a1980: job queue: use instances on port 6378 as aggregators (duration: 00m 30s)
* 19:21 ori: krypton: updated Grafana to 2.6.0-beta1 for bug fix for issue 3422
* 15:52 Jeff_Green: add mx record for donate.wikimedia.org
* 15:33 godog: ms-be2019 rebooted by itself, ilo event log shows "Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000038, Bank 0x00000003, Status 0xFE000040'00020135, Address 0x00000000'FEB82F63, Misc 0x00000000'00002285)"
* 08:52 godog: reimage restbase1009
* 05:59 gwicke: ran systemctl mask cassandra on restbase1009; it is important that this node does not start up.
* 05:53 gwicke: moved /var/lib/cassandra out of the way in an attempt to stop puppet restarting cassandra on decommissioned restbase1009
* 05:49 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec  4 05:49:46 UTC 2015 (duration 3h 21m 36s)
* 02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 19s)
* 02:15 ori: CirrusSearch-common.php sync was for I826d000ca: Turn off backoff throttling of CirrusSearch jobs
* 02:15 logmsgbot: ori@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 29s)
* 01:33 bd808: Updated scholarships.wikimedia.org to af73bf6
* 00:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT (duration: 00m 32s)
* 00:01 ejegg: pointed SmashPig IPN listener to live Adyen account
 
== 2015-12-03 ==
* 23:09 bblack: restarting pybal (w/ BGP enabled) on lvs100[123] (newly-installed w/ jessie)
* 22:59 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/jobqueue/JobRunner.php: temporarily disable job throttling (duration: 00m 29s)
* 22:35 cwd: updated paymentswiki configuration [redux]
* 22:15 cwd: updated paymentswiki configuration
* 22:09 bd808: Removed zirconium.wikimedia.org from Trebuchet minions list for scholarships/scholarships
* 22:04 bd808: Updated scholarships.wikimedia.org to cb94319 plus local i18n filtering
* 21:48 Reedy: finished removing bogus msg_resource rows
* 21:28 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: re-sync (re-merged the change) (duration: 00m 29s)
* 21:27 bd808: Applied database migrations and purged last year's data from Wikimania Scholarships db
* 21:21 ottomata: restarted eventlogging with 4 mysql consumer processes running in parallel
* 21:21 bblack: rebooting lvs100[123] for reinstall to jessie
* 21:18 Reedy: Cleaning up msg_resource rows with bogus language codes
* 21:15 gwicke: stopped cassandra on 1009 as it's decommissioned & will be reimaged
* 21:13 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Re-fix the jobqueue on wikitech after redis cleanup (duration: 00m 26s)
* 20:55 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 47s)
* 20:45 _joe_: opening connection from mw1001 to silver, mysql
* 20:29 ori: on palladium: salt -G 'cluster:jobrunner' cmd.run 'service jobrunner status | grep running && service jobrunner restart'  ;  salt -G 'cluster:jobrunner' cmd.run 'service jobchron status | grep running && service jobchron restart'
* 20:28 ori: ran srem jobqueue:aggregator:s-wikis:v2 labswiki on rdb1001 aggr
* 19:41 bblack: disabling pybal on lvs100[123] over the next few minutes (for reinstall to jessie later after confirmation everything is still ok on [456])
* 19:10 jynus: restarting eventlogging_sync on db1047 and dbstore1002
* 19:04 jynus: starting m4 slave again on dbstore2002
* 18:45 andrewbogott: disabling puppet on labcontrol1002 to test openldap with pdns
* 18:33 mutante: neon - remove icinga user from "dialout" group
* 18:27 jynus: disabling eventlogging_sync process on dbstore1002 and db1047 and replication on the other m4 slaves
* 18:18 jynus: disabling event scheduler on db1046 (m4-master)
* 17:26 ejegg: updated fundraising dashboard from 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c to bed9895563fc0f5d3b96791fae6ed104b7778f0d
* 17:03 logmsgbot: kartik@tin Finished scap: Update ContentTranslation (duration: 05m 52s)
* 16:57 logmsgbot: kartik@tin Started scap: Update ContentTranslation
* 16:50 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 28s)
* 15:23 andrewbogott: stopping pdns on labcontrol2001
* 15:11 moritzm: restarting cassandra on restbase100[56] (subsequently) to effect openjdk security update
* 14:57 mobrovac: restbase end of deployment of 262da91a
* 14:48 mobrovac: restbase start deployment of 262da91a
* 14:06 moritzm: installed dpkg updates across the cluster
* 11:35 moritzm: restarting cassandra on aqs cluster (subsequently) to effect openjdk security update
* 10:51 jynus: restarting, upgrading and general maintenance for es1013 (depooled)
* 10:36 _joe_: imported dh-python into precise/universe from the ubuntu cloud archive
* 10:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1013 for maintenance (duration: 00m 30s)
* 05:50 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec  3 05:50:08 UTC 2015 (duration 50m 7s)
* 02:32 ejegg: updated CiviCRM from d5afa10383cfd41adbe798c3ed2aa97839be8bc2 to 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 54s)
* 00:18 awight: Deploy paymentswiki config for T112181
 
== 2015-12-02 ==
* 23:47 ejegg: updated fundraising dashboard from 01a82b5e4909075e3190496e7fcda2b60f8d5d3c to 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c
* 23:07 cwd: updated payments from 02a3771cae6e4e74d9d754a8b075de20aa171adc to ff7a706219e72f7362f4e06386e6732933a40478
* 22:09 jynus: unscheduled restart of dbstore1002 (analytics-slave)
* 21:44 jynus: disabling all alert notifications for dbstore1002
* 21:30 bblack: rebooting lvs1007 for interface config test (not active, no BGP)
* 20:37 Jeff_Green: enable mail queue monitoring for fundraising
* 19:13 jynus: restarting mysql on db2067 to test a configuration change
* 18:07 mutante: mw1136 - hhvm restart
* 14:45 Coren: deploying cleanup of labs PAM configuration - this should be a functional noop but may cause some puppet noise
* 12:41 akosiaris: restart cassandra on maps-test200{1,2,3,4}.codfw.wmnet
* 11:23 moritzm: restarting cassandra on restbase100[78] (subsequently) (to effect openjdk security updates plus related libs)
* 11:06 moritzm: restarting cassandra on restbase100[2-4] (subsequently) (to effect openjdk security updates plus related libs)
* 11:04 moritzm: restart cassandra on restbase1001 (to effect openjdk security updates plus related libs)
* 10:21 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 27s)
* 10:20 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 30s)
* 10:14 _joe_: clearing the job cache for ocg1003
* 09.43 +
* 09:22 _joe_: stopped ocg service on ocg1003
* 08:38 _joe_: planet1001 recovered as soon as I did get into console via gnt-instance
* 06:25 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/NavigationTiming: Idb675cdce: Add isHiDPI and isHttp2 properties; drop isHttps (T119014) (duration: 00m 50s)
* 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec  2 05:53:21 UTC 2015 (duration 53m 20s)
* 02:28 mutante|away: labcontrol2001 - disable puppet, kill from puppet stored configs
* 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 47s)
* 00:29 mutante: puppetstoredconfigclean.rb labcontrol2001.wikimedia.org  fixes icinga config
 
== 2015-12-01 ==
* 22:41 mutante: stop/start ntp service on lvs2002
* 21:04 ejegg: updated fundraising from 2ee15180150c8b685b14abdbf7c6769736fb49a0 to 01a82b5e4909075e3190496e7fcda2b60f8d5d3c
* 20:54 mutante: re-enabled puppet on cache::misc, replaced planet cert
* 20:19 mutante: temp. disabling puppet on cache::misc
* 18:14 twentyafterfour: restarted apache on iridium to deploy https://phabricator.wikimedia.org/rPHEXd724c51a4144f7de548546875097da57ee1b38d7
* 17:05 moritzm: restarting cassandra instances on restbase200[3-6] (to effect openjdk security updates and a few depending libs)
* 17:00 moritzm: restarting cassandra instances on restbase2002 (to effect openjdk security updates and a few depending libs)
* 16:55 moritzm: restarting cassandra instances on restbase2001 (to effect openjdk security updates and a few depending libs)
* 16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all accounts on eswiki. Part II [[gerrit:250474]] (duration: 00m 28s)
* 16:21 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for all accounts on eswiki. Part I [[gerrit:250474]] (duration: 01m 29s)
* 15:30 ejegg: changed donation qc time limit from 90 to 100 sec
* 12:59 godog: reimage mw2208
* 10:50 moritzm: restarting on cassandra instances on restbase1008 (to effect openjdk security updates and a few depending libs)
* 06:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec  1 06:52:40 UTC 2015 (duration 52m 39s)
* 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 14s)
* 02:18 mutante: restarted hhvm on mw1147
* 01:47 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 30s)
* 00:51 ejegg: updated fundraising dash from bc0ec4bb2427ebeb2d4cd60eda65e3d82d006937 to 2ee15180150c8b685b14abdbf7c6769736fb49a0
* 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor: SWAT (duration: 00m 29s)
* 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: SWAT (duration: 00m 30s)
* 00:45 ejegg: updated Adyen credentials in SmashPig IPN listener
* 00:39 ejegg: updated Adyen credentials on payments-wiki
* 00:35 ejegg: updated payments-wikifrom 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 02a3771cae6e4e74d9d754a8b075de20aa171adc
 
== 2015-11-30 ==
* 22:53 ejegg: updated fundraising dash from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
* 22:37 ejegg: updated fundraising dashboard from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
* 22:16 awight: Configuring paymentswiki to not exlicitly include DefaultSettings.php
* 21:55 mutante: re-wrote l10nupdate cron; restarted cron service on tin
* 21:53 awight: Tweak paymentswiki-staging config: remove line that exlicitly includes DefaultSettings.php
* 20:05 apergos: re-enabled puppet on neodymium, minion testing concluded for now
* 19:47 gwicke: running `nodetool decommission` on restbase1009 in preparation for the conversion to the multi-instance setup, per https://phabricator.wikimedia.org/T95253#
* 19:31 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm deprecated/unused rate limit log config (duration: 00m 28s)
* 17:27 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/: need maint script errywhere (duration: 00m 28s)
* 16:51 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/ContentTranslation/modules/draft/ext.cx.draft.js: SWAT: Add some extra information to save failure logging [[gerrit:255956]] (duration: 00m 28s)
* 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable QuickSurveys reader segmentation survey [[gerrit:255448]] (duration: 00m 28s)
* 16:30 paravoid: mw1002 service hhvm restart
* 16:17 paravoid: rolling back to kernel 3.19 on lvs2001/2/3
* 15:29 paravoid: stopping pybal on lvs2001/2/3
* 15:21 paravoid: switching lvs2004/5/6 traffic back to lvs2001/2/3
* 15:13 paravoid: switching lvs2001/2/3 traffic to lvs2004/5/6 and upgrading kernels
* 15:12 _joe_: restarting HHVM on mw1147 too, same reason as mw1114
* 15:10 _joe_: restarting hhvm on mw1114, stuck in __pthread_cond_wait () [folly::EventBase::runInEventBaseThreadAndWait ()], apparently blocked in writing to stdout
* 15:02 paravoid: switching traffic from lvs4002 to lvs4004; upgrading lvs4002's kernel
* 15:02 paravoid: switching traffic back to lvs4001
* 14:57 paravoid: switching traffic from lvs4001 to lvs4003; upgrading lvs4001's kernel
* 14:45 paravoid: switching traffic from lvs3001 to lvs3003; upgrading lvs3001's kernel
* 14:38 paravoid: switching traffic back to lvs3002
* 14:31 paravoid: switching traffic from lvs3002 to lvs3004; upgrading lvs3002's kernel
* 14:07 bblack: upgrading varnishkafka package on all caches
* 13:52 bblack: updating varnishkafka on cp1065
* 11:03 godog: upgrade python-statsd to 3.0.1 in eqiad
* 10:59 godog: upgrade python-statsd to 3.0.1 in codfw
* 10:15 godog: reenable puppet on graphite1001
* 10:10 paravoid: re-enabling OSPF over cr2-eqiad:xe-5/2/2 <-> cr1-ulsfo:xe-0/0/3.538
* 10:09 paravoid: re-enabling cr2-eqiad:xe-5/2/0 and xe-5/2/1
* 10:01 jynus: performing schema change on db1046 (analytics master)
* 09:32 jynus: removing old snapshots from db1046
* 06:39 ori: Restarted statsv on hafnium
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:56 gwicke: started `nodetool cleanup` on restbase1002 to get rid of unnecessary data from earlier 1001 decommission attempt
* 01:05 logmsgbot: bd808@tin sync-l10n completed (1.27.0-wmf.7) (duration: 01m 19s)
* 01:04 bd808: testing l10n cache rebuild as l10nupdate user (take 2)
* 00:57 Krenair: test
 
== 2015-11-29 ==
* 21:25 jynus: importing user.user_touched (s7) from dbstore1002 to sanitarium. s7 lag on labs replicas will be higher for some minutes.
* 20:51 jynus: importing user.user_touched (s6) from dbstore1002 to sanitarium. s6 lag on labs replicas will be higher for some minutes.
* 20:28 jynus: importing user.user_touched (s5) from dbstore1002 to sanitarium. s5 lag on labs replicas will be higher for some minutes.
* 19:51 jynus: importing user.user_touched (s4) from dbstore1002 to sanitarium. s4 lab will be affected for some minutes.
* 04:50 gwicke: restarted cassandra on restbase1009 to avoid it running out of disk space; had large compaction (~2TB) at 80% and only 64G disk space left
* 03:01 YuviPanda: run chown -R l10nupdate: /var/lib/l10nupdate/mediawiki  for Reedy on tin
* 02:28 Reedy: l10nupdate failed because some git objects owned by 997:l10nupdate
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-28 ==
* 22:48 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/cache/l10n: bd808 testing l10nupdate sync-dir using stale branch (duration: 01m 29s)
* 20:49 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
* 20:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 07m 11s)
* 20:35 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie33ae3b6a: Increase $wgCopyUploadTimeout to 90 seconds (from default 25) (T118887) (duration: 00m 27s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-27 ==
* 21:23 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/255726/1 (duration: 00m 31s)
* 17:46 paravoid: re-enabling cr2-eqiad:xe-5/2/2 (link to cr1-ulsfo) and xe-5/2/3 (link to cr2-codfw)
* 15:08 godog: kill stray cpu hog xelatex on ocg1003, orphan and started on oct22
* 14:16 Jeff_Green: fundraising database maintenance
* 12:36 godog: ban some grafana dashboards from graphite
* 10:56 apergos: restarted gitblit on antimony about 40 minutes ago, it takes about 20 minutes to load up caches of commits, now responding but... it's taking 17-18 seconds to load repos instead of 1-3 as normal. so timeouts
* 10:47 jynus: importing user.user_touched (s3) from dbstore1002 to sanitarium, it will affect s3 labsdbs lag
* 10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repooling previouly problematic servers (duration: 00m 28s)
* 10:10 apergos: removed gitblit.log.1 on antimony, it was 58GB
* 10:03 mark: Reenabling cr2-eqiad:xe-5/3/0
* 09:53 mark: Disabled cr2-eqiad:xe-5/2/*
* 09:53 paravoid: issue has been identified as a a problematic PIC (5/2) on cr2-eqiad and is being worked around
* 09:51 mark: Set VRRP priority 50 on all cr2-eqiad VRRP sessions, making it backup
* 09:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling problematic servers (duration: 00m 27s)
* 09:48 mark: Disabling cr2-eqiad:xe-5/2/1
* 09:45 mark: Disabling cr2-eqiad:xe-5/{2,3}/0
* 09:42 paravoid: network troubles within eqiad still being investigated
* 09:42 paravoid: network troubles; ulsfo was depooled ~20 minutes ago due to backhauling issues
* 08:31 _joe_: rolling restart of all redis instances on rdb1001 and 1003 (later 1002/4) to fix the upstart bug
* 07:55 _joe_: restarting redis rdb1003:6380 to test rdb disabling
* 06:57 ori: ran eventloggingctl stop / eventloggingctl start on eventlog1001
* 06:18 ori: coal metrics flat since 01:25:00 AM UTC
* 02:47 Reedy: manually running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
* 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
* 02:45 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 09m 10s)
* 02:14 Reedy: killed pre 1.27 l10n cache files from tin:/var/lib/l10nupdate/caches
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-26 ==
* 20:59 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust connection timeout and maxPartitionsTry (duration: 00m 34s)
* 16:50 jynus: backfilling user.user_touched from dbstore1002 to sanitarium on all wikis
* 16:30 paravoid: switch lvs3001 traffic to lvs3003dd
* 16:22 paravoid: rebooting lvs3003 with Linux 4.3-1~wmf1
* 15:58 Krinkle: Deployed patch for T119707
* 15:49 godog: reenable puppet on graphite1001
* 14:30 _joe_: run /usr/local/bin/foreachwiki updateSpecialPages.php manually in a screen on terbium
* 13:27 moritzm: restarted opendj on nembus for export of LDAP data
* 13:25 moritzm: stopping opendj on nembus for export of LDAP data
* 11:54 apergos: disabling puppet on neodymium for salt minon tracing
* 11:35 apergos: re-enabled puppet on neodymium, config testing done for now
* 11:32 godog: drop local_group_globaldomain_T_mathoid_data and local_group_globaldomain_T_mathoid_request CF stats from graphite
* 11:25 jynus: manually executing some updateSpecial pages script to try to debug its problems
* 11:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1053 after maintenance (duration: 00m 28s)
* 11:13 apergos: disabled puppet on neodymium for salt master testing
* 09:49 moritzm: restarted cassandra nodes on the restbase staging cluster to pick up openjdk security updates
* 09:38 jynus_: upgrading/rebooting/restarting mysql on db1053 (depooled)
* 09:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1053 for maintenance (duration: 00m 27s)
* 07:15 ori: rdb100x migrated to redis::instance, meaning they each have two additional redis instances, on port 6380 and 6381, accordingly.
* 06:46 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I3e6021ed3: Update job queue config to use 2 new rdb1003 instances (duration: 00m 28s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:31 awight: updating paymentswiki-staging from 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 988ba71418c370e0365a102a0b87f823671ea1e3
* 01:03 awight: from f867beecd3a4fb596e2d08c8d2b0e4173ad0b356 to d5afa10383cfd41adbe798c3ed2aa97839be8bc2
* 00:14 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I1e0d0ed8215f: pool rdb1007:6381 (duration: 00m 30s)
 
== 2015-11-25 ==
* 23:21 logmsgbot: krenair@tin Synchronized private/README_BEFORE_MODIFYING_ANYTHING: 334ca105e92aaf7046e244ff39189f3823d31a7d (duration: 00m 32s)
* 22:14 logmsgbot: demon@tin Finished scap: new MW release, swapping extdist config + msgs (duration: 23m 59s)
* 21:50 logmsgbot: demon@tin Started scap: new MW release, swapping extdist config + msgs
* 20:31 gwicke: running `nodetool cleanup` on restbase1007 to make sure that we don't have extra sstables from the 1001 decommision taking up space
* 19:13 mobrovac: restbase deploy end of 74662c
* 19:13 moritzm: installed django security updates on stat* and graphite hosts
* 18:47 robh: removing cr2-ulsfo:xe-1/2/0, Patch ID 1062 as T118171 cancels that link
* 18:46 mobrovac: restbase deploy start of 74662c
* 18:19 logmsgbot: bd808@tin Synchronized README: Testing l10nupdate uid fix for T119165 (duration: 00m 28s)
* 18:04 andrewbogott: restored pdns-recursor on holmium, again
* 18:00 mobrovac: restbase canary deploy to restbase1001 of 74662c6
* 17:44 andrewbogott: killing pdns-recursor on holmium
* 16:53 andrewbogott: killing pdns-recursor on holmium
* 16:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 47s)
* 15:53 godog: ban grafana kafka dashboard temporarily from graphite
* 14:57 godog: bounce uwsgi on graphite1001
* 14:52 godog: stop puppet on restbase1* / restbase2* before https://gerrit.wikimedia.org/r/#/c/254372/
* 13:09 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.7/Rakefile: Added Rakefile https://gerrit.wikimedia.org/r/#/c/254423/ (duration: 00m 28s)
* 11:19 jynus: applying ferm and p_s to db1044 (depooled)
* 10:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1044 for a brief maintenance (duration: 00m 28s)
* 08:08 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s)
* 08:06 akosiaris: update puppet compilers fact database
* 03:44 Tim: on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639"
* 03:35 logmsgbot: krinkle@tin Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s)
* 03:12 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s)
* 02:40 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:31 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s)
* 01:03 mutante: re-armed keyholders on mira
* 00:46 mutante: starting redis on rdb1007 (it died and was status: dead)
* 00:40 mutante: tin: fixing l10nupdate UID (997->10002), file ownership
* 00:37 ori: Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart.
* 00:21 ejegg: updated payments-wiki from 5358960d6857f589939c9b86ea37af168a4f8b0b to 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe
* 00:20 mutante: rebooting mira
* 00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s)
* 00:13 awight: Disabled Amazon on donatewiki
* 00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/254443/ (duration: 00m 28s)
 
== 2015-11-24 ==
* 23:43 mutante: fixinx l10nupdate UID / file ownership on mira
* 23:40 csteipp: deployed patch for T119309
* 23:36 ostriches: phabricator: restarted daemons, were complaining about out of sync config
* 22:07 ejegg: updated SmashPig from 6243d4f172dbfa5e3820f2e1e3a3165efff58bac to b3ee85aab38fdeef3f045ba1733f582e9a5006b2
* 21:39 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I80d38773057 (duration: 00m 28s)
* 20:31 mutante: gnt-instance reboot seaborgium.wikimedia.org
* 20:01 bblack: disabling puppet on neon to avoid monitoring race-condition spam
* 19:30 moritzm: restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
* 19:27 bblack: misc-cluster: all 4x nodes on new config and pooled
* 19:18 bblack: misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config)
* 18:52 bblack: repooled cp1056, depooled cp1057 (misc 2layer)
* 18:38 YuviPanda: restarted apache2 on etherpad
* 18:30 bblack: depooling cp1056 for misc-cluster 2layer work...
* 17:59 godog: nodetool cleanup on restbase1007
* 17:20 godog: stop decommissioning restbase1001, bounce cassandra
* 16:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c [[gerrit:255032]] (duration: 00m 28s)
* 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki [[gerrit:255122]] (duration: 00m 27s)
* 16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign [[gerrit:255069]] (duration: 00m 27s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey [[gerrit:254908]] (duration: 00m 30s)
* 16:08 paravoid: starting pybal on lvs3001 again, testing is over
* 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki [[gerrit:250473]] (duration: 00m 46s)
* 15:38 moritzm: restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
* 15:33 hashar: enabled puppet on both gallium and scandium
* 15:27 hashar: disabling puppet on both gallium and scandium
* 14:24 godog: upgrade diamond to 3.5-4 on precise hosts in eqiad
* 14:22 godog: upgrade diamond to 3.5-4 on precise hosts in ulsfo
* 13:56 godog: upgrade diamond to 3.5-4 on precise hosts in esams
* 12:18 paravoid: stopping pybal on lvs3001, switching its traffic to lvs3003
* 11:48 paravoid: rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1
* 10:28 jynus: dropping user_daily_contribs view from labsdb hosts
* 10:07 godog: stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies
* 08:24 jynus: dbstore1002 (analytics-slave) mysql is not responding, forcing restart
* 05:58 _joe_: powercycling ms-be1010, xfs kernel lockups
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s)
* 02:12 ejegg: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to 6243d4f172dbfa5e3820f2e1e3a3165efff58bac
* 01:00 ejegg: updated payments-wiki from ecbc49c63b259a4363c51491ddff273e3579f0e9 to 5358960d6857f589939c9b86ea37af168a4f8b0b
* 00:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s)
* 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s)
* 00:15 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable wikidatawiki for elasticsearch labs replica (duration: 00m 27s)
 
== 2015-11-23 ==
* 23:55 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 28s)
* 23:30 awight: campaigns reenabled
* 23:29 awight: reenabling paymentswiki
* 23:28 awight: reenabling fundraising-jenkins jobs
* 23:08 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable elasticsearch labs replica for enwiki and dewiki (duration: 00m 28s)
* 23:04 awight: shutting down Jenkins
* 22:52 awight: disable paymentswiki in config
* 21:16 hoo: Updated Wikidata's property suggester with data from today's json dump
* 21:15 jynus: enforcing limits on running queries on all labsdb databases
* 20:47 jynus: restarting labsdb1002, hopfully for the last time
* 17:54 jynus: temporarily stopping labsdb1002 to solve replication issues
* 17:32 Coren: emergency hotfix on labstore1001 (apt-get remove nfsd-ldap) as it appears to not work in all situations.
* 17:30 ori: restarting hhvm on mw1122 after lock-up; backtrace saved as /tmp/hhvm.32016.bt.
* 17:28 logmsgbot: thcipriani@tin Finished scap: SWAT: Update CentralNotice [[gerrit:254865]] (duration: 26m 31s)
* 17:02 logmsgbot: thcipriani@tin Started scap: SWAT: Update CentralNotice [[gerrit:254865]]
* 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: noindex user namespace on en.wikipedia.org [[gerrit:237330]] (duration: 00m 28s)
* 16:43 jynus: restarting mysqld on labsdb1003 again
* 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: End first mobile QuickSurveys campaign [[gerrit:254831]] (duration: 00m 28s)
* 16:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/includes/specials/SpecialWatchlist.php: SWAT: Special:Watchlist: Add user preference to "Show last" options, fix float comparison [[gerrit:254880]] (duration: 00m 29s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable article-recommender-1 campaign as default [[gerrit:254117]] (duration: 00m 52s)
* 15:56 paravoid: rebooted lvs3003 back with (an updated) 3.19
* 15:53 _joe_: live-resizing /var/lib/wdqs on wdqs1002
* 15:15 Coren: NFS server on labstore1001 restarted with no issues.
* 15:11 Coren: Restarting NFS daemon on labstore1001 w/ the new LDAP shim.
* 14:53 paravoid: powercycling cp1053, unresponsive, blank console
* 14:53 bblack: depooling cp1053
* 14:50 ottomata: removing erbium udp2log varnishncsa instances from all caches
* 14:38 paravoid: rebooting lvs3003 with new kernel
* 13:18 jynus: restarting mysql on labsdb1003, unresponsive
* 11:05 _joe_: stopping apache on mw2051 for testing
* 08:58 godog: restbase1001 nodetool decommission
* 08:57 mobrovac: mathoid deploying ff03d56
* 08:12 moritzm: restarted gitblit on antimony
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 57s)
* 01:27 Jamesofur: end enwiki test election from cli to avoid confusion with real election
 
== 2015-11-22 ==
* 21:30 Reedy: restarting stuck Jenkins
* 02:16 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 22s)
 
== 2015-11-21 ==
* 19:14 ejegg|away: updated fundraising tools from d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132 to ebed29c0eccf38c812b20a957b3487a15bfa9cbc
* 18:43 ejegg|away: rolled back SmashPig d7534bde8c247a1e877a4bcd768b9963f89666d7 --> 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
* 18:35 ejegg|away: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to d7534bde8c247a1e877a4bcd768b9963f89666d7
* 03:19 ori: SecurePoll arbcomlist.php run for enwiki (requested by Jamesofur) complete.
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 18s)
* 00:50 ori: update-ubuntu-mirror on carbon completed successfully ; going to re-enable icinga-wm
* 00:46 ori: rampant puppet failures appear to be the result of a Hash Sum mismatch error during apt-get update; attempting to fix by running sudo -u mirror /usr/local/sbin/update-ubuntu-mirror on carbon.
* 00:37 ori: disabled puppet on neon for one hour and killed icinga-wm to silence a tide of puppet failure alerts
* 00:24 logmsgbot: demon@tin Synchronized .arcconfig: no-op, for completeness (duration: 00m 31s)
 
== 2015-11-20 ==
* 21:45 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.7/includes/User.php: Deploy fix for T119021 (duration: 00m 28s)
* 18:55 ori: resuming interrupted arbcomlist invocation on terbium
* 16:51 _joe_: decommissioning mw1041: removed facts and certs, removed from conftool and pybal
* 16:44 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Add throttle config for Viquimarató nit digital/2015 (duration: 00m 28s)
* 16:17 _joe_: depooled mw1041
* 15:41 Reedy: running sync-common on mw1041
* 15:32 _joe_: powercycling mw1041
* 15:26 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Update throttle for event tomorrow (duration: 00m 28s)
* 14:35 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 3000
* 10:06 ori: Running SecurePoll's arbcomlist.php on terbium with performance fix from https://gerrit.wikimedia.org/r/#/c/254375/ , per request from Jamesofur. Looks to be behaving well, can be killed if it acts up.
* 07:44 _joe_: rebooted pybal-test2001
* 07:36 _joe_: powercycled mw1041
* 04:55 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: inconsequential sync to test mirroring (duration: 02m 14s)
* 04:41 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: deploying I2ec91aa33e05c2dc367db8a6f6ba56be5397906b to test scap mirroring (duration: 00m 20s)
* 04:36 twentyafterfour: deploying scap on tin
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 10m 47s)
* 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki.widgets/mw.widgets.TitleWidget.js: Fix conflicting configuration name in TitleInputWidget (duration: 00m 19s)
* 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/includes/widget/TitleInputWidget.php: Fix conflicting configuration name in TitleInputWidget (duration: 00m 20s)
* 01:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/lib/jquery.i18n/src/jquery.i18n.language.js: Unbreak IE8 support in jquery.i18n (duration: 00m 25s)
* 01:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Echo/includes/formatters/BasicFormatter.php: unstub wgLang to fix type hint errors (duration: 00m 20s)
* 01:11 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Reduce sampling rate for QuickSurveys survey (28c69d2) (duration: 00m 20s)
* 00:37 awight: Updating fundraising/crm from e486ba424092866cdd8a15d26aeb17a412cd631a to f867beecd3a4fb596e2d08c8d2b0e4173ad0b356
* 00:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
* 00:17 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
* 00:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: SWAT https://gerrit.wikimedia.org/r/254320 and https://gerrit.wikimedia.org/r/254319 (duration: 00m 20s)
* 00:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/254322/ (duration: 00m 20s)
 
== 2015-11-19 ==
* 23:35 awight: Update fundraising/tools from c6315eddf6f495e45401844d5f61bb240464c1e5 to d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132
* 22:31 csteipp: deployed followup for T109724
* 22:21 csteipp: deployed patch for T118682
* 22:13 csteipp: deployed patch for T118032
* 22:11 csteipp: deployed patch for T97897
* 21:31 mutante: magnesium - apt-get upgrade
* 21:13 mutante: racktables moved to krypton and upgraded to 0.20.6
* 20:28 mutante: racktables - in migration - brb
* 20:22 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.7/extensions/QuickSurveys/: Deploy https://gerrit.wikimedia.org/r/#/c/254184/ (duration: 00m 19s)
* 20:19 awight: push paymentswiki config change, T115746
* 20:13 bblack: upgrading pybal to 1.13.1 on lvs400[12] and failing back traffic
* 19:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 22s)
* 19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.7
* 18:53 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata: Adjust watchlist filter for core change (duration: 00m 30s)
* 16:24 godog: repool restbase1002
* 16:19 cmjohnson1: replacing pem 0 cr1-eqiad
* 14:55 godog: roll restart restbase in eqiad
* 14:41 godog: roll-restart restbase on aqs100*
* 14:40 moritzm: uploaded linux-meta 1.3 to carbon
* 14:39 moritzm: uploaded linux 3.19.3-9 to carbon
* 14:14 godog: roll-restart restbase in codfw
* 12:39 bblack: failing over ulsfo traffic to lvs400[34] (for 1.13.1 testing)
* 11:23 hashar: restarted zuul-merger on gallium. We now have two instances running (gallium and scandium)
* 11:21 hashar: stopped zuul-merger on gallium, started the one on scandium
* 11:00 jynus: modifying filters on all labs db replicas (sanitarium), small amounts of lag could be created
* 10:42 godog: hhvm-dump-debug --full and bounce hhvm on mw1147
* 10:40 godog: hhvm-dump-debug --full and bounce hhvm on mw1131
* 10:13 godog: disable puppet on deployment_target:restbase/deploy
* 10:08 hashar: going to restart Jenkins a few times
* 09:53 godog: delete old mobileapps CF stats from graphite1001 / graphite2001
* 06:19 YuviPanda: remove role::labs::instance puppetClass from all labs instances
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 58s)
* 01:31 awight: updating fundraising/crm from 67b855cbce358d215b8025da6fe7031bf3d221d9 to e486ba424092866cdd8a15d26aeb17a412cd631a
* 01:30 twentyafterfour: finished phabricator upgrade
* 00:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Flow/: SWAT: better permissions errors for opt-in (duration: 00m 21s)
* 00:19 RoanKattouw: Ran namespaceDupes.php on eswiktionary
* 00:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T118844, T119006 (duration: 00m 19s)
* 00:12 logmsgbot: catrope@tin Synchronized portals: SWAT (duration: 00m 19s)
 
== 2015-11-18 ==
* 23:51 yurik: git deploy sync latest graphoid
* 23:49 bblack: upgrading pybal to 1.13.1 on lvs300[34]
* 23:45 bblack: upgrading pybal to 1.13.1 on lvs400[34]
* 22:44 awight: Updating fundraising/crm from 230cebb4a3b46a75fef758d46f4db22de29286cb to 67b855cbce358d215b8025da6fe7031bf3d221d9
* 22:38 mutante: powercycled analytics1030 - no ssh response, mgmt console login timed out
* 21:15 subbu: finished deploying parsoid sha e0a4fc91
* 21:10 subbu: synced code; restarted parsoid on wtp1003 as a canary
* 21:06 subbu: starting parsoid deploy
* 20:43 bblack: upgrading pybal to 1.13.1 on lvs200[456]
* 20:09 logmsgbot: maxsem reset User:人神之间 email (confirmed in person with Matanya)
* 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.7
* 18:18 andrewbogott: repooled mw1160.  And that’s all of them!
* 18:09 andrewbogott: repooled mw1159, depooled mw1160
* 17:59 andrewbogott: repooling mw1157, depooled mw1159
* 17:48 andrewbogott: repooled mw1155, depooled 1157 (I did 56 and 58 list night)
* 17:36 mutante: releases.wikimedia.org now enforcing https protocol
* 17:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/includes/db/loadbalancer/LoadBalancer.php: I03e1386b: Make getLaggedSlaveMode() use reuseConnection() as needed (T118162) (duration: 00m 19s)
* 17:26 andrewbogott: repooled mw1154, depooled mw1155
* 17:17 andrewbogott: repooled mw1153, depooling mw1154
* 17:09 andrewbogott: depooled mw1153 for https://phabricator.wikimedia.org/T118888
* 17:07 thcipriani: updateCollation.php for srwiki 1670608 rows processed
* 17:05 godog: restart restbase2002-a cassandra instance, bootstrap failed with stream error
* 16:23 bblack: upgrading pybal on lvs1006 (1.13.1)
* 16:23 _joe_: upgrading pybal on lvs1006 (1.13.1)
* 16:21 bblack: upgrading pybal on lvs1005 (1.13.1)
* 16:21 bblack: upgrading pybal on lvs1004 (1.13.1)
* 16:16 thcipriani: updateCollation.php for srwiktionary complete 82713 rows processed
* 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to "uca-sr" at srwiki and srwiktionary [[gerrit:253903]] (duration: 00m 20s)
* 16:11 bblack: upgrading pybal on lvs1004 (1.13.1)
* 16:07 mobrovac: restbase cassandra dropping old mobileapps keyspaces
* 16:05 mobrovac: restbase end deploy of 6b5a602
* 15:56 mobrovac: restbase start deploy of 6b5a602
* 15:50 bblack: upgrading pybal to 1.13 on lvs100[56] (with service clean)
* 15:37 jynus: warming up srwiki.page and srwiki.categorylinks on s3 slaves
* 15:29 mobrovac: restbase canary deploy to rb1001 of 6b5a602
* 15:28 bblack: upgrading pybal to 1.13 on lvs1004, with cleared tables
* 15:23 _joe_: uploaded pybal 1.13 to jessie-wikimedia
* 15:14 hashar: stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit
* 14:58 hashar: Pooled zuul-merger instance on scandium.eqiad.wmnet . In case it screw up one has to stop the zuul-merger service on scandium.
* 14:57 godog: bounce mobileapps on scb1001/scb1002
* 14:47 mobrovac: mobileapps deploying 151c312
* 13:55 bblack: disabling pybal on lvs100[123] - fallback to lvs100[456]
* 12:00 hashar: reenabled puppet on gallium. Havent pooled zuul-merger yet
* 11:57 hashar: disabled puppet on gallium
* 11:56 hashar: pooling new zuul-merger on scandium
* 11:44 mark: Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1
* 10:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s)
* 10:31 hashar: gallium: apt-get upgrade
* 10:30 hashar: upgrading Zuul on gallium (CI interruption for a minute)
* 09:48 jynus: several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled)
* 09:25 godog: reimage restbase2002
* 09:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s)
* 09:09 _joe_: restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock
* 06:31 _joe_: rsyncing terbium homes from rutherfordium
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s)
* 01:46 andrewbogott: repooling mw1156
* 01:43 yurik_: synced graphoid service
* 01:40 andrewbogott: updated kernel on mw1156, now rebooting.
* 01:35 andrewbogott: repooling mw1158, depooling mw1156
* 01:22 andrewbogott: rebooting mw1158
* 01:19 andrewbogott: upgrading mw1158 to 3.13.0-62
* 00:26 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s)
* 00:04 logmsgbot: maxsem@tin Synchronized portals/: SWAT (duration: 00m 32s)
 
== 2015-11-17 ==
* 23:43 mutante: deactivated wekipedia.com
* 23:40 YuviPanda: remove realm variable from all instances in ldap
* 23:18 mutante: deactivated wikiartpedia.[biz|co|info|me|mobi|net|org]
* 23:17 awight: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to ecbc49c63b259a4363c51491ddff273e3579f0e9
* 23:13 mutante: deactivated border-wikipedia.de
* 23:01 mutante: deactivated wikimedia.xyz
* 22:57 mutante: deactivated visualwikipedia.[com|net]
* 22:56 mutante: deactivated softwarewikipedia.[com|net|org]
* 22:54 mutante: deactivated indiawikipedia.com
* 22:52 mutante: deactivated vikipedi.com.tr, vikipedia.com.tr
* 22:38 awight: updated fundraising/tools from 182f217ea20a496fe0f9737447d04ed9a3bccde8 to c6315eddf6f495e45401844d5f61bb240464c1e5
* 22:23 mutante: deactivated wikimaps.com, wikimaps.net, wikimaps.org
* 22:20 mutante: deactivated wikimedia.biz, webhostingwikipedia.com
* 22:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.7
* 21:07 logmsgbot: twentyafterfour@tin Finished scap: sync new branch 1.27.0-wmf.7 and enable for testwiki (duration: 29m 40s)
* 20:39 mutante: deleted sitemap.wikimedia.org (T101486)
* 20:38 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 20:31 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2994655917"/* "/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/l10n"' returned non-zero exit status 1 (duration: 00m 42s)
* 20:30 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 20:26 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4014066321" --threads=4 --lang en  --quiet' returned non-zero exit status 1 (duration: 01m 30s)
* 20:25 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 18:54 andrewbogott: depooled mw1158 due to kernel errors.  https://phabricator.wikimedia.org/T118888
* 18:10 cmjohnson1: swapping failing disk 32:8 on db1027..will cause icinga alert (jynus)
* 18:02 bblack: upgrading and re-enabling pybal on lvs300[12].esams.wmnet (1.10 -> 1.12)
* 17:54 yurik: deployed latest graphoid service
* 16:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation for bs.wikipedia.org [[gerrit:250862]] (duration: 00m 26s)
* 16:42 bblack: upgrading / re-enabling pybal on lvs200[123].codfw.wmnet (1.10 -> 1.12)
* 16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set import sources on en.wikiversity [[gerrit:250709]] (duration: 00m 27s)
* 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ru.wikipedia.org [[gerrit:252911]] (duration: 00m 26s)
* 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Editaton contra la violencia hacia las mujeres throttle rule [[gerrit:253541]] (duration: 00m 27s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 50% of new accounts on eswiki [[gerrit:250472]] (duration: 00m 26s)
* 16:02 hashar: Zuul-merger deployment aborted / uncomplte (I hate puppet)
* 16:02 bblack: re-enabling pybal on lvs400[12] (upgraded to 1.12)
* 15:36 cmjohnson1: reseating pem 0 on cr1-eqiad
* 15:30 mobrovac: restbase end deploy of e749f6ff
* 15:20 mobrovac: restbase start deploy of e749f6ff
* 14:59 bblack: stopping pybal on lvs300[12], fallback to lvs300[34] (pybal 1.10 vs 1.12)
* 14:55 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 1500
* 14:51 bblack: stopping pybal on lvs200[123], fallback to lvs200[456] (pybal 1.10 vs 1.12)
* 14:37 bblack: stopping pybal on lvs400[12], fallback to lvs400[34] (pybal 1.10 vs 1.12)
* 12:32 akosiaris: depool restbase1002
* 09:54 godog: nodetool decommission on restbase2002
* 06:18 gwicke: restbase cassandra: Increased tombstone_threshold from 0.02 to 0.1 for wikipedia and wikimedia html and data-parsoid to reduce single-sstable compaction rate.
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 26s)
* 00:21 logmsgbot: krinkle@tin Synchronized wmf-config/jobqueue-codfw.php: (no message) (duration: 00m 26s)
* 00:16 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: REL1_26 knocks on the door - https://gerrit.wikimedia.org/r/#/c/252399/ (duration: 00m 27s)
 
== 2015-11-16 ==
* 22:47 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.6/includes/Revision.php: Don't claim model validation failed if the content couldn't be loaded (duration: 00m 26s)
* 22:45 gwicke: rolling restbase restart to apply https://gerrit.wikimedia.org/r/#/c/253468/
* 21:52 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Don't log sysops with short passwords (duration: 00m 27s)
* 21:47 mobrovac: restbase end deployment of 54bc0517
* 21:38 mobrovac: restbase start deployment of 54bc0517
* 21:26 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log username of users with short passwords (duration: 00m 26s)
* 21:23 subbu: finished deploying parsoid 3a6f3b9e
* 21:21 mobrovac: restbase canary deploy to restbase1001 of 54bc0517
* 21:07 subbu: synced fresh code; restarting parsoid on wtp1005 as a canary
* 21:06 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log privileged users with short passwords. Set minimum password length of 10 for wikitech. (duration: 00m 26s)
* 21:03 subbu: starting parsoid deploy
* 19:52 anomie: Running cleanupBlocks.php on arwiki, eswikibooks, eswikinews, and testwiki for [[phab:T118625]]
* 19:23 RoanKattouw: Deployed patch for T116095
* 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/squid.php: SWAT: wgHTCPRouting: use separate address for upload [[gerrit:249121]] (duration: 00m 26s)
* 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable first QuickSurveys survey [[gerrit:253038]] (duration: 00m 26s)
* 14:40 godog: carbon daemon bounce on graphite1001
* 13:43 bblack: reinstalling lvs100[45]
* 13:39 jynus: dropping ipblocks_old table from all wikis
* 13:19 bblack: reinstalling lvs1006
* 11:21 jynus: defragementing, upgrading, general maintenance to mysql at db1027. Expect lag, etc. (it is depooled)
* 10:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1015, depool db1027 for maintenance (duration: 00m 49s)
* 10:03 jynus: performing schema change on wikishared.bounce_records (x1)
* 09:39 godog: bootstrap restbase2001-c cassandra instance
* 09:34 jynus: adding missing tables to vewikimedia https://phabricator.wikimedia.org/rMW7389d7c69035c968553fbd2eaf1cf47038c4577c
* 09:19 _joe_: reimaging terbium
* 09:16 godog: swift eqiad-prod: add ms-be1019 / ms-be1020 / ms-be1021 weight 500
* 08:56 _joe_: rsyncing files from terbium to rutherfordium
* 08:10 akosiaris: powered off plutonium, removed it from puppet and salt. context: https://phabricator.wikimedia.org/T118586
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 29s)
 
== 2015-11-15 ==
* 18:03 jynus: disabling puppet, restarting mysql on es2008, es2009, es2010 for configuration check/upgrade
* 16:41 jynus: stopping eventlogging_sync for manual resync on dbstore1002 and db1047
* 15:33 logmsgbot: reedy@tin Synchronized wmf-config/: noop after chmod (duration: 00m 27s)
* 14:31 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: revert last, no file created on fluorine (duration: 00m 27s)
* 14:28 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily enable error log. Will revert in a few minutes (duration: 00m 27s)
* 14:19 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Pull in a few fixes and improvements for BounceHandler (duration: 01m 12s)
* 12:17 jynus: restarting mysql at db1015 to finish upgrade/maintenance process
* 12:10 jynus: uploading upgraded mariadb package for trusty, too
* 09:02 _joe_: rebooting krypton
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 07s)
* 01:39 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.1
* 01:36 ostriches: deploying scap master@54d93b3
 
== 2015-11-14 ==
* 23:31 Reedy: dsh delete php-1.26wmf22 23 and 24
* 23:06 Reedy: dsh delete php-1.27.0-wmf.4/cache/l10n/*
* 23:06 Reedy: dsh delete php-1.27.0-wmf.3/cache/l10n/*
* 23:04 Reedy: dsh delete php-1.27.0-wmf.2/cache/l10n/*
* 23:04 Reedy: dsh delete php-1.27.0-wmf.1/cache/l10n/*
* 23:03 Reedy: dsh delete php-1.26wmf24/cache/l10n/*
* 23:01 Reedy: dsh delete php-1.26wmf23/cache/l10n/*
* 23:00 Reedy: dsh delete php-1.26wmf22/cache/l10n/*
* 22:58 Reedy: dsh delete php-1.26wmf21
* 22:57 Reedy: dsh delete php-1.26wmf20
* 22:57 Reedy: dsh delete php-1.26wmf19
* 22:56 Reedy: dsh delete php-1.26wmf18
* 22:55 Reedy: dsh delete php-1.26wmf17
* 22:54 Reedy: dsh delete php-1.26wmf16
* 22:51 logmsgbot: reedy@mira Synchronized phpunit.xml: (no message) (duration: 00m 29s)
* 22:49 logmsgbot: reedy@tin Synchronized php-1.26wmf24/: purge l10n cache... test if deletes are propogated (duration: 01m 12s)
* 22:43 Reedy: deleted more old l10ncaches from tin staging 1.27 wmf1-4
* 22:37 logmsgbot: reedy@tin Synchronized phpunit.xml: (no message) (duration: 00m 30s)
* 22:26 logmsgbot: reedy@tin Synchronized php-1.26wmf24: remove old localisation cache (duration: 01m 38s)
* 22:22 logmsgbot: reedy@tin Synchronized php-1.26wmf23: remove old localisation cache (duration: 01m 40s)
* 22:10 logmsgbot: reedy@tin Synchronized phpunit.xml: noop test (duration: 01m 03s)
* 20:51 Reedy: Deleted php-1.26wmf11, php-1.26wmf12 and php-1.26wmf13 from mira /srv/mediawiki-staging
* 20:40 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Email header logging on unsubscribe (duration: 00m 53s)
* 13:04 anomie: Ran cleanupBlocks.php on ruwiki for [[phab:T118625]]
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 58s)
* 01:58 ori: restarted HHVM on all app servers
 
== 2015-11-13 ==
* 23:30 logmsgbot: ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s)
* 23:27 awight: Reenabled fundraising campaigns outside of France
* 23:09 awight: Disabled Fundraising campaigns due to PayPal outage
* 21:23 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
* 21:17 YuviPanda: bounced etherpad again
* 20:39 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
* 20:38 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
* 20:35 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
* 20:30 logmsgbot: maxsem@tin Synchronized portals/: Bump to master (duration: 00m 39s)
* 20:19 robh: cr2-xe-1/3/0 optic install completed (detects fine)
* 20:18 robh: cr1-ulsfo:xe-0/0/2  optic install completed (detects fine)
* 20:17 robh: cr1-ulsfo:xe-0/0/2 (spare port) xfp optic detects fine, now installing other spare into cr2-xe-1/3/0
* 20:14 robh: installing xfp optics into cr1-ulsfo:xe-0/0/2 (spare port)
* 18:04 bblack: added neodymium to salt term in common-infrastructure4 on cr[12]-eqiad
* 17:37 csteipp: applied and reverted patch for T118032 (lots of exceptions on deploy)
* 16:16 jynus: restarting mysql on db2056, db2063 and db2064 for regular upgrade
* 15:16 godog: bootstrap restbase2001-b cassandra instance
* 15:03 jynus: restarting db2058, db2065 for regular mysql upgrade
* 14:43 jynus: restarting db2059, db2066 for regular mysql upgrade
* 11:42 jynus: restarting db2060 mysql for upgrade
* 09:20 jynus: performing table optimization on db1015, expect some lag while depooled
* 08:37 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100%, depool db1015 (duration: 01m 31s)
* 06:58 _joe_: trying to manually syncing the ubuntu mirrors, since they are corrupted at the moment
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 21s)
* 01:27 YuviPanda: bounced mw1045 HHVM for ebernhardson
* 01:15 logmsgbot: catrope@tin Finished scap: I totally forgot that the SWAT contained i18n changes (duration: 26m 46s)
* 00:48 logmsgbot: catrope@tin Started scap: I totally forgot that the SWAT contained i18n changes
* 00:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/QuickSurveys: SWAT (duration: 00m 30s)
* 00:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/Flow: SWAT (duration: 00m 32s)
* 00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/CentralNotice: SWAT (duration: 00m 31s)
* 00:16 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Remove hardcoded max upload size for UploadWizard (duration: 00m 30s)
* 00:15 awight: Banner impressions loader service restored
* 00:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Turn off language detection user test for CirrusSearch (duration: 00m 30s)
* 00:04 awight: update DjangoBannerStats from 08ffecf412a6a9ff3e173dded6814092834c7d1e to 26b48add5c6f70b395a3195c401a12612c626796
 
== 2015-11-12 ==
* 23:41 awight: update DjangoBannerStats from 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 to 08ffecf412a6a9ff3e173dded6814092834c7d1e
* 23:05 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Monolog: wrap channel handlers in a WhatFailureGroupHandler (b08aaf4) (duration: 00m 29s)
* 22:46 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.6
* 22:42 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/TranslationNotifications/TranslationNotificationJob.php: hotfix T116960 (duration: 00m 30s)
* 21:21 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/resources/lib/jquery.i18n/src/jquery.i18n.language.js: T118242 - Unbreak IE8 (duration: 00m 30s)
* 21:10 chasemp: reboot analytics1040 it locked up
* 19:37 awight: Update DjangoBannerStats from 12e819a04a40ee6fab5dd55fcaf072661df31106 to 9cc2acba7f6b46e243ada0fac6c48c9a49460a07
* 19:24 awight: banner impressions job disabled
* 19:11 awight: Disabled WMDE and WMF Fundraising campaigns to do banner impression maintenance
* 19:05 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Set throttle exception for University of Haifa wiki event [[gerrit:252010]] (duration: 00m 29s)
* 18:21 logmsgbot: maxsem@tin Synchronized portals/: The rest of portals (duration: 00m 31s)
* 17:25 subbu: finished parsoid deploy sha 392e25eb
* 17:02 subbu: starting parsoid deploy
* 16:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy QuickSurveys to English Wikipedia [[gerrit:252606]] (duration: 00m 29s)
* 16:28 ostriches: gerrit: reloading replication plugin to pick up new config
* 16:11 andrewbogott: depooled T118469.  https://phabricator.wikimedia.org/T118469
* 16:08 bd808: cleaned up stale branch checkouts on mw1041: php-1.26wmf16 php-1.26wmf17 php-1.26wmf18 php-1.26wmf19 php-1.26wmf20 php-1.26wmf21
* 16:02 mobrovac: restbase end deploy of e5e45d11ad3f
* 15:48 mobrovac: restbase start deploy of e5e45d11ad3f
* 15:46 mobrovac: restbase canary deploy on rb1001 of e5e45d11ad3f
* 15:39 jynus: powercycling mw1041
* 15:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 after maintenance (duration: 00m 29s)
* 10:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 00m 31s)
* 09:53 jynus: rebooting/upgrading/restarting mariadb on db1035 after maintenance finished
* 09:52 jynus: uploading new mariadb package linked to openssl instead of yassl
* 09:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 01m 15s)
* 09:39 moritzm: corrected apt sources on db2057, db2058, db2062, db2065, db2066, db2067, db2068, db2069
* 08:42 moritzm: restarted salt-master on palladium
* 07:35 _joe_: restarting pybal on lvs1004
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 56s)
* 01:02 gwicke: restbase: finished full deploy of restbase-deploy ce4abe784
* 00:50 gwicke: restbase: starting full deploy of restbase-deploy ce4abe784
* 00:37 gwicke: restbase: canary deploy of restbase-deploy ce4abe784 on restbase1001
* 00:32 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in QuickSurveys config (duration: 00m 31s)
 
== 2015-11-11 ==
* 23:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
* 23:44 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/includes/jobqueue/aggregator/: sync https://gerrit.wikimedia.org/r/#/c/252588/ (duration: 00m 32s)
* 21:13 subbu: finished deploying parsoid sha 7ca999c1
* 21:06 subbu: synced new code + restarted parsoid on wtp1001 as canary; monitoring graphs for a little bit
* 21:02 subbu: starting parsoid deploy
* 20:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
* 19:51 hoo: Started rebuildItemsPerSite for wikidatawiki on mw1152. Feel free to kill, should it cause troubles.
* 19:49 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
* 19:37 twentyafterfour: deploying 1.27.0-wmf.6 to group1
* 19:09 YuviPanda: nfs-exports on labstore1001 failed because of http failure of wikitech
* 19:05 YuviPanda: started nfs-exports service on labstore1001
* 18:18 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2067 after maintenance (duration: 00m 29s)
* 16:46 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II [[gerrit:252349]] (duration: 00m 30s)
* 16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I [[gerrit:252349]] (duration: 00m 30s)
* 16:34 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II [[gerrit:252347]] (duration: 00m 30s)
* 16:33 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I [[gerrit:252347]] (duration: 00m 40s)
* 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: First QuickSurvey for reader segmentation research - external survey [[gerrit:251133]] (duration: 00m 30s)
* 16:02 mutante: uploaded jenkins 1.625.2 to apt.wm.org, upgrading on gallium
* 15:47 moritzm: restarted kafka broker on kafka1022
* 15:34 moritzm: stopping kafka broker on kafka1022 (to enable ferm)
* 15:31 moritzm: restarted kafka broker on kafka1020
* 15:25 moritzm: stopping kafka1020 to enable ferm
* 15:22 moritzm: restarted kafka1018 to enable ferm
* 15:19 moritzm: stopping kafka1018 to enable ferm
* 15:04 moritzm: disabled puppet on kafka1018, kafka1020, kafka1022 (for enabling ferm)
* 14:18 godog: start cassandra instance a on restbase2001
* 13:54 mobrovac: restbase deployed 0d961a2 (deploy end)
* 13:42 mobrovac: restbase deploying 0d961a2
* 11:28 godog: reimage restbase2001.codfw.wmnet - take #2
* 10:16 jynus: disabling puppet and restarting mysql on db2067
* 10:16 godog: reimage restbase2001.codfw.wmnet
* 10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2067 for maintenance (not db1067) (duration: 00m 30s)
* 10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1067 for maintenance (duration: 00m 30s)
* 09:57 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: really fix T118372 this time (duration: 00m 29s)
* 09:43 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: oauth login broken by php error. This fixes T118372 (duration: 00m 31s)
* 08:51 jynus: importing wmf-mariadb10_10.0.22-1_amd64.deb wmf-mysql57_5.7.9-1_amd64.deb on carbon
* 04:19 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm some old log channels (duration: 00m 31s)
* 03:23 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/includes/cache/MessageBlobStore.php: Logging for T93800 (duration: 00m 31s)
* 02:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.5/includes/Title.php: Deploy I47517021471 which was merged earlier today but forgotten (duration: 01m 58s)
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 07m 01s)
* 01:48 mutante: repooled mw1083 (T116184)
* 00:58 mutante: added jenkins_1.625.1 to APT repo. precise-wikimedia
* 00:50 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.6
* 00:38 logmsgbot: twentyafterfour@tin Finished scap: grr: sync 1.27.0-wmf.6 (duration: 55m 05s)
 
== 2015-11-10 ==
* 23:42 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
* 23:41 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.7rNhS6PIRi" ' returned non-zero exit status 1 (duration: 00m 09s)
* 23:41 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
* 23:40 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.yIUW9abUwO" ' returned non-zero exit status 1 (duration: 00m 15s)
* 23:39 logmsgbot: twentyafterfour@tin Started scap: trying again: sync 1.27.0-wmf.6
* 23:23 ori: ran sudo chgrp -R wikidev /srv/mediawiki-staging/php-1.27.0-wmf.6 on tin
* 22:50 ejegg: rolled back paymentswiki from 3730cff201e314fc54ad76fbc54f33f5b032def5 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
* 22:43 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.sEpytaqlt1" ' returned non-zero exit status 1 (duration: 01m 42s)
* 22:41 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.6
* 21:51 hashar: installed Zuul package on scandium.eqiad.wmnet . zuul-merger not running
* 21:33 ejegg: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to 3730cff201e314fc54ad76fbc54f33f5b032def5
* 21:16 ejegg: enabled banner_history module on CiviCRM
* 21:15 mutante: scb1001 - mobile endpoint health CRIT
* 21:15 mutante: mira unmerged changes in mw-staging
* 21:14 mutante: restbase2001: cassandra CQl refused
* 21:14 mutante: aqs1001-1003: CRITICAL: Test Get aggregate page views
* 21:12 mutante: tungsten - signed new puppet certs, re-installed
* 20:48 moritzm: kafka re-enabled on kafka1013
* 20:42 moritzm: stopping kafka broker on kafka1013 to activate ferm
* 20:26 ottomata: starting kafka broker on kafka1012 after applying base ferm
* 20:14 ottomata: stopping kafka broker on kafka1012 to apply base ferm
* 20:09 moritzm: disabled puppet on kafk1012/1013 in preparation of ferm activation
* 20:08 _joe_: restarting jobchron across the cluster
* 18:41 ottomata: powercycling analytics1031
* 16:51 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s)
* 16:45 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s)
* 16:44 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s)
* 16:41 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s)
* 16:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s)
* 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s)
* 16:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s)
* 15:51 mutante: restarted krypton
* 15:32 godog: upgrade rsync on ms-be1* per T93587
* 14:02 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000
* 10:00 jynus: performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected.
* 09:46 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000
* 09:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s)
* 06:13 ejegg|away: updated fundraising tools from f2f571903d32ec73aeab73ce58c6bee71717213b to 182f217ea20a496fe0f9737447d04ed9a3bccde8
* 03:12 ottomata: restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) )
* 03:10 ottomata: kafka preferred-replica-election to bring kafka1013  back as leader for its partitions
* 03:00 mutante: started kafka on kafka1013
* 02:55 mutante: tried to start kafka on kafka1018
* 02:31 mutante: switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
* 00:13 hoo: Attached Richardelainechambers@commonswiki to the global account of the same name
* 00:06 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s)
 
== 2015-11-09 ==
* 23:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Disable cirrus writes to labsearch, the machine cant take the load and some jobs are timing out (duration: 00m 34s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable CirrusSearch writes to labsearch for all but enwiki and dewiki (duration: 00m 35s)
* 22:49 ottomata: rebooting analytics1032: https://phabricator.wikimedia.org/T118175
* 21:52 bearND: MobileApps deployed sha1 6c63984
* 19:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1060 after performance patch (duration: 00m 35s)
* 18:42 jynus: restarting mysql in db1060 to test new performance configuration
* 17:55 bblack: upgrading pybal (1.10 -> 1.12) on lvs200[456].codfw.wmnet
* 16:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250454/ (duration: 00m 34s)
* 16:44 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init: https://gerrit.wikimedia.org/r/#/c/251972/ (duration: 00m 34s)
* 16:40 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251674/ (duration: 00m 35s)
* 16:35 logmsgbot: krenair@tin Synchronized docroot/wikipedia.org/apple-app-site-association: https://gerrit.wikimedia.org/r/#/c/250897/ (duration: 00m 34s)
* 16:30 matt_flaschen: Ran mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php --wiki=wikidatawiki
* 16:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251677/ (duration: 00m 35s)
* 16:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow/modules/mw.flow.Initializer.js: https://gerrit.wikimedia.org/r/#/c/251560/ (duration: 00m 44s)
* 16:10 _joe_: removing old builds from copper
* 15:45 godog: reimage graphite1002
* 15:06 ottomata: running kafka preferred-replica-election
* 15:02 ottomata: starting kafka broker on kafka1014
* 14:56 ottomata: stopping kafka broker on kafka1014
* 14:48 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 1000
* 14:40 ottomata: restarting eventlogging to see if it is ok after enabling firewall rules on kafka1014
* 14:36 _joe_: reducing /tmp size on copper by shrinking the logical volume
* 13:16 godog: nodetool decomission restbase2001
* 13:13 mobrovac: restbase finished deploying ae2a44f
* 12:59 mobrovac: restbase start deploying ae2a44f
* 12:56 hashar: restarting Jenkins to refresh the cli-shutdown.groovy script -- https://gerrit.wikimedia.org/r/251935 (https://phabricator.wikimedia.org/T118064)
* 12:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling db1060 again (duration: 01m 13s)
* 11:16 hashar: Jenkins back and apparently happy
* 11:09 hashar: Upgrading Jenkins from LTS 1.609.3 to LTS 1.625.1 https://phabricator.wikimedia.org/T118157
* 10:53 dcausse: resuming writes to elasticsearch indices in eqiad (test)
* 10:02 _joe_: repooled mw1061
* 09:29 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 3000
* 09:19 dcausse: err (1007 no 1008): restarting elastic on elastic1007.eqiad.wmnet (test)
* 09:18 dcausse: restarting elastic on elastic1008.eqiad.wmnet (test)
* 09:10 dcausse: freezing elasticsearch indices in eqiad (test)
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
 
== 2015-11-08 ==
* 20:43 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ConfirmEdit/SimpleCaptcha/Captcha.php: https://gerrit.wikimedia.org/r/#/c/251673/ (duration: 00m 34s)
* 20:21 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources: (no message) (duration: 00m 34s)
* 20:14 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources/src: I24d9b16ed: Rework the Preferences to prevent FOUC (T115692) (duration: 00m 34s)
* 17:10 _joe_: powercycled ms-be1009, unresponsive to ping, console unconneptable
* 13:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1060 (duration: 00m 34s)
* 13:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 01m 29s)
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 21s)
 
== 2015-11-07 ==
* 19:50 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/251739/ (duration: 00m 35s)
* 17:54 ejegg|away: updated SmashPig from 5860ef47790010d65981822f30914679df06ae14 to 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
* 17:38 awight: more syntax fixing
* 17:35 awight: fix SmashPig failmail syntax
* 17:29 awight: point SmashPig failmail at ejegg|away and awight
* 17:28 jynus: restarting labsdb1004 mysql to test mariadb upgrade
* 17:09 jynus: restarting db2070 mysql to test mariadb upgrade
* 17:01 jynus: restarting es2010 mysql to test mariadb upgrade
* 15:17 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Return db weights on s2 api back to normal (duration: 00m 34s)
* 11:24 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1054 (duration: 00m 34s)
* 11:02 jynus: setting SET GLOBAL use_stat_tables = 0; on db1060
* 10:45 jynus: restarting db1054 (I may need to do it several times)
* 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1054 (duration: 00m 35s)
* 10:27 jynus: reverting last schema change
* 10:08 jynus: performing schema change on db1054 (zhwiki)
* 05:00 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 34s)
* 04:59 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: (no message) (duration: 01m 09s)
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 12s)
* 00:23 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
* 00:05 ejegg: updated SmashPig from ac9772765287e79178e07ddfcc7e532d2f82b6d0 to 5860ef47790010d65981822f30914679df06ae14
* 00:04 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
 
== 2015-11-06 ==
* 23:48 ejegg: updated SmashPig listener from afefccff0f8d5e0b37044adff964c6cd913e1d80 to ac9772765287e79178e07ddfcc7e532d2f82b6d0 and set failmail target back to fr-tech
* 23:34 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Enable DynamicPageList on officewiki (duration: 01m 03s)
* 22:42 legoktm: mwscript updateSpecialPages.php --wiki=enwiki --only=GadgetUsage
* 19:22 andrewbogott: disabled puppet on holmium while testing and staging an update for bug T117610
* 19:13 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I920627cf8b: Add rdb1007 as fallback aggregator (duration: 00m 35s)
* 18:54 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9e5be66680: Add rdb1007 to jobqueue pool (duration: 00m 35s)
* 18:19 ori: running sync-common on mw1083
* 17:45 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: no-op outside tin; just for completeness (duration: 00m 36s)
* 15:29 paravoid: draining esams for network maintainance (deployed about 7mins ago)
* 14:52 moritzm: uploaded debdeploy 0.0.9 to apt.wikimedia.org
* 14:31 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool all available db servers in codfw (duration: 00m 49s)
* 13:58 godog: bounce ganglia-monitor-aggregator-instance ID=2027 on install2001, debugging missing ms-be2020
* 13:38 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 2000
* 13:36 bblack: cp4007 repooled - T117746
* 13:16 bblack: upgrading pybal 1.10 -> 1.12 on lvs300[34].esams.wmnet
* 13:11 bblack: re-enabled aws-d-eqiad in librenms
* 13:09 bblack: upgrading pybal 1.10 -> 1.12 on lvs4003
* 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10G line card
* 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10
* 12:18 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 online
* 12:10 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 offline
* 11:31 jynus: stopping mysql and cloning db2042 -> db2070, db2054 -> db2068, db2053 -> db2067, db2049 -> db2056
* 11:12 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2042, db2049, db2053 and db2054 for cloning (duration: 00m 34s)
* 09:09 jynus: Stopping mysql and cloning db2044 -> db2065, db2048 -> db2069, db2045 -> db2066
* 08:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db20 34,42,35,38,39,40. New: 58,64,60,61,62,59. Depool: 44,45,48 (duration: 01m 10s)
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 05m 56s)
* 01:43 logmsgbot: krenair@mira Synchronized README: done (duration: 00m 34s)
* 01:38 logmsgbot: krenair@mira Synchronized README: test (duration: 00m 34s)
* 01:33 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251446/ (duration: 00m 35s)
* 01:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (forgot to apply after fetching) (duration: 00m 34s)
* 01:28 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touch (duration: 00m 35s)
* 01:26 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (duration: 00m 34s)
* 01:10 logmsgbot: krenair@tin Synchronized multiversion/defines.php: https://gerrit.wikimedia.org/r/#/c/251447/ (duration: 00m 34s)
* 01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 34s)
* 00:59 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: https://gerrit.wikimedia.org/r/#/c/251301/ https://gerrit.wikimedia.org/r/#/c/251306/ and https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 35s)
* 00:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251317/ (duration: 00m 35s)
* 00:18 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/251416/ (duration: 00m 37s)
 
== 2015-11-05 ==
* 23:34 bd808: Decreased replica count of logstash-2015.10.13 and logstash-2015.10.14 to free disk space on cluster
* 23:30 bd808: Logging volume into ELK cluster down dramatically; investigating
* 23:16 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ieb8c602a: monolog: Ensure that context data added by WebProcessor is utf-8 safe (duration: 00m 36s)
* 22:35 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents: Ic99ac31f740956: Log backend response time on edit requests (duration: 00m 35s)
* 21:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync 1.27.0-wmf.5 to group2
* 20:17 logmsgbot: demon@tin Finished scap: no changes, testing permissions on mira co-master (duration: 07m 29s)
* 20:10 logmsgbot: demon@tin Started scap: no changes, testing permissions on mira co-master
* 20:07 _joe_: chown mwdeploy:wikidev recursively on mira for /srv/mediawiki-staging
* 19:37 logmsgbot: twentyafterfour@tin Finished scap: Sync everything just to be sure (duration: 22m 01s)
* 19:15 logmsgbot: twentyafterfour@tin Started scap: Sync everything just to be sure
* 18:58 logmsgbot: demon@tin Synchronized README: no-op, testing new scap code (duration: 00m 19s)
* 18:52 ostriches: scap: deploying master@b44c268
* 18:05 jynus: schema change on x1 - flowdb
* 17:47 ostriches: 17:45:13 Synchronized wmf-config/: Remove +superprotect, I579c11a2 (duration: 00m 18s)
* 17:44 jynus: Deploying schema change on officewiki - flow (s3)
* 17:11 godog: nodetool decommission on praseodymium
* 14:12 jynus: reinstalling db2056.codfw.wmnet
* 13:57 jynus: shuting down mysql and cloning db2042 -> db2062, db2038 -> db2059, db2039 -> db2060, db2040 -> db2061
* 13:43 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2051; Depool db2042, 38, 39, 40 for cloning (duration: 00m 18s)
* 12:33 _joe_: manually disabling mod_cgi on terbium
* 09:56 godog: run removenode on cerium.eqiad.wmnet -- decomission was missed before reimaging
* 09:33 jynus: stopping mysql and cloning db2034 -> db2062, db2035 -> db2063, db2051 -> db2058
* 09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool db2048, db2049, db2050,db2055, db2057, db2063. Depool db2034, db2035, db2051 (duration: 00m 17s)
* 08:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Nov  5 08:05:13 UTC 2015 (duration 5m 12s)
* 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-05 03:21:45+00:00
* 03:15 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 17s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-05 02:47:20+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 10m 31s)
* 02:17 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3c397e892e: Only set $wgDisableOutputCompression to 'true' on the scalers (duration: 00m 18s)
* 02:04 Krenair: Someone has left tin:/srv/mediawiki-staging/php-1.27.0-wmf.5 in a mess, see `git log origin/wmf/1.27.0-wmf.5..HEAD --oneline`. Note there is one commit waiting to be merged on tin (https://gerrit.wikimedia.org/r/#/c/251168/) that hasn't been yet because of this.
* 00:13 ejegg: updated fundraising tools from e1b60fa2c258fd4ff55905b03a4d8886132278c1 to f2f571903d32ec73aeab73ce58c6bee71717213b
 
== 2015-11-04 ==
* 23:56 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: ced9a62893c (duration: 00m 17s)
* 23:55 ejegg: updated CiviCRM from 827faf37b1b8f4ade6f8d03077c376a09d9525c7 to 230cebb4a3b46a75fef758d46f4db22de29286cb
* 23:09 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: 33d378bee6c4ac (duration: 00m 17s)
* 22:33 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: fb43e95858b9 (duration: 00m 18s)
* 21:36 ejegg: enabled Adyen payments listener
* 21:17 subbu: finished deploying parsoid sha 04893a18
* 21:05 subbu: synced code + restarted parsoid on wtp1002 as a canary
* 21:01 subbu: starting parsoid deploy
* 20:45 ejegg: disabled Adyen payment listener
* 20:36 ejegg: re-enabled Adyen payment listener with updated credentials
* 20:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
* 18:37 mutante: correction. ^ that was mw1159
* 18:35 mutante: mw1159: power cycled - crashed, no console output
* 18:23 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/includes/debug/logger/MonologSpi.php: MonologSpi: add support for customizing Monolog\Logger instances (59eed42) (duration: 00m 17s)
* 18:23 mutante: ms-be2016-2020 - accepted salt keys, hafnium.wm.org - deleted salt key - still unaccepted: capella, haedus, rhodium
* 17:57 mutante: ms-be2020 - added to puppet
* 17:54 mutante: ms-be2021 - fixing puppet certs after reinstall
* 17:51 papaul: ms-be2021 installation complete
* 17:45 papaul: reinstalling ms-be2021
* 17:20 ejegg: updated SmashPig from be9bf87c5583bca75616284f6a4d16ff8ebd3679 to afefccff0f8d5e0b37044adff964c6cd913e1d80
* 17:12 chasemp: rebooting cp4007.mgmt.ulsfo.wmnet as it's uresponsive
* 17:11 bblack: cp4007 depooled in pybal + confctl
* 17:10 papaul: ms-be2020 installation complete
* 16:56 ejegg: disabled adyen payments listener
* 16:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250971/ (duration: 00m 18s)
* 16:29 bd808: restarted elasticsearch on logstash1004; it fell out of the cluster during ferm firewall application
* 16:28 jynus: db2056 clone failed due to wrong partitioning, needs reinstall. Cloning db2049 -> db2063 instead
* 16:01 godog: reimage cerium for multi-instance
* 15:22 _joe_: moving email batch, flagged reviews, update article count off of terbium
* 15:20 jynus: syncing sqldata on db2049 -> db2056, db2050 -> db2057, eta 2-4 hours
* 15:15 _joe_: moved translationnotification, updatetranslationstats, pagetriage from terbium to mw1152
* 14:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2049 ans db2050 for maintenance (duration: 00m 17s)
* 14:30 jynus: cloning sqldata from db2048 to db2055
* 14:04 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2048 for maintenance (duration: 00m 18s)
* 13:03 bblack: upgrading pybal (1.10 -> 1.12) on lvs4004.ulsfo.wmnet (inactive low-traffic LVS @ ulsfo)
* 11:59 _joe_: moved refreshlink jobs off terbium
* 11:32 _joe_: migrating wikidata jobs off of terbium
* 11:07 jynus: testing new mariadb-server version on db2056.codfw.wmnet
* 09:25 godog: bootstrap cassandra xenon-b.eqiad.wmnet
* 08:47 jynus: reduced durability on dbstore1002 and db1047 to improve performance
* 08:27 _joe_: moved parser cache purging off of terbium
* 08:25 _joe_: moved clean_uploadstash to mw1152 from terbium
* 06:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Nov  4 06:44:23 UTC 2015 (duration 44m 22s)
* 05:52 logmsgbot: demon@tin Finished scap: using wfLoadExtension() for ContentTranslation, shut up warning spam (duration: 49m 22s)
* 05:03 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
* 05:02 logmsgbot: demon@tin scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 24 (duration: 02m 45s)
* 04:59 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
* 04:54 ejegg|away: updated smashpig from c43db1742d5663f9d3685fd3e0a254ff72a29fa0 to be9bf87c5583bca75616284f6a4d16ff8ebd3679, changed credentials
* 03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-04 03:09:44+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 08s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-04 02:36:48+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 34s)
* 01:36 legoktm: ran delete from globaluser where gu_name="" limit 1 on centralauth. (T96233)
* 01:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/250874/ (duration: 00m 17s)
* 01:02 hoo: Manually recreated https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf using Zend php (instead of HHVM). See T117534.
* 00:25 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250869/ (duration: 00m 19s)
* 00:09 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250712/ (duration: 00m 18s)
* 00:03 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ (duration: 00m 17s)
 
== 2015-11-03 ==
* 23:11 ejegg: deployed Adyen credentials change to payments wiki
* 22:36 ostriches: tin: scap now pointing to Phab repo instead of Gerrit.
* 21:59 logmsgbot: krenair@tin Synchronized README: rv, was just demonstrating for https://phabricator.wikimedia.org/T117574#1779384 (duration: 00m 17s)
* 21:42 logmsgbot: krenair@tin Synchronized README: (no message) (duration: 00m 18s)
* 21:21 mutante: ms-be2018,ms-be2019: sign puppet certs, initial run
* 20:45 csteipp: deployed patch for T109724 to wmf4/5
* 20:41 gwicke: restbase cassandra: switched local_group_wikipedia_T_parsoid_html to the Date-Tiered Compaction Strategy (DTCS)
* 20:38 gwicke: cleared cassandra snapshots on aqs servers
* 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.5
* 19:55 mutante: hafnium - revoke puppet cert, salt key for the .wikimedia.org name
* 19:54 mutante: hafnium - reboot into PXE for jessie reinstall
* 19:47 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.27.0-wmf.5 (duration: 30m 00s)
* 19:35 awight: update CRM from 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7 to 827faf37b1b8f4ade6f8d03077c376a09d9525c7
* 19:33 awight: update SmashPig from c431c8d77521270236c72532a50806b2e852cf7b to c43db1742d5663f9d3685fd3e0a254ff72a29fa0
* 19:17 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.5
* 19:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/objectcache/BagOStuff.php: 46dff75b5b: Make makeKeyInternal() limit more conservative (duration: 00m 18s)
* 19:06 twentyafterfour: deploying 1.27.0-wmf.5 to group0
* 18:34 ori: Moved statsv script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
* 18:29 ori: Moved navtiming script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
* 17:13 moritzm: installed security updates on the internal ntpds (system-local ones to follow)
* 16:27 dcausse: restarting elastic on nobelium to check new settings
* 16:19 moritzm: installed unzip and audiofile security updates across the fleet
* 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of CirrusSearchRequestSet log channel [[gerrit:250691]] (duration: 00m 18s)
* 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on eswiki [[gerrit:250470]] (duration: 00m 18s)
* 14:19 jynus: disabling puppet on dbstore1002 and db1047 to debug replication issue
* 14:07 godog: manual invocation of /usr/local/bin/archive-instances on labmon1001 to test
* 13:33 jynus: migrating eventlogging_sync from terbium to db1047 and dbstore1002 (analytics-store)
* 12:15 _joe_: updated pybal for jessie to 1.12
* 11:00 godog: start cassandra-a on xenon
* 09:56 godog: reimage xenon.eqiad.wmnet
* 09:45 godog: decomission xenon.eqiad.wmnet from cassandra, pending conversion to multi-instance
* 05:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Nov  3 05:38:33 UTC 2015 (duration 38m 32s)
* 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-03 02:26:14+00:00
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 00s)
* 01:05 awight: enabled CRM module amazon_audit
* 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s)
* 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s)
* 00:37 logmsgbot: ebernhardson@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s)
* 00:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
* 00:34 logmsgbot: ebernhardson@tin Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
* 00:28 ebernhardson: sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615
* 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s)
* 00:17 awight: update CRM from 77cd64317404b2fcec9500cfbb1c0b19e5846705 to 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7
* 00:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250590/ (duration: 00m 18s)
* 00:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: add www.webarchive.co.uk and *.unesco.org to server side upload whitelist (duration: 00m 18s)
* 00:06 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/246169/ (duration: 00m 18s)
 
== 2015-11-02 ==
* 23:11 RoanKattouw: Running convertAllLqtPages.php on ptwikibooks
* 22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Flow on ptwikibooks with English-only Topic namespace name (duration: 00m 18s)
* 22:38 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgFlowEnglishNamespaceOnly (duration: 00m 18s)
* 21:56 cwd: updated worldpay request timeout to 30s
* 21:31 _joe_: restarting parsoids in codfw
* 21:25 subbu: deployed parsoid f0d77afc
* 20:49 mutante: ms-be2017 - initial puppet run, fresh install
* 20:38 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.4/includes/: (no message) (duration: 00m 22s)
* 20:38 robh: lawrencium having puppet stopped and reclaiming into spares per T117477 disregard any alerts
* 20:06 mutante: ms-be2016 - signing puppet certs, salt-key, initial run
* 20:05 mutante: started morebots (production-logbot job was gone)
* 19:46 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/includes/MobileFrontend.skin.hooks.php: Fix changing the license message key via the "MobileLicenseLink" hook (duration: 00m 17s)
* 19:14 gwicke: restbase cassandra: switching local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ to DTCS
* 19:00 matt_flaschen: Completed FlowFixLinks run on all Flow wikis
* 18:53 csteipp: deployed patches for T97897 & T115522
* 18:53 awight: updated CRM from 60c4a8e5fc4efb6ac38a520e652afe187e8c177b to 77cd64317404b2fcec9500cfbb1c0b19e5846705
* 18:48 andrewbogott: dist-upgrade and rebooting labvirt1005
* 18:41 Coren: manually starting service replicate-tools on labstore1001
* 18:10 awight: removed failing SmashPig job runner task
* 18:01 jynus: performing some fixes on labsdb1004 database. Slightly higher load on labsdb1005.
* 17:59 dcausse: restarting elastic on nobelium
* 17:43 awight: disabled Donations queue consume job
* 17:39 gwicke: restbase cassandra: switched local_group_wikisource_T_parsoid_html and local_group_wikibooks_T_parsoid_html to DTCS
* 17:09 bd808: Decreased replica count to 1 for logstash-2015.10.04 thru logstash-2015.10.12 to free cluster disk space; see T117438
* 17:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/249762/ (duration: 00m 19s)
* 17:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247088/ (duration: 00m 17s)
* 16:48 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
* 16:47 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
* 16:45 logmsgbot: krenair@tin Synchronized dblists/closed.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ - close wikimania2014.wikimedia.org (duration: 00m 18s)
* 16:36 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249794/ - should be a noop for prod (duration: 00m 17s)
* 16:30 bd808: Deleted logstash-2015.10.03 index to free disk space on logstash1004; see T113571
* 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/249940/1 (duration: 00m 20s)
* 16:26 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249935/ (duration: 00m 18s)
* 16:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250401/ (duration: 00m 18s)
* 14:43 ottomata: shutting down analytics1038 to run fsck on root filesystem.  Currently it is read only
* 14:36 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-br-fr_0.5.0+svn~57870-1
* 14:35 mobrovac: citoid deployed 70abb90
* 13:34 paravoid: draining ulsfo, T116928
* 12:19 mobrovac: mobileapps deploying 59e8c30
* 10:52 hashar: Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion"
* 10:10 akosiaris: powering off pollux. migrating to a VM
* 09:39 _joe_: moved tor exit node, update_special_pages off of terbium to mw1152
* 09:24 mobrovac: citoid restarted zotero on sca100x
* 09:07 jynus: deleting user_daily_contribs view on labs (it points to a non-existent table)
* 08:57 mobrovac: restbase deploying 8036232
* 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Nov  2 05:33:07 UTC 2015 (duration 33m 6s)
* 02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-02 02:25:42+00:00
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 44s)
 
== 2015-11-01 ==
* 16:59 gwicke: restarting cassandra on aqs1002; was out of heap space
* 05:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Nov  1 05:25:49 UTC 2015 (duration 25m 48s)
* 02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-01 02:24:46+00:00
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 36s)
 
== 2015-10-31 ==
* 08:57 dcausse: elastic1008 deleting /var/log/elasticsearch/production-search-eqiad_index_indexing_slowlog.log.[2-7]
* 05:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 31 05:15:43 UTC 2015 (duration 15m 42s)
* 02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda4768 (duration: 00m 19s)
* 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-31 02:26:22+00:00
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 26s)
 
== 2015-10-30 ==
* 23:53 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ifd3543d99: Re-enabled sidebar cache per 47eb083a0fe4 (duration: 00m 17s)
* 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/cache/MessageCache.php: Id9a27ba2bbd3 (duration: 00m 17s)
* 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/skins/Skin.php: Id9a27ba2bbd3 (duration: 00m 17s)
* 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Collection: b89cff0a1d: Updated mediawiki/core Project: mediawiki/extensions/Collection  092204bf333e65b2c749e4bc4a32fd2b0254089b (duration: 00m 18s)
* 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/DismissableSiteNotice: a20aa8a544: Updated mediawiki/core Project: mediawiki/extensions/DismissableSiteNotice  5b8f1cfac48704fd740eb61f873aabb600c4c5fb (duration: 00m 17s)
* 23:03 gwicke: fixed restbase labs install by deploying it; the code had gone missing
* 23:03 ori: Added MaxSem and gwicke to Gerrit Project Creators group
* 22:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/objectcache/ObjectCache.php: If12aedae7f: objectcache: Use singleton cache in newAccelerator() (duration: 00m 18s)
* 22:05 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
* 22:01 mutante: labvirt1010: salt-minion always in "stop/waiting", doesn't restart
* 21:54 mutante: labvirt1010 - start salt
* 21:41 ori: Removing MediaWiki.xhprof.* from graphite{1,2}001
* 21:41 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s)
* 19:07 K4-713: antifraud tweak on payments wiki (again)
* 18:56 K4-713: antifraud tweak on payments wiki
* 18:56 jynus: disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact
* 17:33 robh: bismuth has no console output, appears locked up in OS, rebooting via mgmt
* 17:22 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 18s)
* 16:56 cmjohnson: reinstalling mw1083 https://phabricator.wikimedia.org/T116184
* 15:35 moritzm: updated db205[6-8] to the 3.19 kernel
* 14:38 chasemp: nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable
* 13:47 Krenair: revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin
* 13:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s)
* 11:57 paravoid: upgrading tor to latest stable on radium
* 11:31 paravoid: ignoring asw-d-eqiad on librenms
* 11:25 _joe_: powercycling dataset1001, stuck in some kernel task, unable to login from console
* 10:42 dcausse: elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki
* 10:30 dcausse: elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181)
* 09:18 dcausse: elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1
* 08:46 _joe_: removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008
* 06:24 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s)
* 05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s)
* 04:19 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s)
* 04:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s)
* 04:03 awight: update CRM from e0cad7215192464941d0ec282a70cb608619cf2f to 60c4a8e5fc4efb6ac38a520e652afe187e8c177b
* 03:42 awight|parentoid: update CRM from 367cd85954697c6ad4311deb2799fe9ba08b5d9d to e0cad7215192464941d0ec282a70cb608619cf2f
* 02:47 awight|parentoid: donations queue consumer reenabled
* 02:45 awight|parentoid: updated CRM from 61c8c13efb31f7e75564d9a01fa879db0690fb78 to 367cd85954697c6ad4311deb2799fe9ba08b5d9d
* 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s)
* 00:48 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s)
* 00:34 mutante: labvirt1010 - started salt-minion (no such process)
* 00:34 mutante: mw1193 - restarted HHVM (socket timeout)
* 00:34 mutante: wdqs1001 - restarted NTP (unknown offset)
* 00:09 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s)
* 00:08 mutante: radium - back in service
* 00:01 bd808: Restarted logstash on logstash1003 again. The first try apparently didn't take
 
== 2015-10-29 ==
* 23:44 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s)
* 23:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s)
* 23:32 bd808: Restarted logstash on logstash1003; died with OOM error
* 23:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s)
* 23:21 mutante: radium: scheduled downtime, reinstalling
* 23:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s)
* 23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s)
* 23:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s)
* 23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s)
* 23:09 awight: set dashboard cache interval to 60m
* 22:33 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s)
* 22:31 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s)
* 22:30 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s)
* 22:29 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s)
* 22:05 awight: CiviCRM upgrade to 4.6.9 succeeded; new data is backed up
* 22:02 gwicke: restbase: switched  local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS)
* 21:40 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s)
* 21:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s)
* 21:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s)
* 20:56 gwicke: restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS)
* 20:47 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 19:50 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I06879b6e6e: Enable ScribuntoGatherFunctionStats (duration: 00m 17s)
* 19:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.3
* 19:39 twentyafterfour: rolling back to 1.27.0-wmf.3 due to increase in log errors
* 19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.4
* 18:59 cmjohnson1: powering down aqs1001 for h/w maintenance
* 17:53 AaronSchulz: Touched/synced PrivateSettings.php symlink via touch -h
* 17:47 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 18s)
* 17:36 AaronSchulz: Did touch of InitialiseSettings.php
* 17:33 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
* 17:32 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 17s)
* 17:17 logmsgbot: kartik@tin Synchronized wmf-config/PrivateSettings.php: Really sync right file this time (duration: 00m 17s)
* 17:10 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Retry syncing for CX token (duration: 00m 17s)
* 17:04 awight: restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC
* 17:02 awight: update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78
* 16:57 YuviPanda: restart etherpad on etherpad1001
* 16:22 moritzm: removed obsolete mysql 5.5 packages on mw102[2-9], mw1032, mw1053, mw1114, mw1163
* 16:19 _joe_: moved purge_abusefilter from terbium to mw1152
* 15:48 _joe_: moving purge_checkuser from terbium to mw1152
* 15:43 _joe_: moving purge_securepoll from terbium to mw1152
* 15:15 moritzm: removed openjdk-7 on restbase100[1-9] (it's using openjdk-8 for a while)
* 13:17 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Fix ContentTranslationCXServerAuth for CX (duration: 00m 18s)
* 13:14 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Fix name of JWT token (duration: 00m 18s)
* 12:33 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Set ContentTranslationCXServerAuth for CX (duration: 00m 17s)
* 11:24 mobrovac: restbase rolling-restart after config change https://gerrit.wikimedia.org/r/249465
* 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 at 100% load. Reduce db1035 weight. (duration: 00m 17s)
* 10:34 _joe_: migrated jobqueue_stats_reporter to mw1152
* 10:21 hashar: restarting Jenkins (java upgrade)
* 10:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance (duration: 00m 18s)
* 10:14 hashar: Upgrading java on gallium and restarting Jenkins
* 10:04 moritzm: removed openjdk-7 from cassandra test hosts (now using openjdk-8)
* 10:01 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable nearby on wikidata (duration: 00m 18s)
* 10:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Config for fetching labels on mobile wikidata (duration: 00m 18s)
* 09:51 jynus: shutdown mw1083 to avoid querying the mysql servers with an outdated config/spaming the error logs
* 09:13 jynus: restarting db1065 for regular maintenace
* 08:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 17s)
* 07:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 29 07:04:04 UTC 2015 (duration 4m 3s)
* 06:43 YuviPanda: disabled two fr-tech related catchpoint tests per awight's request
* 06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 29ccbd248 (duration: 00m 17s)
* 05:46 ebernhardson: finished manually running 3M enwiki/enqueue, enwiki/wikibase-addUsagesForPage, and wikidatawiki/cirrusSearchLinksUpdatePrioritized jobs from mw1011 and mw1012
* 05:38 ebernhardson: restarting elasticsearch on nobelium to attempt to clear up extra log GC pauses in the old generation (50s+)
* 05:37 ebernhardson: restarting elasticsearch on nobelium
* 03:55 gwicke: cassandra *staging*: testing DateTieredCompactionStrategy (https://labs.spotify.com/2014/12/18/date-tiered-compaction/) on wikipedia html and data-parsoid tables
* 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-29 03:05:52+00:00
* 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 03s)
* 02:59 YuviPanda: start pybal on lvs1007
* 02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-29 02:46:27+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 47s)
* 02:05 AaronSchulz: Restarted stuck hhvm on mw1016; dump at /tmp/hhvm.25097.bt
* 01:06 gwicke: lowered gc_grace on wikipedia parsoid html and data-parsoid keyspaces to 24 hours
 
== 2015-10-28 ==
* 23:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Translate/tag/PageTranslationHooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 23:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249642 (duration: 00m 17s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 17s)
* 23:22 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 17s)
* 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
* 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 18s)
* 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
* 23:15 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/249585 (duration: 00m 19s)
* 23:08 logmsgbot: ebernhardson@tin Synchronized portals: Switch www portals to be deployed from Git, but not being served from anywhere yet (duration: 00m 18s)
* 23:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 22:57 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/includes/Hooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 22:38 urandom: Cassandra cleanup on restbase-test2001-a
* 22:31 awight: Fundraising CRM db migration started, estimated completion 10:15 UTC (tomorrow)
* 22:11 awight: update fundraising CRM to civi-4.6.9-deployment branch, from f2fa7b942625b34ede520e11f20e7e0835ecb17d to 23d2020448f343c8c1b2f4d779be66f57552f935
* 21:11 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.4/includes/changes/EnhancedChangesList.php: Fix diff/history links not showing up for ungrouped enhanced RC - https://gerrit.wikimedia.org/r/#/c/249556/ (duration: 00m 19s)
* 21:08 YuviPanda: enable puppet on labstore1001
* 20:55 YuviPanda: reverted changes to nsswitch.conf from puppet run manually on labstore2001
* 20:29 YuviPanda: disable puppet on labstore1001
* 20:20 awight: Disabled fundraising CiviCRM
* 19:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.4
* 18:50 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiversity
* 18:42 logmsgbot: legoktm@tin Synchronized wmf-config/mc.php: https://gerrit.wikimedia.org/r/#/c/249463/ (duration: 00m 17s)
* 18:41 ostriches: rolled scap back to master@62a250a, needs puppet changes before new code goes live
* 18:34 gwicke: restbase deploy done
* 18:34 yurik: updated kartotherian & tilerator services
* 18:23 gwicke: rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster
* 18:18 gwicke: canary deploy of restbase deploy 3b1f6488f2 to restbase1001
* 18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hiwiki
* 18:12 Krinkle: mwscript deleteEqualMessages.php --wiki fawiki
* 17:59 godog: rolling-restart cassandra-metrics-collector, staggered this time
* 17:23 ostriches: deployed scap master@f823129 to cluster
* 17:16 ostriches: deployed scap master@abe1973 to cluster
* 16:54 godog: unblacklist 'max' cassandra metrics and restart cassandra-metrics-collector
* 16:54 moritzm: installed openjdk security updates on zookeeper hosts
* 16:18 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: add settings for displaying labels on test.wikidata in mobile (duration: 00m 18s)
* 16:15 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: enable wikibase descriptions on test.wikidata (duration: 00m 17s)
* 15:46 andrewbogott: testing the bot after the nfs move
* 15:39 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s)
* 14:22 kart_: T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki)
* 14:21 moritzm: repooled restbase1006
* 14:09 moritzm: depooled restbase1006 for kernel update
* 14:05 moritzm: repooled restbase1005
* 13:54 moritzm: depooled restbase1005 for kernel update
* 13:47 moritzm: repooled restbase1004
* 13:35 moritzm: depooled restbase1004 for kernel update
* 13:34 moritzm: repooled restbase1003
* 13:16 moritzm: depooled restbase1003 for kernel update
* 13:12 moritzm: repooled restbase1002
* 12:56 moritzm: depooled restbase1002 for kernel update
* 11:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s)
* 10:39 moritzm: updated kernel on restbase1001 to latest 3.19
* 10:37 _joe_: manually removed crontab from mw1152, erroneously created by puppet
* 10:11 _joe_: reimaging mw1152
* 09:15 _joe_: preparing to reimage mw1152, disabling puppet, scheduling downtime.
* 06:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s)
* 05:15 legoktm: ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium
* 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00
* 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s)
* 02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s)
* 00:55 yurik: deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/
* 00:37 logmsgbot: tgr@tin Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s)
* 00:35 mutante: mw1135 temp. unresponsive - OOM killer killing hhvm
* 00:27 mutante: powercycling unresponsive mw1127
 
== 2015-10-27 ==
* 23:53 logmsgbot: tgr@tin Started scap: Updating MediaViewer with r246112
* 23:13 ori: running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7)
* 23:12 gwicke: ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state
* 23:11 gwicke: rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies
* 23:06 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s)
* 23:01 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s)
* 22:39 mutante: powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248
* 21:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s)
* 21:50 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s)
* 21:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4
* 21:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s)
* 19:46 logmsgbot: twentyafterfour@tin Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s)
* 19:44 bd808: Ran sync-common on mw2187 to rebuild l10n caches
* 19:38 ejegg: enabled donation queue consumer
* 19:15 logmsgbot: twentyafterfour@tin Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch
* 19:09 awight: updated DjangoBannerStats from 57a0392b3f43b65050b01a0465e120ed609a769e to 12e819a04a40ee6fab5dd55fcaf072661df31106
* 17:56 paravoid: updating jessie debian-installer to 20150422+deb8u2
* 17:50 Jeff_Green: add fundraising-banner-logger hosts to icinga/nsca
* 17:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s)
* 17:37 ejegg: disabled donation queue consumer to build up queue for benchmark
* 17:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s)
* 16:07 Jeff_Green: round of fundraising OS updates, occasional icinga noise is expected
* 16:02 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language [[gerrit:249081]] (duration: 00m 17s)
* 15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia [[gerrit:248910]] (duration: 00m 17s)
* 15:15 godog: reenable puppet on graphite1001
* 15:01 kart_: TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki)
* 13:00 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s)
* 12:59 akosiaris: disabling puppet and bringing down OTRS service on mendelevium
* 12:21 jynus: Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables.
* 11:56 godog: reimage restbase-test2003
* 11:23 godog: cassandra OOM'd on restbase1007, restarting
* 10:57 godog: downtime restbase endpoints health for restbase1* while investigating
* 10:56 hashar: Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-(
* 10:50 hashar: stopping Jenkins due to an unclean state
* 10:11 jynus: disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs
* 09:55 godog: convert restbase-test2003 to cassandra multi-instance
* 09:08 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s)
* 05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s)
* 03:42 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00
* 02:24 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s)
* 01:02 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s)
* 00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s)
* 00:11 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s)
 
== 2015-10-26 ==
* 23:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248371/ (duration: 00m 20s)
* 23:22 logmsgbot: krenair@tin Synchronized wmf-config/extension-list-labs: https://gerrit.wikimedia.org/r/#/c/248632/ (duration: 00m 17s)
* 23:19 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/vecwiki.png: https://gerrit.wikimedia.org/r/#/c/248633/ (duration: 00m 18s)
* 23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248478/ (duration: 00m 17s)
* 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
* 23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
* 23:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248871/3 (duration: 00m 18s)
* 22:39 ejegg: updated payments from 7fd1e880b6a45d79fc305b39f3fee2f324324136 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
* 21:48 cmjohnson1: powering off wdqs1001 to update idrac settings
* 21:33 ejegg: update payments from 71d2d927f4efac1a639aaf7627c765f48d1b129c to 7fd1e880b6a45d79fc305b39f3fee2f324324136
* 20:50 subbu: deployed parsoid version 660c59a9
* 20:21 ebernhardson: started copy of eqiad elasticsearch indices to noeblium
* 20:19 YuviPanda: stress test on nobelium complete, CPU temperature didn't go above 65C
* 20:13 paravoid: deactivating ulsfo<->NTT BGP peering due to upcoming network migration
* 20:05 YuviPanda: running stress on nobelium
* 19:36 cmjohnson1: swapped bad disk on db1030
* 15:44 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s)
* 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s)
* 15:31 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s)
* 15:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
* 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
* 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s)
* 15:06 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s)
* 15:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s)
* 14:58 bblack: repooling cp1059 varnish mobile frontend (wiped)
* 14:18 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s)
* 14:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s)
* 13:32 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s)
* 13:27 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s)
* 09:42 dcausse: deleting unused elasticsearch indices in eqiad (T112863)
* 09:20 _joe_: restarting etcd on conf1001
* 09:16 jynus: rebooting and installing jessie on db2060-db2070
* 05:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
 
== 2015-10-25 ==
* 05:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 25 05:36:33 UTC 2015 (duration 36m 32s)
* 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-25 02:28:08+00:00
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 59s)
 
== 2015-10-24 ==
* 19:34 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium
* 06:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s)
* 05:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s)
* 05:11 hoo: Set an email address for user "Ymnes", after request. Confirmed by several, including.
* 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00
* 02:33 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s)
* 00:21 ebernhardson: started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads
 
== 2015-10-23 ==
* 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki hewikisource
* 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki ukwiki
* 23:09 hoo: Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer
* 22:23 robh: powercycled nobelium
* 21:24 mutante: nobelium - powercycled, no console output
* 20:04 jynus: rename user job had created lag on almost all enwiki dbs, things should be better now
* 19:08 urandom: starting nodetool cleanup on restbase-test2001-b
* 19:07 urandom: starting nodetool cleanup on restbase-test2001-a
* 18:16 godog: bounce grafana-server on krypton
* 17:58 YuviPanda: moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch
* 17:57 godog: delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries
* 17:53 YuviPanda: stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856
* 17:13 akosiaris: enable_notifications=0 in neon's icinga for a few mins while the storm dies down
* 17:04 cwd: updated payments from 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c to 71d2d927f4efac1a639aaf7627c765f48d1b129c
* 16:47 godog: stop puppet on graphite1001 and graphite-index cron, suspected root cause
* 16:26 godog: uwsgi timing out while serving requests, bounce also carbon daemons
* 16:20 godog: bounce uwsgi for graphite-web on graphite1001
* 16:14 urandom: starting rebuild of restbase-test2002-b
* 15:42 jynus: copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org
* 15:42 urandom: bouncing restbase-test2002-a Just To See
* 15:06 urandom: running nodetool cleanup on restbase-test2003
* 14:47 godog: remove outdated cassandra metrics from graphite2001
* 14:27 godog: remove  restbase-test2001 restbase-test2002 cassandra metrics
* 14:12 mobrovac: restbase rolling-restarting after applying aab840f
* 13:55 jynus: Rebooting and installing jessie on db2055-db2059
* 13:45 godog: remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
* 13:34 godog: remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
* 11:51 godog: roll-restart cassandra-metrics-collector on restbase cluster after  https://gerrit.wikimedia.org/r/248313
* 10:27 jynus: retrying schema change on EventLogging (db1046) after failure
* 10:21 jynus: End of online schema change to geo_tags; all wikis on dblist have been updated
* 09:34 godog: reimage restbase-test2002.codfw.wmnet
* 09:23 jynus: restarting schema change on geo_tags for all wikis
* 09:17 mobrovac: restbase rolling-restart after config changes
* 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s)
* 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s)
* 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s)
* 03:15 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s)
* 00:10 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s)
 
== 2015-10-22 ==
* 23:58 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s)
* 23:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s)
* 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 23:12 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 22:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s)
* 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
* 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
* 21:11 logmsgbot: reedy@tin Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s)
* 21:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s)
* 21:05 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s)
* 20:39 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s)
* 20:38 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s)
* 20:26 bd808: Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview
* 20:12 bd808: Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover)
* 19:53 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner: Deploying ZeroBanner T116309 patch 248116 (duration: 00m 18s)
* 19:28 bd808: Forced ELK Elasticsearch to allocate replica of logstash-2015.10.22 shard 0 on logstash1004
* 18:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.27.0-wmf.3
* 16:48 bd808: Restarted Elasticsearch on logstash1004
* 16:39 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: T116134: Set CX JWT token (duration: 00m 17s)
* 16:19 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f690589: Increase abusefilter emergency disable threshold on MediaWiki.org (duration: 00m 17s)
* 15:30 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf/highlight.php: SWAT: Remove urlencode from phabricator links [[gerrit:248049]] (duration: 00m 17s)
* 15:13 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf: SWAT: noc: change Gitblit links to Diffusion [[gerrit:248027]] (duration: 00m 17s)
* 15:11 moritzm: uploaded openjdk-8 8u66-b17 for jessie-wikimedia to carbon
* 14:46 jynus: performing schema change on eventlogging database on db1046
* 14:41 mutante: mw1083 - Error: Could not run Puppet configuration client: Read-only file system
* 14:25 jynus: setting thread_pool_size to 32 dynamically on all MariaDB hosts
* 14:01 jynus: performing schema change on officewiki-flow (s3)
* 13:19 jynus: performing schema change on x1-master (flowdb)
* 12:31 jynus: Rolling schema change for GeoData on all wikis (geo_tags)
* 11:35 mobrovac: restbase deployed 2bc05f40
* 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 00m 17s)
* 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 07m 33s)
* 06:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 22 06:17:39 UTC 2015 (duration 17m 38s)
* 06:14 AaronSchulz: Restarted hhvm on mw1011, it was stuck doing nothing at 100 cpu
* 05:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksUpdate.php: fe323f9b68bbb (duration: 00m 17s)
* 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-22 03:05:21+00:00
* 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 52s)
* 02:37 AaronSchulz: Started running 5 threads of enwiki refreshLinks jobs on tin
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-22 02:35:41+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 44s)
* 01:10 logmsgbot: krenair@mira Synchronized README: testing sync from mira (duration: 00m 17s)
* 01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
* 01:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
* 00:56 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
* 00:55 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 17s)
* 00:49 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
* 00:20 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibf752b832: $wgMathCheckFiles = false (duration: 00m 18s)
* 00:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
* 00:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
 
== 2015-10-21 ==
* 23:33 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 19s)
* 23:32 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 18s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Provide category suggestions from the right wiki (duration: 00m 17s)
* 23:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Rearrange messages to always display license name (duration: 00m 18s)
* 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throtle exception for eswiki (duration: 00m 18s)
* 23:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
* 23:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
* 21:10 ebernhardson: starting copy of elasticsearch eqiad indices to codfw
* 20:46 ebernhardson|lch: cancel copying elasticsearch eqiad to labsearch, looks to be writing to wrong disks and will fill up
* 20:16 AaronSchulz: Started running 8 threads of commonswiki refreshlinks jobs on terbium
* 20:14 ebernhardson|lch: starting copy of elasticsearch indices from eqiad cluster to labsearch cluster
* 18:42 ebernhardson: initializing elasticsearch index mapping for all wikis in the codfw and labsearch ES clusters
* 18:12 ejegg|mtg: updated crm from 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435 to f2fa7b942625b34ede520e11f20e7e0835ecb17d
* 16:53 hoo: Attached WeiaR@enwiki to the global account of the same name. T115699
* 15:50 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
* 15:49 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
* 15:38 akosiaris: depooled mw1083
* 15:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: touch and re-sync InitialiseSettings.php to bust cache (duration: 00m 17s)
* 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:18 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 17s)
* 15:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 19s)
* 15:00 aude: rsync failed on mw1083: "failed to set times on "/srv/mediawiki/wmf-config": Read-only file system"
* 14:49 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on test.wikidata (duration: 00m 18s)
* 14:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: enabled geodata on beta wikidata (duration: 00m 18s)
* 14:08 jynus: applying schema change to testwikidatawiki (s3) and wikidatawiki(s5)
* 13:56 jynus: performing schema change on testwiki.geo_tags
* 13:35 jynus: stopping and fixing replication on labsdb1004 (not in production)
* 08:40 mobrovac: restbase deployment of 3006b77e {{done}}
* 08:32 mobrovac: restbase deploying 3006b77e
* 07:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 21 07:36:04 UTC 2015 (duration 36m 3s)
* 03:13 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-21 03:13:42+00:00
* 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 24s)
* 02:42 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-21 02:42:44+00:00
* 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 05s)
* 01:12 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
* 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: revert (duration: 00m 18s)
* 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
* 00:29 logmsgbot: legoktm wikidata'd everything
* 00:22 logmsgbot: hoo@tin Synchronized dblists/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 17s)
* 00:19 logmsgbot: hoo@tin Synchronized wmf-config/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 19s)
 
== 2015-10-20 ==
* 23:54 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (duration: 00m 18s)
* 23:47 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (testwikidata) (duration: 00m 18s)
* 23:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEventsHooks.php: https://gerrit.wikimedia.org/r/#/c/247650/ (duration: 00m 17s)
* 23:26 logmsgbot: hoo@tin Synchronized wmf-config/: Revert "Temporarily disable "item-merge" right on Wikidata" (duration: 00m 18s)
* 23:25 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred: 2a1e1d7dd88a62aba9 (duration: 00m 17s)
* 23:22 logmsgbot: hoo@tin Synchronized wmf-config/: Bump the cache epoch for (test)wikidata (duration: 00m 18s)
* 23:20 logmsgbot: hoo@tin Finished scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies (duration: 48m 44s)
* 23:16 mutante: mw1232: restarted hhvm
* 22:31 logmsgbot: hoo@tin Started scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies
* 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 17s)
* 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 18s)
* 20:23 gwicke: reverted restbase deploy on restbase1001 to a4c55e40
* 20:15 mutante: starting restbase on restbase1001
* 20:09 mutante: errors argon' on argon
* 20:04 ottomata: uninstalling hadoop packages on analytics1017
* 19:46 ori: previous syncs was of I46200d4edb3: Revert "Revert "Revert "Enable config for all three search clusters, but only write to eqiad"""
* 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
* 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/: (no message) (duration: 00m 18s)
* 18:25 mutante: re-enabling icinga notifications for icinga (neon) itself that were disabled for some reason though all OK
* 16:50 mutante: temp. disabled ircecho / neon puppet
* 16:50 jynus: enabling query profiling on a sample of queries on db1072
* 16:27 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Revert Temporarily increase redis logging to debug (duration: 00m 18s)
* 16:25 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily increase redis logging to debug (duration: 00m 17s)
* 16:21 aude: re-populated sites table on metawiki, mediawikiwiki and specieswiki with https protocol links
* 16:13 cwdent: updated fundraising crm from 738e8c3f8079765841ed4c5f79ecf066c541c7b9 to 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435
* 15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part V (duration: 00m 18s)
* 15:48 godog: bump netdev_max_backlog to 10000 on graphite1001, T101141
* 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part IV (duration: 00m 17s)
* 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part III (duration: 00m 17s)
* 15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part II (duration: 00m 17s)
* 15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part I (duration: 00m 17s)
* 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation suggestion in all Wikipedia [[gerrit:247515]] (duration: 00m 17s)
* 15:23 awight: update fundraising CRM from 6ad0ad090c23ee41003138a6676131abf70c72f4 to 738e8c3f8079765841ed4c5f79ecf066c541c7b9
* 15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch graphoid to the local restbase proxy [[gerrit:247494]] (duration: 00m 17s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Move ForeignUploadTargets config to production [[gerrit:246703]] part 2 (duration: 00m 17s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Move ForeignUploadTargets config to production [[gerrit:246703]] part I (duration: 00m 18s)
* 14:33 cmjohnson1: removing tele2(patchid 2953) from dmarc panel @eqiad
* 14:12 ottomata: deployed varnishreqstats diamond collector to remaining varnish caches
* 13:48 jynus: backing up and renaming user_daily_contribs table from all wikis as a previous step for its deletion
* 12:44 kart_: Update cxserver to 6452b68
* 12:01 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Temporarily disallow item-merge until T115892 is resolved (duration: 00m 19s)
* 11:16 mobrovac: mathoid deploying 8e1a3327
* 07:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 20 07:23:56 UTC 2015 (duration 23m 55s)
* 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-20 03:14:15+00:00
* 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 16s)
* 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Bring phase0 and phase1 inline with phase2 (duration: 00m 18s)
* 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Elastica/: Bring phase0 and phase1 inline with phase2 (duration: 00m 21s)
* 02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-20 02:43:53+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 20s)
 
== 2015-10-19 ==
* 23:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: testwiki Graphoid to restbase (duration: 00m 17s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/: resync after touching InitialiseSettings.php to bust caches (duration: 00m 18s)
* 23:12 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Revert multidc cirrussearch config, seeing unexplained errors on commonswiki (duration: 00m 18s)
* 23:11 mutante: restarted hhvm on mw1231
* 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throttle exception for dewiki (duration: 00m 17s)
* 23:08 mutante: restarted apache on mw1231
* 23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Disable cirrus suggester AB test (duration: 00m 17s)
* 23:05 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Bump sampling rate of common terms test from 1:1000 to 1:200 (duration: 00m 17s)
* 23:01 logmsgbot: ebernhardson@tin Synchronized tests/: noop sync mediawiki-config test dir (duration: 00m 17s)
* 22:42 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable cirrusearch multi cluster configuration, only write to eqiad (duration: 00m 18s)
* 22:40 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Handle ElasticaWrite job failures internally (duration: 00m 18s)
* 22:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Deploy multi-dc cirrusearch code for CirrusSearch extension (duration: 00m 18s)
* 22:28 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Elastica/: Deploy multi-dc cirrusearch code for Elastica extension (duration: 00m 17s)
* 22:20 ebernhardson: sync-common on mw1017 to pre-test cirrussearch multi-dc deployment
* 22:17 paravoid: salt-run deluser --delete-home gmetric; delgroup systemusers
* 20.23 subbu: cherrypick deploy for parsoid completed: b317f33f and 60a82ae0 cherrypicked from parsoid master
* 15:42 logmsgbot: anomie@tin Started scap: SWAT: Add a change tag to cross-wiki uploads [[gerrit:246701]]
* 15:39 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref [[gerrit:247255]] (duration: 00m 18s)
* 15:30 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref [[gerrit:247256]] (duration: 00m 18s)
* 15:25 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob [[gerrit:247268]] (duration: 00m 18s)
* 15:24 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob [[gerrit:247267]] (duration: 00m 17s)
* 15:12 urandom: deploying a4c55e40 to RESTBase
* 15:11 joal: Scheduling icinga downtime for CQL checks on aqs while heavily loading data - joal (me) babysites the jobs - 1 day downtime, will reiterate tomorrow if needed
* 15:09 ottomata: enabling varnish reqstats diamond collector on all upload caches
* 15:09 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Search/Connection.php: SWAT: Backport [[gerrit:246134]] because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
* 15:07 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/includes/Search/Connection.php: SWAT: Backport [[gerrit:246134]] because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
* 14:52 akosiaris: disabled puppet on maps-test200{1,2,4}. Debugging cassandra multi-instance setup aftermath. Not to be enabled
* 14:34 urandom: canary deploy (a4c55e40) to restbase1001.eqiad
* 12:47 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl-eng_0.1.0~r20599-1
* 12:22 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl_0.1.0-1
* 06:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 19 06:19:09 UTC 2015 (duration 19m 8s)
* 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-19 03:04:49+00:00
* 02:59 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 40s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-19 02:36:53+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 15s)
 
== 2015-10-18 ==
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-mlt-ara_0.1.0~r57554-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-is-sv_0.1.0~r56030-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-ro_0.7.3~r57551-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-it_0.1.0~r51165-1
* 06:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 18 06:11:37 UTC 2015 (duration 11m 36s)
* 02:57 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-18 02:57:48+00:00
* 02:52 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-18 02:29:50+00:00
* 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 25s)
 
== 2015-10-17 ==
* 20:02 godog: powercycle analytics1034, no console no ssh
* 14:05 godog: reboot krypton, unable to ssh and no console (VM) iowait through the roof
* 06:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 17 06:27:04 UTC 2015 (duration 27m 3s)
* 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-17 03:02:58+00:00
* 02:58 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-17 02:35:14+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 09s)
 
== 2015-10-16 ==
* 20:52 hashar: Restarting Jenkins to remove potential dead locks before the week-end
* 20:48 ejegg: updated payments from 33b3bd6bee11b3cc9de1570584a23354d0b6525f to 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c
* 19:29 ori: deleted /var/lib/carbon/whisper/MediaWiki/MediaWiki on graphite1001 & graphite2001 per tgr's request
* 16:14 awight: Update paymentswiki logging config per T107918
* 15:52 awight: Fundraising: remove FR fraud exception
* 13:18 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I14ecd0ae87: Turn off UserDailyContribs extension (duration: 00m 18s)
* 06:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 16 06:25:56 UTC 2015 (duration 25m 55s)
* 03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-16 03:11:14+00:00
* 03:06 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
* 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-16 02:43:42+00:00
* 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 38s)
* 02:27 jynus: repairing table and restarting replication on s7 from labsdb1002
* 02:23 jynus: repairing table and restarting replication on s5 from dbstore1002 (non-production host)
* 01:36 Jamesofur: reset email address for User:INeverCry after identify verification
 
== 2015-10-15 ==
* 22:09 greg-g: don't worry, _joe_ was around and we approved Roan's last deploy as an exception :)
* 22:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Back out categories in sidebar feature (duration: 00m 20s)
* 20:30 _joe_: rebooting ms-be1011
* 19:41 awight: update crm from ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1 to a4e74f6f38e6cff16ecf79d28d6e4de9499a5017
* 19:08 cwdent: updated payments from bdec3220030a396e2a447763e40b940a332e2ab8 to 33b3bd6bee11b3cc9de1570584a23354d0b6525f
* 18:41 ejegg: updated refund queue name in IPN listener settings
* 18:13 ejegg: updated CiviCRM from 17dc351d92a8437c93b4a9fa2385840b3581dad6 to ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1
* 17:18 cwdent: updated payments from ba8f80ec7a858074cd3856a52f3758cf96571f67 to bdec3220030a396e2a447763e40b940a332e2ab8
* 09:54 _joe_: restarted gitblit, unresponsive
* 05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 15 05:41:25 UTC 2015 (duration 41m 24s)
* 02:51 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-15 02:51:27+00:00
* 02:49 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 05m 02s)
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-15 02:35:15+00:00
* 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 57s)
 
== 2015-10-14 ==
* 23:46 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
* 22:26 akosiaris: restarted zotero on sca1002
* 21:54 akosiaris: restarted zotero on sca1001. complaining out of memory in logs
* 20:46 ejegg: updated civicrm from ba39f3181431c8409416e580dcf6e15ac5f96a21 to 17dc351d92a8437c93b4a9fa2385840b3581dad6
* 20:20 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
* 19:52 robh: reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
* 19:50 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
* 19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
* 19:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
* 19:10 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
* 18:06 logmsgbot: demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
* 17:59 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
* 17:58 logmsgbot: demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
* 15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target [[gerrit:246243]] (duration: 00m 18s)
* 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246225]] (duration: 00m 17s)
* 15:14 ori: restarted navtiming and statsv services on hafnium
* 15:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246226]] (duration: 00m 18s)
* 15:05 Krinkle: Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
* 14:24 ori: cleaned up tessera leftovers on graphite1001
* 14:15 mutante|1way: mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
* 13:30 ottomata: kafka-preferred-replica election after kafka1012's broker restarted last night
* 13:12 ottomata: restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
* 12:55 hashar: ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25  (from labs instances)
* 12:32 logmsgbot: krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
* 12:07 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
* 03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
* 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
* 03:07 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
* 02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
* 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
* 00:19 ejegg: updated civicrm from c7af7634e75eb8702f5e16081a86ab86ce69c7c2 to ba39f3181431c8409416e580dcf6e15ac5f96a21
 
== 2015-10-13 ==
* 23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default (duration: 01m 14s)
* 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Turn on cirrus common terms test (duration: 01m 15s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Add information for common terms ab test (duration: 01m 15s)
* 23:21 Krenair: restarted apache on silver
* 23:19 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Flow: Make flow board descriptions editable again (duration: 01m 16s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Flow: Bump flow submodule in 1.27.0-wmf.3 (duration: 01m 15s)
* 23:10 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Update WME for common terms AB test (duration: 01m 14s)
* 23:08 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Update cirrus for common terms AB tes (duration: 01m 15s)
* 23:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn on logging of the "warning" channel (duration: 01m 13s)
* 23:00 ejegg: updated SmashPig from b5ff2a7d5f17aaaa33a169ca101cbea639769c90 to c431c8d77521270236c72532a50806b2e852cf7b
* 21:05 ottomata: reenabling puppet on cp1052
* 20:51 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
* 20:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/db: I0e5f2d3b2: Revert Enforce lagged-slave read-only mode on the DB layer (duration: 01m 14s)
* 20:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/autoload.php: (no message) (duration: 01m 14s)
* 20:40 akosiaris: restart gitblit (once more)
* 20:27 _joe_: rebooting mw1157, stuck in a kernel soft lockup
* 20:22 ori: locally hacked testwiki and mediawikiwiki to point to php-1.27.0-wmf.3 on mw1017
* 20:19 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 20:18 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 20:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
* 19:59 ejegg: updated civicrm f25fbe856b92373104985185db77311ea3a4d841 to c7af7634e75eb8702f5e16081a86ab86ce69c7c2
* 19:38 JohnFLewis: delete wikidata-l mailing list (archives still accessible)
* 19:30 logmsgbot: twentyafterfour@tin Finished scap: Full scap sync for 1.27.0-wmf.3 (duration: 31m 26s)
* 18:58 logmsgbot: twentyafterfour@tin Started scap: Full scap sync for 1.27.0-wmf.3
* 18:45 ottomata: FYI i have puppet disabled on cp1052 while I try to figure out a diamond+VSL+multiprocessing bug
* 18:11 mutante|1way: started salt on mw2083
* 18:09 mutante|1way: restarted gitblit
* 17:58 ori: testing ldap integration in grafana 2
* 17:54 urandom: performing deploy of a4c55e4 to restbase staging
* 17:42 urandom: performing canary deploy of a4c55e4 to restbase staging (xenon)
* 16:14 ostriches: created education program tables for srwiki, T110619
* 16:14 mobrovac: restbase deploying a01c62a6
* 16:08 ottomata: restarting diamond on cp1052 in gdb in attempt to figure out why vanrishreqstats segfaults...
* 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions for de, fa, fi, he, nn, pa, pl and te wikipedias [[gerrit:245862]] (duration: 01m 13s)
* 15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default" (duration: 01m 13s)
* 15:15 ori: grafana-test: Imported Grafana dashboards from ElasticSearch
* 14:12 bblack: note to self: we should migrate all our service to Java
* 14:11 bblack: restarted gitblit on antimony
* 13:16 bblack: repooling cp2017 (codfw upload)
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 13 02:32:17 UTC 2015 (duration 32m 16s)
* 02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-13 02:32:17+00:00
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 01s)
* 01:53 logmsgbot: faidon@tin Synchronized wmf-config/CommonSettings.php: unbreak BounceHandler (duration: 01m 14s)
* 00:51 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Reapply: Flow-occupy talk namespaces on sewikimedia (duration: 01m 13s)
* 00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/modules/flow-initialize.js: https://gerrit.wikimedia.org/r/#/c/245596/ (duration: 01m 13s)
* 00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244724/ (duration: 01m 13s)
* 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245608/ (duration: 01m 13s)
* 00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 13s)
* 00:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245588/ and https://gerrit.wikimedia.org/r/#/c/245589/ (duration: 01m 13s)
 
== 2015-10-12 ==
* 23:53 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s)
* 23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s)
* 23:35 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s)
* 23:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s)
* 23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s)
* 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s)
* 23:11 logmsgbot: krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s)
* 23:06 logmsgbot: krenair@tin Synchronized database lists: (no message) (duration: 01m 13s)
* 20:21 bearND: MobileApps deployed sha1 95293e5
* 19:09 Tim: on ruthenium installed iotop for stall investigation
* 16:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed [[gerrit:244737]] (duration: 01m 13s)
* 16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 [[gerrit:245472]] (duration: 01m 13s)
* 15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki [[gerrit:244953]] (duration: 01m 14s)
* 15:39 logmsgbot: thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist [[gerrit:244435]] (duration: 01m 12s)
* 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes [[gerrit:241079]] (duration: 01m 14s)
* 15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces [[gerrit:242096]] (duration: 01m 13s)
* 15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis [[gerrit:244165]] (duration: 01m 13s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata [[gerrit:244591]] (duration: 01m 14s)
* 15:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary [[gerrit:244649]] (duration: 01m 12s)
* 14:54 dcausse: closing unused cirrus indices in eqiad (T112863)
* 14:28 cmjohnson: rebooting mw1154
* 12:10 yurik: deployed kartotherian & tilerator to maps-test200{1-4}
* 08:54 hashar: zuul-merger process leaked file descriptors and ended up unable to open any more files.  Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243
* 08:44 hashar: Zuul CI in trouble.  zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243
* 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s)
* 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s)
 
== 2015-10-11 ==
* 04:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s)
* 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00
* 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s)
 
== 2015-10-10 ==
* 06:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s)
* 02:41 ejegg|afk: enabled fundraising banner campaigns
* 02:36 ejegg|afk: updated payments-wiki from 24d5be6886d34b3600031290c7f55ee84f3dcee2 to ba8f80ec7a858074cd3856a52f3758cf96571f67
* 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s)
 
== 2015-10-09 ==
* 22:20 ejegg: took down Fundraising campaigns
* 19:33 logmsgbot: ori@tin Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s)
* 18:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s)
* 16:04 godog: rolling restart cassandra test cluster T95253
* 15:35 urandom: performing Cassandra cleanup on restbase-test2003.codfw
* 15:15 urandom: bouncing Cassandra on restbase-test2001-a
* 15:07 urandom: starting nodetool cleanup on restbase-test2002
* 14:37 jynus: more configuration testing (with puppet disabled) and several mysql restarts on db1022
* 13:53 hashar: Restarted Zuul, had a deadlocked job
* 11:50 godog: bounce mathoid on sca100[12], stray instance found not running firejail
* 11:22 mobrovac: restbase deploying aaee7c31
* 11:16 godog: force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/
* 11:00 jynus: restarting db1022's mysql (depooled) for configuration testing
* 09:17 jynus: deployed visual glitch fix to dbtree
* 09:11 akosiaris: poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
* 09:11 akosiaris: poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
* 08:20 ori: Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/
* 05:13 mutante: @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s)
* 01:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s)
* 01:18 RoanKattouw: Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh
* 01:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s)
 
== 2015-10-08 ==
* 23:56 logmsgbot: catrope@tin Finished scap: SWAT (duration: 29m 10s)
* 23:51 yurik: reverted Kartotherian to HEAD^^ - the service wouldn't start
* 23:27 logmsgbot: catrope@tin Started scap: SWAT
* 23:06 yurik: deployed kartotherian
* 18:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2
* 16:45 mobrovac: mathoid deploying 110abaf
* 15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) [[gerrit:244332]] (duration: 00m 18s)
* 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki [[gerrit:244142]] (duration: 00m 17s)
* 14:53 paravoid: replacing cr1-codfw<->asw-a-codfw QSFPs
* 14:48 logmsgbot: reedy@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
* 14:13 jynus: performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002)
* 13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s)
* 13:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s)
* 12:09 godog: update facts on puppet compiler from palladium
* 11:38 _joe_: rebooting mc2002
* 10:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s)
* 10:37 _joe_: rebooting mw1008, in oom spiral
* 10:15 paravoid: salt rm /home/*/.ssh/authorized_keys
* 09:21 jynus: downtime for db1055 for maintenance (kernel update, mysql update, config update)
* 08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s)
* 07:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s)
* 07:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s)
* 06:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct  8 06:15:16 UTC 2015 (duration 15m 15s)
* 05:56 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s)
* 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00
* 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s)
* 01:17 twentyafterfour: phabricator is now up and running again
* 01:09 twentyafterfour: buggy update, stopping apache2 on iridium.
* 01:02 twentyafterfour: finished phabricator update
* 00:51 mutante: mw1160 - was oom-killer (convert, mw job 31497)
* 00:50 twentyafterfour: phabricator maintenance/upgrade. Expect 10 minutes downtime
* 00:44 mutante: powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt)
* 00:08 logmsgbot: krenair@tin Synchronized w/static/apple-touch/commons.png: https://gerrit.wikimedia.org/r/#/c/243685/ (duration: 00m 17s)
 
== 2015-10-07 ==
* 23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242842/ (duration: 00m 17s)
* 23:34 logmsgbot: ori@tin Synchronized wmf-config: I924d8e19e17: Make the redis cache configuration multi-DC-ready (T111575) (duration: 00m 17s)
* 23:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244177/ (duration: 00m 17s)
* 23:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244176/ (duration: 00m 17s)
* 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243837/ (duration: 00m 17s)
* 21:25 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Revert "Don't use nutcracker on wikitech" (duration: 00m 16s)
* 20:56 mutante: applied firewalling on IRCd server, rc bot still working fine, all public IRC ports as before
* 20:25 mutante: argon: installing package upgrades
* 20:24 mutante: resetting drac on argon
* 20:14 logmsgbot: ori@tin Synchronized wmf-config/session.php: Ie25c368a: Switch mw1017 to use DC-specific redis cluster names (duration: 00m 17s)
* 19:50 urandom: RESTBase deploy complete
* 19:41 urandom: doing full deploy of c20e6336 to RESTBase
* 19:29 urandom: canary deploy to restbase1001.eqiad complete
* 19:14 urandom: deploying c20e6336 to canary node restbase1001.eqiad
* 19:09 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.2
* 18:56 logmsgbot: twentyafterfour@tin Synchronized wmf-config/CommonSettings.php: fix undefined variable warning that has been spamming logs (duration: 00m 17s)
* 17:56 awight_: update fundraising crm from 7003cc38797848631d0c4d5f6ff68ab1d6118ad8 to f25fbe856b92373104985185db77311ea3a4d841
* 16:39 mutante: powercycling analytics1035
* 16:14 mobrovac: citoid deploying ec149fd5
* 16:02 logmsgbot: marktraceur@tin Synchronized wmf-config/: Adding new config variable for uploads to Commons (duration: 00m 17s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/MobileFrontend: SWAT (duration: 00m 17s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 18s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
* 15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1051 after maintenance (duration: 00m 17s)
* 15:03 logmsgbot: krenair@tin Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 17s)
* 15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 18s)
* 14:59 mobrovac: AQS restarting restbase on aqs100x
* 13:00 godog: decomission restbase-test2001 and reimage
* 12:27 paravoid: salt-rm'ing /var/lib/apt/lists/ubuntu.wikimedia.org_ubuntu_dists_trusty_main_i18n_Translation-en%5fUS
* 10:55 godog: reenable puppet on restbase / maps-test / aqs
* 10:10 _joe_: depooling cp1059 from pybal, varnish
* 08:44 godog: disable puppet on restbase, maps, aqs before merging https://gerrit.wikimedia.org/r/#/c/243127
* 08:05 moritzm: installed spice security updates on labvirt*
* 06:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct  7 06:01:01 UTC 2015 (duration 1m 0s)
* 05:25 ebernhardson: generating elasticsearch indices in codfw, should run ~3 hours
* 04:33 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Touch InitiialiSettings.php to force config regeneration (duration: 00m 18s)
* 04:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/: enable second es cluster in testwiki one more time (duration: 00m 18s)
* 04:15 ebernhardson: sync-common on mw2187.codfw.wmnet to fix localisation cache errors in exception.log
* 03:37 logmsgbot: ebernhardson@tin Synchronized wmf-config: redisable second cluster only on testwiki (duration: 00m 16s)
* 03:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Reenable second ES cluster on testwiki only (duration: 00m 18s)
* 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-07 03:10:02+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 10m 18s)
* 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-07 02:37:32+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
* 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: Fix JS error (duration: 00m 18s)
* 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: Fix JS error (duration: 00m 17s)
* 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 17s)
* 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
 
== 2015-10-06 ==
* 23:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: Revert multicluster config for testwiki (duration: 00m 18s)
* 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable multicluster ES on testwiki (duration: 00m 17s)
* 22:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 22:40 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.2: Deploy https://gerrit.wikimedia.org/r/#/c/244066/ (duration: 01m 40s)
* 21:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
* 21:15 twentyafterfour: restarted phd in response to a phabricator setup issue
* 20:58 bblack: disabling puppet on caches for VCL testing
* 20:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 20:53 logmsgbot: twentyafterfour@tin Finished scap: sync wmf/1.27.0-wmf.1 (duration: 32m 13s)
* 20:29 mutante: service "ishmael" has been removed (T109777) - removed docroot on neon. tarball exists in /root just in case. code is on https://github.com/asher/ishmael
* 20:20 logmsgbot: twentyafterfour@tin Started scap: sync wmf/1.27.0-wmf.1
* 18:48 yuvipanda: fixing https://phabricator.wikimedia.org/T109216 on labstore1002
* 18:10 godog: upgrade videoscalers to ffmpeg2theora 0.29.0~git+20150813-2
* 15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Suggestion in af, gl, gu, mk, oc, sh and simplewiki [[gerrit:243919]] (duration: 00m 18s)
* 11:33 jynus: performing schema change on db1051 enwiki revision
* 11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1051 for more maintenance (duration: 00m 17s)
* 10:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1051 after maintenance (duration: 00m 17s)
* 10:41 jynus: potential extra load on mediawiki recent changes and watchlist on enwiki, please report any slowdown
* 10:22 jynus: dropping temp recovered tables from db1051 to prepare for repool
* 09:50 _joe_: uploaded a new pybal package for jessie
* 06:19 paravoid: eqord is back up
* 04:24 paravoid: all waves to eqord down, probably related to RT#9619
* 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-06 02:34:31+00:00
* 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 53s)
 
== 2015-10-05 ==
* 23:53 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/243833/ (duration: 00m 17s)
* 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 00m 17s)
* 23:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/243729/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242942/ (duration: 00m 17s)
* 21:16 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Take2: Deploying ZeroBanner 242661+243808 (duration: 00m 17s)
* 21:00 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Rolling back ZeroBanner 242661 (duration: 00m 18s)
* 20:55 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Deploying ZeroBanner 242661 (duration: 00m 17s)
* 18:38 mutante: restarted gitblit
* 17:44 gwicke: rolling restart of eqiad restbase nodes done
* 17:27 gwicke: rolling restart of restbase cluster to rule out driver issues causing the increased p99 read latency
* 17:19 robh: analytics1035 pegged out, ssh unresponsive and raid failures, and then fixed itself 5 minutes later
* 16:32 legoktm: running backPopulateRenameQueueLogs.php ([[gerrit:237169]]) on metawiki
* 16:05 hoo: Updated Wikidata's property suggester with data from today's json dump
* 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.1/extensions/Flow: SWAT: Fix exception on board and topic history pages [[gerrit:243337]] (duration: 00m 20s)
* 14:41 hashar: puppet-lint  Jenkins job is now strict and will -1 on errors as well as warnings https://gerrit.wikimedia.org/r/#/c/243185/
* 10:25 godog: roll-restart cassandra in restbase eqiad
* 09:58 _joe_: installing the new HHVM package to appservers
* 09:50 _joe_: rebooted mw1153, soft lockup due to bnx2 failure
* 09:50 godog: roll-restart cassandra in restbase codfw
* 09:18 godog: roll-restart cassandra on restbase test cluster
* 09:15 godog: stop puppet on restbase and maps in preparation for https://gerrit.wikimedia.org/r/#/c/242896/1
* 08:07 _joe_: upgrading HHVM on all API appservers
* 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct  5 05:33:18 UTC 2015 (duration 33m 17s)
* 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-05 02:33:57+00:00
* 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 25s)
 
== 2015-10-04 ==
* 23:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/extension.json: 8c80ec1273: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation (duration: 00m 17s)
* 05:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct  4 05:13:32 UTC 2015 (duration 13m 31s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-04 02:31:56+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 06s)
 
== 2015-10-03 ==
* 15:33 jynus: stopping temporarily labsdb1004 mariadb to complete clone process
* 09:38 _joe_: rolling restarting all parsoids in eqiad
* 09:14 _joe_: restarting parsoid on wtp1021
* 05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct  3 05:28:54 UTC 2015 (duration 28m 53s)
* 03:12 ori: done; graphite-web back up; url shortening will now work.
* 03:11 ori: shutting down graphite-web for brief sqlite database schema update
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-03 02:36:33+00:00
* 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
 
== 2015-10-02 ==
* 22:46 mutante: holmium: apt-get clean for a little disk space - /var/log/designate/designate-mdns.log is more than half the size of / - needs logrotate
* 21:33 Krinkle: mwscript cleanupRemovedModules.php --wiki zhwiktionary
* 21:21 Krinkle: mwscript cleanupRemovedModules.php --wiki testwikidatawiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki dewiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki nlwiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki test2wiki
* 21:06 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
* 18:39 ottomata: kafka preferred-replica-election
* 17:53 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I680f3fda66c5: Configure ResourceLoader-specific ObjectCache instance (duration: 00m 17s)
* 17:53 ottomata: rolling restart of all hadoop-yarn-nodemanagers to pick up python3 + spark fix
* 17:12 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I8318fe892: Configure MultiWriteBagOStuff for ResourceLoader (duration: 00m 17s)
* 16:29 ori: deployed grafana 8e92884bae (with backport of upstream fb9f9548829f2d4cecf35cda933700e5c2fa1bd6)
* 15:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/resources/src/mediawiki/mediawiki.js: I8029208: Don't clobber existing styles when adding more in IE9 (duration: 00m 17s)
* 14:37 jynus: providing extra grants to wiki db users for heartbeat monitoring
* 14:32 Coren: Change of NFS mount options in labs pushed - puppet may report failures to refresh the mounts (once) on instances; expected and harmless.
* 11:03 moritzm: installed rpcbind security updates on all Ubuntu servers which runs it (jessie was already updated, since the DSA was released earlier)
* 10:53 jynus: restarting HHVM on mw1130
* 08:45 hashar: restarting Nodepool to take in account changes made to the logging configuration https://gerrit.wikimedia.org/r/#/c/240986/
* 06:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct  2 06:07:05 UTC 2015 (duration 7m 4s)
* 02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.1/extensions/ParsoidBatchAPI: stats (duration: 00m 17s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-02 02:41:33+00:00
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 04s)
* 00:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I21bb3f08e7f and follow-ups (duration: 00m 18s)
* 00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/vendor: Add wikimedia/relpath 1.0.3 (duration: 00m 21s)
 
== 2015-10-01 ==
* 22:04 RoanKattouw: Running FlowFixLinks.php on all wikis
* 20:33 subbu: deployed parsoid version 62971510b
* 19:33 twentyafterfour: re-deployed robots.txt patch and restarted apache on iridium (to expand the phabricator robots.txt)
* 18:22 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: Ic1d802ee2: ResourceLoader: cache minified user and site modules (duration: 00m 17s)
* 18:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.1
* 16:40 ottomata: brought analytics1049 back into hadoop after missing a disk since sept. 25th
* 16:34 andrewbogott: test log
* 16:26 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 16:26 valhallasw`cloud: testing, testing
* 15:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/240049 (duration: 00m 17s)
* 15:56 cmjohnson1: db1051 replacing failed disk slot 6
* 15:51 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/modules/dashboard/styles/ext.cx.dashboard.less: Fix: Clicking on down arrow in language selector should trigger ULS (duration: 00m 17s)
* 15:48 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki (duration: 00m 17s)
* 15:47 cmjohnson1: analytics1049 replacing failed disk /dev/sdi at slot 7
* 15:46 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 18s)
* 15:45 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 17s)
* 15:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 17s)
* 15:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 18s)
* 15:33 cmjohnson1: replacing failed disk ms-be1012 /dev/sdf slot 5
* 15:09 matt_flaschen: Did final run of convertAllLqtPages.php on sewikimedia immediately before freezing LQT
* 15:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on se.wikimedia (duration: 00m 18s)
* 14:58 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 14:19 andrewbogott: log test
* 14:17 andrewbogott: testing the log
* 14:07 _joe_: installing the new hhvm package to all the canaries
* 13:41 hashar: Am I logging?
* 13:32 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm7
* 13:03 bblack: repooling cp1046 (eqiad mobile) with caches wiped clean just before
* 12:24 akosiaris: disable SessionRemoteIPcheck on mendelevium's OTRS installation for checking
* 12:18 logmsgbot: aude@tin Finished scap: Put Wikidata extension back on wmf/1.27.0-wmf.1 (duration: 30m 46s)
* 12:09 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 11:47 logmsgbot: aude@tin Started scap: Put Wikidata extension back on wmf/1.27.0-wmf.1
* 10:35 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.1/extensions/: Use 1.27.0-wmf.1 for Wikidata again after fixing T114290 (duration: 01m 07s)
* 10:20 akosiaris: uploaded php5_5.3.10-1ubuntu3.20+wmf1 on apt.wikimedia.org precise-wikimedia
* 09:51 godog: reboot praseodymium to test cassandra systemd unit
* 09:47 godog: stop puppet on restbase* in preparation for    https://gerrit.wikimedia.org/r/242548
* 08:59 akosiaris: disabling puppet on maps-test200* in expectance of https://gerrit.wikimedia.org/r/242548
* 06:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct  1 06:56:51 UTC 2015 (duration 56m 50s)
* 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-01 03:21:39+00:00
* 03:14 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 38s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-10-01 02:47:31+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 54s)
* 01:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
* 01:29 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
* 00:44 mutante: applying puppet fix on fermium (illegal byte sequence in utf-8) as in T114289#1691038 but for other languages
 
== 2015-09-30 ==
* 23:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
* 22:55 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s)
* 22:44 ottomata: perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif
* 22:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
* 22:12 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
* 21:36 cwdent: updated payments from bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11 to 24d5be6886d34b3600031290c7f55ee84f3dcee2
* 21:27 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s)
* 20:43 cscott: updated Parsoid to version 39c60c67
* 20:40 mutante: fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates
* 19:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s)
* 18:51 jynus: stopping replication on s2, s3 and s7 for dbstore1001
* 18:50 ottomata: restarted eventlogging with blacklist=^Analytics$
* 18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
* 18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
* 17:39 ottomata: restarting eventlogging with 12 client side processors
* 16:28 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s)
* 16:20 logmsgbot: krenair@tin Finished scap: swat (duration: 53m 33s)
* 15:27 logmsgbot: krenair@tin Started scap: swat
* 13:53 moritzm: rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier)
* 12:34 moritzm: added debdeploy 0.0.8 to carbon
* 05:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s)
* 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s)
* 01:02 logmsgbot: ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor  ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s)
 
== 2015-09-29 ==
* 23:59 ottomata: restarting eventlogging so that processors use etcd to pick up shared token with which to consistently hash IPs
* 23:46 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo/: SWAT (duration: 00m 17s)
* 23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on mediawikiwiki (duration: 00m 16s)
* 23:33 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 18s)
* 23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 17s)
* 23:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 17s)
* 22:50 ejegg: updated payments-wiki from 3b0915a51a0fd567bdf22f3d4e17548a83e735d8 to bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11
* 22:47 ejegg: updated SmashPig from bf302444eae8236734fd43883b06c7b2512b1532 to 513ec01123e6dbb97b00888a3610a7c5ec24a63b
* 22:38 twentyafterfour: finished deployment of 1.27.0-wmf.1 to group0
* 22:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
* 22:34 logmsgbot: twentyafterfour@tin Finished scap: php-1.27.0-wmf.1/extensions/ (duration: 04m 29s)
* 22:30 twentyafterfour: 22:27:17 sync-dir failed: <ValueError> /srv/mediawiki-staging/php-1.27.0-wmf.1/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening <?php tag
* 22:30 chasemp: es-tool unban-node elastic1031
* 22:29 logmsgbot: twentyafterfour@tin Started scap: php-1.27.0-wmf.1/extensions/
* 22:23 chasemp: unbanning elastic1006 for shard population
* 22:05 ori: 22:03:51 Synchronized php-1.26wmf24/extensions/CentralNotice: 30bdfcb386: Updated mediawiki/core Project: mediawiki/extensions/CentralNotice  6bd658e155a02edb4cc506bc3494a3f4699d3e94 (duration: 00m 17s)
* 19:34 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.1 for validation on testwiki (duration: 31m 49s)
* 19:30 bd808: Updated iegreview.wikimedia.org to c3ac5e6 (Update to Twig 1.20.0) and applied latest schema changes
* 19:02 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.1 for validation on testwiki
* 18:11 cmjohnson1: shutting down elastic1031 to relocate rack/row
* 18:09 cmjohnson1: shutting down elastic1006 to relocate row/rack
* 17:46 mutante: size of conntrack table on iron might be increased due to test scans
* 17:32 ejegg: updated appeal template setting on payments-wiki
* 16:30 cmjohnson1: swapping failed disk db1050
* 16:14 subbu: reverted configuration hotfix from yesterday's Parsoid deploy (re-enabled use of Parsoid batching API)
* 15:58 bblack: ending VCL tests on cp1053
* 15:52 bblack: ending VCL tests on cp1065, starting on cp1053 instead
* 15:39 bblack: cp1065: live-testing some VCL patches, puppet disabled, etc...
* 15:19 chasemp: ban elastic1031 for T112559
* 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Set TransitionDefault true for the English Wikipedia [[gerrit:242040]] (duration: 00m 17s)
* 14:08 paravoid: repooling eqiad; 24h codfw test window is over
* 10:49 godog: bounce pybal on lvs2003 / lvs2006
* 09:13 paravoid: powercycling pybal-test2003
* 08:51 jynus: starting cloning of labsdb1005 (Tools DB), minimal disruption is expected
* 07:03 twentyafterfour: restarted apache2 and phd or iridium to get phabricator back into the correct state
* 06:58 twentyafterfour: checked out correct phabricator release tag on iridium. Something, somewhere, had reverted everything to an old deployment.
* 05:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 29 05:01:30 UTC 2015 (duration 1m 29s)
* 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-29 02:34:55+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 57s)
* 02:25 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually
* 02:24 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20*
* 01:55 robh: upload-lb.codfw.wikimedia.org_ipv6 page flap
* 01:43 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s)
* 01:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s)
* 01:13 robh: didnt powercycle analytics1035 yet, it recovered on its own.
* 01:13 robh: powercycling analytics1035, seems oom, cannot login via ssh or serial console
* 00:56 logmsgbot: tstarling@tin Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s)
 
== 2015-09-28 ==
* 23:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s)
* 23:43 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s)
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s)
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s)
* 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s)
* 20:51 bearND: MobileApps deployed sha1  9df72ec
* 20:42 subbu: deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors
* 20:26 yuvipanda: depooled cp2017 from pybal config too
* 20:22 yuvipanda: depooled cp2017 since it's down
* 19:02 chasemp: powercycle iridium via console as it's unresponsive
* 18:01 awight: update crm from 190f689ff7aec7fecefdf5af501293685c55e041 to 7003cc38797848631d0c4d5f6ff68ab1d6118ad8
* 15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s)
* 15:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s)
* 15:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s)
* 14:00 paravoid: running failover eqiad->codfw test for all frontend traffic
* 06:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s)
* 05:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s)
* 02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s)
 
== 2015-09-27 ==
* 23:17 logmsgbot: ori@tin Synchronized php-1.26wmf24/./resources/src/mediawiki.toolbar/toolbar.less: I94ced06178: mediawiki.toolbar: temporary workaround for T113868 (duration: 00m 17s)
* 16:51 Krenair: ran sync-common on snapshot1001 to bring it up to date
* 16:43 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf21
* 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf20
* 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf19
* 15:50 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/241354/ - fix AbuseFilter block durations (duration: 00m 18s)
* 10:05 jynus: leaving innodb compression tests on es2005 running (could affect lag on that host)
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 05s)
 
== 2015-09-26 ==
* 19:04 hashar: restarting Jenkins. Just in case :-D
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 48s)
 
== 2015-09-25 ==
* 23:54 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s)
* 22:16 mutante: sodium - shutting down