You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Labslogbot
(rolling restart of cassandra instances to rule out a single node in funky state causing elevated p99 latency (gwicke))
imported>Stashbot
(pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2435'])
 
Line 1: Line 1:
== June 24 ==
== 2023-02-08 ==
* 01:01 gwicke: rolling restart of cassandra instances to rule out a single node in funky state causing elevated p99 latency
* 01:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2435']
* 00:43 ori: experimenting with httpd on mw1041 again
* 01:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2434']
* 00:19 gwicke: rolling restart of restbase instances to rule out backend connections as a source for high p99 latencies
* 01:00 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2435']
* 00:14 ori: experimenting with HHVM shutdown via /stop on the admin server on mw1041
* 01:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2433']
* 01:00 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2434']
* 00:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2432']
* 00:52 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2433']
* 00:52 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2432']
* 00:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2431']
* 00:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2430']
* 00:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2431']
* 00:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2430']
* 00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2429']
* 00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2428']
* 00:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2429']
* 00:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2427']
* 00:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2428']
* 00:27 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2426']
* 00:22 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2427']
* 00:17 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2426']
* 00:07 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mw2424']
* 00:06 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mw2425']


== June 23 ==
== 2023-02-07 ==
* 23:38 logmsgbot: ori Finished scap: scapping to all apaches for --restart test (duration: 07m 03s)
* 23:56 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2425']
* 23:30 logmsgbot: ori Started scap: scapping to all apaches for --restart test
* 23:56 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2424']
* 23:24 bblack: nginxes all updated for ssl stapling bugfix
* 23:51 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2423']
* 23:24 logmsgbot: ori Finished scap: scapping to scap-test dsh group for --restart test (duration: 06m 02s)
* 23:49 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2422']
* 23:18 logmsgbot: ori Started scap: scapping to scap-test dsh group for --restart test
* 23:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2423']
* 23:16 logmsgbot: ori scap aborted: scapping to scap-test dsh group for --restart test (duration: 00m 06s)
* 23:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2422']
* 23:16 logmsgbot: ori Started scap: scapping to scap-test dsh group for --restart test
* 23:31 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2421']
* 22:14 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: RejectParserCacheValue may pass a WikiPage or Article (duration: 00m 13s)
* 23:30 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2420']
* 22:07 mutante: tmp. disabling puppet on mw1033
* 23:23 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2421']
* 21:53 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: (no message) (duration: 00m 15s)
* 23:22 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2420']
* 21:50 logmsgbot: ori Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 12s)
* 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2434.mgmt.codfw.wmnet with reboot policy FORCED
* 21:40 mutante: starting instance planet1001 on ganeti1003 - cant get console
* 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2435.mgmt.codfw.wmnet with reboot policy FORCED
* 21:40 logmsgbot: legoktm Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 13s)
* 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2435.mgmt.codfw.wmnet with reboot policy FORCED
* 21:36 bd808: updated scap to 33f3002 (Ensure that the minimum batch size used by cluster_ssh is 1)
* 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2434.mgmt.codfw.wmnet with reboot policy FORCED
* 21:34 logmsgbot: ori Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: 3c8bb2c493: Update SyntaxHighlight_GeSHi for cherry-pick (duration: 00m 13s)
* 22:56 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2432.mgmt.codfw.wmnet with reboot policy FORCED
* 20:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf11
* 22:56 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2433.mgmt.codfw.wmnet with reboot policy FORCED
* 20:19 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change to add Flow_test to enwiki (duration: 00m 11s)
* 22:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2433.mgmt.codfw.wmnet with reboot policy FORCED
* 19:59 logmsgbot: ori scap failed: OSError [Errno 10] No child processes (duration: 01m 46s)
* 22:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2432.mgmt.codfw.wmnet with reboot policy FORCED
* 19:58 logmsgbot: ori Started scap: (no message)
* 22:44 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:52 ori: updated scap to master
* 22:44 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B8 - pt1979@cumin2002"
* 19:11 ori: running apache graceful-stop on mw1042 to test mod_status behavior during graceful stop
* 22:43 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B8 - pt1979@cumin2002"
* 19:02 logmsgbot: twentyafterfour Finished scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed) (duration: 03m 50s)
* 22:41 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2430.mgmt.codfw.wmnet with reboot policy FORCED
* 18:58 logmsgbot: twentyafterfour Started scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed)
* 22:41 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:53 logmsgbot: twentyafterfour Finished scap: New deployment branch: 1.26wmf11 (duration: 26m 37s)
* 22:41 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2431.mgmt.codfw.wmnet with reboot policy FORCED
* 18:31 godog: start rolling-downgrade of cassandra to 2.1.3 T102015
* 22:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2431.mgmt.codfw.wmnet with reboot policy FORCED
* 18:27 logmsgbot: twentyafterfour Started scap: New deployment branch: 1.26wmf11
* 22:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2430.mgmt.codfw.wmnet with reboot policy FORCED
* 18:13 logmsgbot: ori Finished scap: (no message) (duration: 04m 34s)
* 22:30 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2429.mgmt.codfw.wmnet with reboot policy FORCED
* 18:11 paravoid: reloading nginx on all cp* for reuseport
* 22:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2428.mgmt.codfw.wmnet with reboot policy FORCED
* 18:08 logmsgbot: ori Started scap: (no message)
* 22:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2429.mgmt.codfw.wmnet with reboot policy FORCED
* 17:57 ori: repooled scap-test servers (mw1170-mw1175 and mw1270-mw1275)
* 22:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2428.mgmt.codfw.wmnet with reboot policy FORCED
* 17:16 logmsgbot: ori Finished scap: (no message) (duration: 01m 42s)
* 22:15 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:14 logmsgbot: ori Started scap: (no message)
* 22:15 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B6 - pt1979@cumin2002"
* 17:10 logmsgbot: ori Finished scap: (no message) (duration: 01m 34s)
* 22:14 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B6 - pt1979@cumin2002"
* 17:09 logmsgbot: ori Started scap: (no message)
* 22:12 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 17:06 logmsgbot: ori scap aborted: (no message) (duration: 01m 23s)
* 22:10 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "provision new Ganeti VM an-airflow1005 - bking@cumin1001 - [[phab:T327970|T327970]]"
* 17:04 logmsgbot: ori Started scap: (no message)
* 22:08 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]] (duration: 08m 23s)
* 16:53 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 4 (duration: 01m 30s)
* 22:07 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "provision new Ganeti VM an-airflow1005 - bking@cumin1001 - [[phab:T327970|T327970]]"
* 16:52 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 4
* 22:02 urbanecm@deploy1002: urbanecm and superpes: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 16:45 cscott: updated OCG to version db7a56965233a74c73917c78b5c8c84c867321d9
* 22:00 urbanecm@deploy1002: Started scap: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]]
* 16:37 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 3 (duration: 01m 12s)
* 21:59 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]] (duration: 10m 20s)
* 16:35 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 3
* 21:51 urbanecm@deploy1002: superpes and urbanecm: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 16:35 bd808: updated scap to da64a65 (Cast pid read from file to an int)
* 21:49 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]]
* 16:26 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 2 (duration: 01m 26s)
* 21:48 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]] (duration: 15m 32s)
* 16:25 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 2
* 21:35 urbanecm@deploy1002: superpes and urbanecm: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 16:22 bd808: updated scap to 947b93f (Fix reference to _get_apache_list)
* 21:34 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2051.codfw.wmnet with OS bullseye
* 16:12 logmsgbot: bd808 scap failed: AttributeError 'Scap' object has no attribute '_get_apache_list' (duration: 02m 15s)
* 21:33 urbanecm: Create extension tables for Wikilove on bnwikiquote ([[phab:T328834|T328834]])
* 16:10 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart
* 21:33 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]]
* 16:01 paravoid: staggered upgrade of cp* fleet to nginx 1.9.2
* 21:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED
* 15:57 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Follow-up 94e5fd2: Default wmgUseContentTranslation true only on Wikipedias [[gerrit:220161]] (duration: 00m 16s)
* 21:31 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]] (duration: 11m 10s)
* 15:49 jynus: rebooting es1004
* 21:29 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1053.eqiad.wmnet with OS bullseye
* 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable CX as default except where it is not deployed [[gerrit:220078]] (duration: 00m 12s)
* 21:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED
* 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable 'frwiki-recommender' campaign in frwiki [[gerrit:220071]] (duration: 00m 13s)
* 21:24 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED
* 14:54 paravoid: reprepro: including nginx 1.9.2-1~bpo8+1 to jessie-wikimedia/backports
* 21:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED
* 14:39 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1003, depool es1004 (duration: 00m 12s)
* 21:22 urbanecm@deploy1002: urbanecm and jdlrobson: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 14:04 cscott: reverted OCG to version ca4f64852de5b1de782b292b50038fbd2dd84266 (bundler failing with exit code 8)
* 21:21 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED
* 13:57 cscott: updated OCG to version d7c698d5bf730d34057945e912ac75dc542dd788
* 21:21 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED
* 13:44 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209744/ (duration: 00m 13s)
* 21:20 urbanecm@deploy1002: Started scap: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]]
* 13:44 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/209744/ (duration: 00m 12s)
* 21:18 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2051.codfw.wmnet with reason: host reimage
* 12:54 moritzm: ssh on precise hosts has been updated to a backport of 6.6p1-2ubuntu2 (the version from trusty). this allows us to use modern crypto (plus labs can simplify key handling)
* 21:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED
* 12:45 jynus: rebooting es1003
* 21:15 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1053.eqiad.wmnet with reason: host reimage
* 12:18 moritzm: uploaded openssh_6.6p1-2ubuntu2~wmfprecise2 to precise-wikimedia on apt.wikimedia.org
* 21:14 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2051.codfw.wmnet with reason: host reimage
* 12:10 logmsgbot: hoo Synchronized arbitraryaccess.dblist: Arbitrary access for ruwiki and cswiki. T102122 (duration: 00m 12s)
* 21:12 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1053.eqiad.wmnet with reason: host reimage
* 11:33 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1002, depool es1003 (part 2/2) (duration: 00m 12s)
* 21:12 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED
* 11:25 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1002, depool es1003 (duration: 00m 12s)
* 21:02 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: wgEventSreams - Fix android session schema path (duration: 07m 26s)
* 09:41 moritzm: updated jsch on gallium and lanthanum to support modern SSH key exchange in Jenkins (actually that happened yesterday, but I forgot to log it back then)
* 21:01 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1053.eqiad.wmnet with OS bullseye
* 09:41 moritzm: added jsch_0.1.50-1ubuntu1~wmfprecise1 to precise-wikimedia on carbon
* 20:58 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2051.codfw.wmnet with OS bullseye
* 09:09 akosiaris: failing over etherpad to db1016
* 20:57 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2050.codfw.wmnet with OS bullseye
* 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 23 04:53:17 UTC 2015 (duration 53m 16s)
* 20:50 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1051.eqiad.wmnet with OS bullseye
* 03:33 springle: xtrabackup clone db2023 to db1045
* 20:44 bking@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host an-airflow1005.eqiad.wmnet
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-23 02:26:44+00:00
* 20:41 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2050.codfw.wmnet with reason: host reimage
* 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 06m 47s)
* 20:38 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2050.codfw.wmnet with reason: host reimage
* 01:17 logmsgbot: krinkle Synchronized docroot and w: (no message) (duration: 00m 12s)
* 20:36 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1051.eqiad.wmnet with reason: host reimage
* 01:00 bd808: Pruned virt1000 from trebuchet minions list: redis-cli srem "deploy:scap/scap:minions" virt1000.wikimedia.org
* 20:33 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1051.eqiad.wmnet with reason: host reimage
* 20:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2050.codfw.wmnet with OS bullseye
* 20:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1051.eqiad.wmnet with OS bullseye
* 20:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED
* 20:09 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 20:08 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED
* 20:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 19:59 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 19:58 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-airflow1005.eqiad.wmnet on all recursors
* 19:57 bking@cumin1001: START - Cookbook sre.dns.wipe-cache an-airflow1005.eqiad.wmnet on all recursors
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-airflow1005.eqiad.wmnet - bking@cumin1001"
* 19:56 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-airflow1005.eqiad.wmnet - bking@cumin1001"
* 19:55 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 19:55 demon@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.22  refs [[phab:T325585|T325585]]
* 19:54 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 19:53 bking@cumin1001: START - Cookbook sre.dns.netbox
* 19:53 bking@cumin1001: START - Cookbook sre.ganeti.makevm for new host an-airflow1005.eqiad.wmnet
* 19:48 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED
* 19:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED
* 19:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED
* 19:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED
* 19:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED
* 19:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED
* 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED
* 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED
* 19:39 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2049.codfw.wmnet with OS bullseye
* 19:33 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1049.eqiad.wmnet with OS bullseye
* 19:23 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2049.codfw.wmnet with reason: host reimage
* 19:20 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2049.codfw.wmnet with reason: host reimage
* 19:18 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1049.eqiad.wmnet with reason: host reimage
* 19:15 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1049.eqiad.wmnet with reason: host reimage
* 19:04 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1049.eqiad.wmnet with OS bullseye
* 19:03 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2049.codfw.wmnet with OS bullseye
* 19:03 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED
* 19:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED
* 19:00 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:00 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2423,25,26,27 DNS - pt1979@cumin2002"
* 19:00 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2423,25,26,27 DNS - pt1979@cumin2002"
* 18:57 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:53 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2048.codfw.wmnet with OS bullseye
* 18:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1047.eqiad.wmnet with OS bullseye
* 18:37 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2048.codfw.wmnet with reason: host reimage
* 18:34 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2048.codfw.wmnet with reason: host reimage
* 18:32 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1047.eqiad.wmnet with reason: host reimage
* 18:29 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1047.eqiad.wmnet with reason: host reimage
* 18:18 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2048.codfw.wmnet with OS bullseye
* 18:17 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1047.eqiad.wmnet with OS bullseye
* 18:02 bking@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts
* 18:02 bking@cumin2002: START - Cookbook sre.hosts.remove-downtime for 13 hosts
* 17:55 inflatador: bking@cumin1001 repooling elastic and wdqs hosts post-maintenance [[phab:T327925|T327925]]
* 17:53 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2047.codfw.wmnet with OS bullseye
* 17:51 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1046.eqiad.wmnet with OS bullseye
* 17:40 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2047.codfw.wmnet with reason: host reimage
* 17:37 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2047.codfw.wmnet with reason: host reimage
* 17:37 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1046.eqiad.wmnet with reason: host reimage
* 17:34 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1046.eqiad.wmnet with reason: host reimage
* 17:22 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1046.eqiad.wmnet with OS bullseye
* 17:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2047.codfw.wmnet with OS bullseye
* 16:50 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2046.codfw.wmnet with OS bullseye
* 16:48 urbanecm@deploy1002: Finished scap: {{Gerrit|58f4d877}}: Finalize mediawiki/page/change schema, produce at rc1.mediawiki.page_change ([[phab:T308017|T308017]]), {{Gerrit|854ff4ac}}: Finalize mediawiki/page/change schema at 1.0.0 ([[phab:T308017|T308017]]) (duration: 07m 32s)
* 16:46 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1045.eqiad.wmnet with OS bullseye
* 16:41 urbanecm@deploy1002: Started scap: {{Gerrit|58f4d877}}: Finalize mediawiki/page/change schema, produce at rc1.mediawiki.page_change ([[phab:T308017|T308017]]), {{Gerrit|854ff4ac}}: Finalize mediawiki/page/change schema at 1.0.0 ([[phab:T308017|T308017]])
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43765 and previous config saved to /var/cache/conftool/dbconfig/20230207-163902-root.json
* 16:34 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage
* 16:31 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage
* 16:31 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1045.eqiad.wmnet with reason: host reimage
* 16:26 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1045.eqiad.wmnet with reason: host reimage
* 16:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43764 and previous config saved to /var/cache/conftool/dbconfig/20230207-162357-root.json
* 16:18 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]] (duration: 17m 44s)
* 16:15 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2046.codfw.wmnet with OS bullseye
* 16:14 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1045.eqiad.wmnet with OS bullseye
* 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43763 and previous config saved to /var/cache/conftool/dbconfig/20230207-160852-root.json
* 16:02 urbanecm@deploy1002: urbanecm: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 16:00 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]]
* 15:53 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43762 and previous config saved to /var/cache/conftool/dbconfig/20230207-155347-root.json
* 15:53 moritzm: installing tiff security updates
* 15:48 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2045.codfw.wmnet with OS bullseye
* 15:47 urbanecm@deploy1002: Finished scap: {{Gerrit|20a79c55b7073e791e297a5389fa66819f596178}}: Don't add custom attributes in unwrapParsoidSections() ([[phab:T328268|T328268]]) (duration: 07m 34s)
* 15:43 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1043.eqiad.wmnet with OS bullseye
* 15:39 urbanecm@deploy1002: Started scap: {{Gerrit|20a79c55b7073e791e297a5389fa66819f596178}}: Don't add custom attributes in unwrapParsoidSections() ([[phab:T328268|T328268]])
* 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43761 and previous config saved to /var/cache/conftool/dbconfig/20230207-153842-root.json
* 15:32 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2045.codfw.wmnet with reason: host reimage
* 15:29 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2045.codfw.wmnet with reason: host reimage
* 15:28 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1043.eqiad.wmnet with reason: host reimage
* 15:26 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]] (duration: 10m 39s)
* 15:25 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1043.eqiad.wmnet with reason: host reimage
* 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43760 and previous config saved to /var/cache/conftool/dbconfig/20230207-152337-root.json
* 15:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet
* 15:17 urbanecm@deploy1002: matmarex and urbanecm: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 15:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet
* 15:15 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]]
* 15:14 volans@cumin2002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool restbase-async in eqiad: [[phab:T327925|T327925]]
* 15:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet
* 15:13 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1043.eqiad.wmnet with OS bullseye
* 15:13 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2045.codfw.wmnet with OS bullseye
* 15:12 vgutierrez: repool codfw edge site - [[phab:T327925|T327925]]
* 15:09 volans@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) restbase-async.discovery.wmnet on all recursors
* 15:09 volans@cumin2002: START - Cookbook sre.dns.wipe-cache restbase-async.discovery.wmnet on all recursors
* 15:09 volans@cumin2002: START - Cookbook sre.discovery.service-route depool restbase-async in eqiad: [[phab:T327925|T327925]]
* 15:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet
* 15:07 volans@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) pool all active/active services in codfw: [[phab:T327925|T327925]]
* 15:05 marostegui: dbmaint deploy schema change on s8 [[phab:T328807|T328807]] [[phab:T328828|T328828]]
* 15:04 vgutierrez: restart pybal in lvs2010 - [[phab:T327925|T327925]]
* 15:01 marostegui: dbmaint deploy schema change on s6 [[phab:T328807|T328807]]
* 15:00 vgutierrez: restart pybal in lvs2009 - [[phab:T327925|T327925]]
* 14:59 marostegui: dbmaint deploy schema change on s6 [[phab:T328828|T328828]]
* 14:53 moritzm: adding nfraison to pwstore [[phab:T328915|T328915]]
* 14:46 volans@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in codfw: [[phab:T327925|T327925]]
* 14:40 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet,service=thanos-web
* 14:40 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=thanos-fe2001.codfw.wmnet,service=thanos-web
* 14:36 claime: repooled appserver, api_appserver, jobrunner, parsoid - [[phab:T327925|T327925]]
* 14:36 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
* 14:36 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=api_appserver
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=jobrunner
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=appserver
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid
* 14:32 Emperor: pool ms-fe2009 (codfw as a whole still depooled) [[phab:T327925|T327925]]
* 14:28 jbond: enable puppet in codfw, uslfo, esams post switch upgrade [[phab:T327925|T327925]]
* 14:26 claime: depooled appserver, api_appserver, jobrunner, parsoid - [[phab:T327925|T327925]]
* 14:25 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
* 14:21 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid
* 14:19 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=appserver
* 14:19 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=jobrunner
* 14:18 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=api_appserver
* 14:13 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=thanos-fe2002.codfw.wmnet,service=thanos-web
* 14:13 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=thanos-fe2001.codfw.wmnet,service=thanos-web
* 14:08 jbond: disable puppet in codfw, uslfo, esams for switch upgrade [[phab:T327925|T327925]]
* 14:07 lucaswerkmeister-wmde@deploy1002: backport aborted:  (duration: 17m 46s)
* 14:06 XioNoX: asw-a-codfw> request system reboot all-members  - [[phab:T327925|T327925]]
* 13:59 XioNoX: disable puppet in ulsfo/esams/codfw for codfw row A switch upgrade - [[phab:T327925|T327925]]
* 13:56 Emperor: depool ms-fe2009 [[phab:T327925|T327925]]
* 13:55 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:55 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2422 and 24 DNS - pt1979@cumin2002"
* 13:54 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2422 and 24 DNS - pt1979@cumin2002"
* 13:51 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 13:33 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 199 hosts with reason: codfw row A upgrade
* 13:32 oblivian@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) depool all active/active services in codfw: [[phab:T327925|T327925]]
* 13:31 vgutierrez: depool codfw edge site - [[phab:T327925|T327925]]
* 13:31 ayounsi@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 199 hosts with reason: codfw row A upgrade
* 13:13 jbond: enable puppet in codfw, ulsfo and esams to allow depools post  switch upgrade [[phab:T327925|T327925]]
* 13:11 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route depool all active/active services in codfw: [[phab:T327925|T327925]]
* 13:05 jbond: diable puppet in codfw, ulsfo and esams for switch upgrade [[phab:T327925|T327925]]
* 12:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm6001.drmrs.wmnet
* 12:28 vgutierrez: depooling authdns2001 - [[phab:T327925|T327925]]
* 12:25 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on doh2001.wikimedia.org with reason: depooled; [[phab:T327925|T327925]]
* 12:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 8:00:00 on doh2001.wikimedia.org with reason: depooled; [[phab:T327925|T327925]]
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm6001.drmrs.wmnet on all recursors
* 12:20 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache testvm6001.drmrs.wmnet on all recursors
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm6001.drmrs.wmnet - jmm@cumin2002"
* 12:19 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm6001.drmrs.wmnet - jmm@cumin2002"
* 12:17 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 12:17 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm6001.drmrs.wmnet
* 12:00 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1041.eqiad.wmnet with OS bullseye
* 11:56 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2044.codfw.wmnet with OS bullseye
* 11:56 marostegui: Install 10.4.28 on db1152 [[phab:T329011|T329011]]
* 11:52 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons.
* 11:44 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1041.eqiad.wmnet with reason: host reimage
* 11:41 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1041.eqiad.wmnet with reason: host reimage
* 11:40 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage
* 11:37 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage
* 11:33 moritzm: installing imagemagick security updates on buster
* 11:29 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1041.eqiad.wmnet with OS bullseye
* 11:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2044.codfw.wmnet with OS bullseye
* 10:51 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons.
* 10:49 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons.
* 10:19 oblivian@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) pool all active/active services in eqiad: Pooling eqiad for codfw depool today
* 10:19 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in eqiad: Pooling eqiad for codfw depool today
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast1003.wikimedia.org with OS bullseye
* 10:13 oblivian@cumin2002: END (FAIL) - Cookbook sre.discovery.datacenter-route (exit_code=93) pool all active/active services in eqiad: Pooling eqiad for codfw depool today
* 10:12 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in eqiad: Pooling eqiad for codfw depool today
* 10:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast1003.wikimedia.org with reason: host reimage
* 09:56 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast1003.wikimedia.org with reason: host reimage
* 09:44 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast1003.wikimedia.org with OS bullseye
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast2002.wikimedia.org with OS bullseye
* 09:24 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync
* 09:23 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync
* 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast2002.wikimedia.org with reason: host reimage
* 09:20 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync
* 09:20 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync
* 09:20 akosiaris: add wiktionary to mobile-sections rerenders. [[phab:T226931|T226931]]
* 09:19 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast2002.wikimedia.org with reason: host reimage
* 09:19 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync
* 09:19 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync
* 09:08 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons.
* 09:02 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast2002.wikimedia.org with OS bullseye
* 08:50 vgutierrez: rolling upgrade to HAProxy 2.4.21 in cp nodes
* 08:48 kostajh: UTC morning deploys done
* 08:48 kharlan@deploy1002: Finished scap: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]] (duration: 12m 48s)
* 08:37 kharlan@deploy1002: urbanecm and kharlan: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 08:35 kharlan@deploy1002: Started scap: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]]
* 08:30 moritzm: installing imagemagick security updates on Thumbor [[phab:T328901|T328901]]
* 08:28 kharlan@deploy1002: Finished scap: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]] (duration: 12m 11s)
* 08:18 kharlan@deploy1002: kharlan: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
* 08:16 kharlan@deploy1002: Started scap: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]]
* 08:14 kharlan@deploy1002: backport aborted:  (duration: 00m 07s)
* 07:00 marostegui: Failover m3 from db1159 to db1164 - [[phab:T328404|T328404]]
* 06:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2110 in API', diff saved to https://phabricator.wikimedia.org/P43758 and previous config saved to /var/cache/conftool/dbconfig/20230207-063147-root.json
* 06:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1187', diff saved to https://phabricator.wikimedia.org/P43757 and previous config saved to /var/cache/conftool/dbconfig/20230207-062826-root.json
* 04:58 mwpresync@deploy1002: Pruned MediaWiki: 1.40.0-wmf.20 (duration: 02m 20s)
* 04:55 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.22  refs [[phab:T325585|T325585]] (duration: 53m 11s)
* 04:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.22  refs [[phab:T325585|T325585]]


== June 22 ==
== 2023-02-06 ==
* 23:42 gwicke: restarted Cassandra on restbase1006
* 23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED
* 23:27 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/MobileFrontend: For real this time (duration: 00m 14s)
* 23:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED
* 23:27 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: For real this time (duration: 00m 13s)
* 22:55 ryankemper: [[phab:T327925|T327925]] Depooled codfw wdqs hosts: `ryankemper@cumin2002:~$ sudo -E cumin -b 3 'wdqs[2003-2004,2009]*' 'sudo depool'`
* 23:17 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: SWAT (duration: 00m 12s)
* 22:51 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 13 hosts with reason: switch upgrade
* 23:17 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/MobileFrontend/: SWAT (duration: 00m 15s)
* 22:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 13 hosts with reason: switch upgrade
* 23:12 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable TinyRGB ICC profile swapping on testwiki (duration: 00m 13s)
* 22:48 ryankemper: [[phab:T327925|T327925]] Banned `elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076]` on codfw elastic
* 22:51 logmsgbot: ori Synchronized php-1.26wmf10/resources/src/mediawiki/mediawiki.Title.js: I0e5f2d3b2: Fix undeclared dependency on jquery.mwExtension (duration: 00m 12s)
* 22:42 inflatador: bking@cumin2002 banning Elastic nodes from cluster in preparation for [[phab:T327925|T327925]]
* 22:45 gwicke: restarting Cassandra on restbase1005 to get the metrics back
* 22:17 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED
* 22:37 gwicke: restarting Cassandra on restbase1004 to get the metrics back
* 22:10 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED
* 22:33 gwicke: restarting Cassandra on restbase1003 to get the metrics back
* 22:08 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mw2421
* 22:24 gwicke: restarting Cassandra on restbase1002 to get the metrics back
* 22:07 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mw2421
* 22:19 bd808: scap error "@ERROR: access denied to common from localhost (127.0.0.1)" from mw2187 and mw2080 on sync-file test.
* 22:06 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 logmsgbot: bd808 Synchronized README: Testing sync-file after scap update (duration: 00m 12s)
* 22:06 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2421 DNS - pt1979@cumin2002"
* 22:08 RoanKattouw: Deployed patch for T103054
* 22:05 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2421 DNS - pt1979@cumin2002"
* 21:59 godog: reboot restbase1008
* 22:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED
* 21:56 bd808: updated scap to 81b7c14 (Move dsh group file names to config)
* 22:01 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:55 bd808: trebuchet checkout for scap/scap failed on 23 hosts: mw1104, mw1222, mw2009, mw2011, mw2021, mw2028, mw2031, mw2034, mw2069, mw2076, mw2080, mw2086, mw2095, mw2099, mw2120, mw2127, mw2131, mw2136, mw2170, mw2187, mw2189, mw2197, virt1000
* 22:00 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED
* 21:50 bd808: trebuchet fetch for scap/scap failed on mw2086.codfw.wmnet, mw1222.eqiad.wmnet and virt1000.wikimedia.org
* 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED
* 21:41 gwicke: restarting Cassandra on restbase1001 to get the metrics back
* 19:32 zabe@deploy1002: say aborted: (duration: 00m 39s)
* 21:20 ori: Depooled mw1170-mw1175 and mw1270-mw1275 for testing Idddcfe46
* 19:30 zabe@deploy1002: backport aborted: (duration: 00m 00s)
* 21:07 chasemp: rebooting mw1101 the hard way
* 19:29 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 92.62.231.190 # [[phab:T328929|T328929]]
* 20:28 cscott: updated Parsoid to version d488783e
* 19:27 zabe@deploy1002: backport aborted: (duration: 00m 23s)
* 19:34 akosiaris: delete pad:ips from etherpad
* 19:25 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886910{{!}}Add a new throttle rule (T328929)]] (duration: 07m 43s)
* 19:01 jynus: rebooting es1002
* 19:18 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886910{{!}}Add a new throttle rule (T328929)]]
* 18:52 logmsgbot: ori Synchronized php-1.26wmf10/includes/OutputPage.php: I0e5f2d3b2: Construct clean canonical URLs for wiki pages, ignoring request URL (T67402) (duration: 00m 14s)
* 19:17 urbanecm@deploy1002: backport aborted: (duration: 00m 01s)
* 18:01 legoktm: live-hacking mw1017 to debug T103053
* 18:53 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED
* 17:49 mutante: Bugzilla has left the building
* 18:52 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:31 jynus: reseting wikitech-static mysql contents to improve fragmentation
* 18:52 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2420 DNS - pt1979@cumin2002"
* 16:26 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1001, depool es1002 (duration: 00m 14s)
* 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2420 DNS - pt1979@cumin2002"
* 16:12 andrewbogott: shutting down virt1000
* 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mw2420
* 16:08 andrewbogott: disabling puppet on virt1000
* 18:50 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mw2420
* 16:07 ottomata: deploying eventlogging 0.9. This includes changes for arbitrary eventlogging URIs in all eventlogging stages, as well as support for schema based kafka topic URIs.
* 18:48 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 15:24 logmsgbot: thcipriani Synchronized php-1.26wmf10/extensions/WikiEditor: SWAT: Reduce 'Edit' EventLogging schema sampling rate to 6.25% (1/16th) [[gerrit:219837]] (duration: 00m 13s)
* 18:48 pt1979@cumin2002: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Default wmgUseWikibaseQuality on beta to true. [[gerrit:219630]] (duration: 00m 14s)
* 18:48 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 14:32 hashar: restarting Jenkins
* 15:10 vgutierrez: rolling upgrade to HAProxy 2.4.21 in ulsfo cp nodes
* 13:26 jynus: rebooting es1001 for regular maintenance
* 14:37 moritzm: installing imagemagick security updates on buster
* 12:08 paravoid: powercycled ms-be1002, stuck at console
* 14:13 vgutierrez: testing HAProxy 2.4.21 in cp4052 and cp4044
* 11:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1001 (duration: 00m 13s)
* 14:11 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]] (duration: 09m 19s)
* 11:06 _joe_: restarting hhvm on the low-memory appservers (main and api)
* 14:09 vgutierrez: fetch HAProxy 2.4.21 for buster and bullseye (apt.wm.o)
* 09:23 hashar: upgrading Jenkins gearman plugin from 0.1.1 to latest master (f2024bd). Restarting Jenkins.
* 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43754 and previous config saved to /var/cache/conftool/dbconfig/20230206-140753-root.json
* 05:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 22 05:11:22 UTC 2015 (duration 11m 21s)
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43753 and previous config saved to /var/cache/conftool/dbconfig/20230206-140627-root.json
* 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-22 02:31:32+00:00
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43752 and previous config saved to /var/cache/conftool/dbconfig/20230206-140623-root.json
* 02:27 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 27s)
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43751 and previous config saved to /var/cache/conftool/dbconfig/20230206-140606-root.json
* 00:44 jgage: restarted gitblit on antimony again
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43750 and previous config saved to /var/cache/conftool/dbconfig/20230206-140602-root.json
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43749 and previous config saved to /var/cache/conftool/dbconfig/20230206-140554-root.json
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43748 and previous config saved to /var/cache/conftool/dbconfig/20230206-140549-root.json
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43747 and previous config saved to /var/cache/conftool/dbconfig/20230206-140541-root.json
* 14:05 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) (duration: 00m 33s)
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43746 and previous config saved to /var/cache/conftool/dbconfig/20230206-140501-root.json
* 14:05 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided)
* 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43745 and previous config saved to /var/cache/conftool/dbconfig/20230206-140449-root.json
* 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43744 and previous config saved to /var/cache/conftool/dbconfig/20230206-140433-root.json
* 14:04 urbanecm@deploy1002: urbanecm and sharvaniharan: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43743 and previous config saved to /var/cache/conftool/dbconfig/20230206-140405-root.json
* 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43742 and previous config saved to /var/cache/conftool/dbconfig/20230206-140338-root.json
* 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43741 and previous config saved to /var/cache/conftool/dbconfig/20230206-140333-root.json
* 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43740 and previous config saved to /var/cache/conftool/dbconfig/20230206-140316-root.json
* 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43739 and previous config saved to /var/cache/conftool/dbconfig/20230206-140310-root.json
* 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43738 and previous config saved to /var/cache/conftool/dbconfig/20230206-140257-root.json
* 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43737 and previous config saved to /var/cache/conftool/dbconfig/20230206-140249-root.json
* 14:02 urbanecm@deploy1002: Started scap: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]]
* 13:57 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3300
* 13:56 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 3300
* 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43736 and previous config saved to /var/cache/conftool/dbconfig/20230206-135248-root.json
* 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43735 and previous config saved to /var/cache/conftool/dbconfig/20230206-135122-root.json
* 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43734 and previous config saved to /var/cache/conftool/dbconfig/20230206-135118-root.json
* 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43733 and previous config saved to /var/cache/conftool/dbconfig/20230206-135101-root.json
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43732 and previous config saved to /var/cache/conftool/dbconfig/20230206-135057-root.json
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43731 and previous config saved to /var/cache/conftool/dbconfig/20230206-135049-root.json
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43730 and previous config saved to /var/cache/conftool/dbconfig/20230206-135044-root.json
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43729 and previous config saved to /var/cache/conftool/dbconfig/20230206-135036-root.json
* 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43728 and previous config saved to /var/cache/conftool/dbconfig/20230206-134956-root.json
* 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43727 and previous config saved to /var/cache/conftool/dbconfig/20230206-134944-root.json
* 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43726 and previous config saved to /var/cache/conftool/dbconfig/20230206-134928-root.json
* 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43725 and previous config saved to /var/cache/conftool/dbconfig/20230206-134901-root.json
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43724 and previous config saved to /var/cache/conftool/dbconfig/20230206-134833-root.json
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43723 and previous config saved to /var/cache/conftool/dbconfig/20230206-134828-root.json
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43722 and previous config saved to /var/cache/conftool/dbconfig/20230206-134811-root.json
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43721 and previous config saved to /var/cache/conftool/dbconfig/20230206-134805-root.json
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43720 and previous config saved to /var/cache/conftool/dbconfig/20230206-134752-root.json
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43719 and previous config saved to /var/cache/conftool/dbconfig/20230206-134744-root.json
* 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43718 and previous config saved to /var/cache/conftool/dbconfig/20230206-133743-root.json
* 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43717 and previous config saved to /var/cache/conftool/dbconfig/20230206-133618-root.json
* 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43716 and previous config saved to /var/cache/conftool/dbconfig/20230206-133613-root.json
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43715 and previous config saved to /var/cache/conftool/dbconfig/20230206-133556-root.json
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43714 and previous config saved to /var/cache/conftool/dbconfig/20230206-133552-root.json
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43713 and previous config saved to /var/cache/conftool/dbconfig/20230206-133544-root.json
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43712 and previous config saved to /var/cache/conftool/dbconfig/20230206-133540-root.json
* 13:35 jbond: add confd to bookworm repos
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43711 and previous config saved to /var/cache/conftool/dbconfig/20230206-133531-root.json
* 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43710 and previous config saved to /var/cache/conftool/dbconfig/20230206-133451-root.json
* 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43709 and previous config saved to /var/cache/conftool/dbconfig/20230206-133439-root.json
* 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43708 and previous config saved to /var/cache/conftool/dbconfig/20230206-133423-root.json
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43707 and previous config saved to /var/cache/conftool/dbconfig/20230206-133356-root.json
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43706 and previous config saved to /var/cache/conftool/dbconfig/20230206-133329-root.json
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43705 and previous config saved to /var/cache/conftool/dbconfig/20230206-133323-root.json
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43704 and previous config saved to /var/cache/conftool/dbconfig/20230206-133306-root.json
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43703 and previous config saved to /var/cache/conftool/dbconfig/20230206-133300-root.json
* 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43702 and previous config saved to /var/cache/conftool/dbconfig/20230206-133247-root.json
* 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43701 and previous config saved to /var/cache/conftool/dbconfig/20230206-133239-root.json
* 13:26 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:26 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:23 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43700 and previous config saved to /var/cache/conftool/dbconfig/20230206-132238-root.json
* 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43699 and previous config saved to /var/cache/conftool/dbconfig/20230206-132113-root.json
* 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43698 and previous config saved to /var/cache/conftool/dbconfig/20230206-132108-root.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43697 and previous config saved to /var/cache/conftool/dbconfig/20230206-132051-root.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43696 and previous config saved to /var/cache/conftool/dbconfig/20230206-132047-root.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43695 and previous config saved to /var/cache/conftool/dbconfig/20230206-132039-root.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43694 and previous config saved to /var/cache/conftool/dbconfig/20230206-132035-root.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43693 and previous config saved to /var/cache/conftool/dbconfig/20230206-132026-root.json
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43692 and previous config saved to /var/cache/conftool/dbconfig/20230206-131947-root.json
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43691 and previous config saved to /var/cache/conftool/dbconfig/20230206-131934-root.json
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43690 and previous config saved to /var/cache/conftool/dbconfig/20230206-131918-root.json
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43689 and previous config saved to /var/cache/conftool/dbconfig/20230206-131851-root.json
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43688 and previous config saved to /var/cache/conftool/dbconfig/20230206-131824-root.json
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43687 and previous config saved to /var/cache/conftool/dbconfig/20230206-131818-root.json
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43686 and previous config saved to /var/cache/conftool/dbconfig/20230206-131801-root.json
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43685 and previous config saved to /var/cache/conftool/dbconfig/20230206-131755-root.json
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43684 and previous config saved to /var/cache/conftool/dbconfig/20230206-131740-root.json
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43683 and previous config saved to /var/cache/conftool/dbconfig/20230206-131734-root.json
* 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43682 and previous config saved to /var/cache/conftool/dbconfig/20230206-130733-root.json
* 13:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43681 and previous config saved to /var/cache/conftool/dbconfig/20230206-130608-root.json
* 13:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43680 and previous config saved to /var/cache/conftool/dbconfig/20230206-130603-root.json
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43679 and previous config saved to /var/cache/conftool/dbconfig/20230206-130547-root.json
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43678 and previous config saved to /var/cache/conftool/dbconfig/20230206-130542-root.json
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43677 and previous config saved to /var/cache/conftool/dbconfig/20230206-130534-root.json
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43676 and previous config saved to /var/cache/conftool/dbconfig/20230206-130530-root.json
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43675 and previous config saved to /var/cache/conftool/dbconfig/20230206-130521-root.json
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43674 and previous config saved to /var/cache/conftool/dbconfig/20230206-130442-root.json
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43673 and previous config saved to /var/cache/conftool/dbconfig/20230206-130429-root.json
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43672 and previous config saved to /var/cache/conftool/dbconfig/20230206-130414-root.json
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43671 and previous config saved to /var/cache/conftool/dbconfig/20230206-130346-root.json
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43670 and previous config saved to /var/cache/conftool/dbconfig/20230206-130319-root.json
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43669 and previous config saved to /var/cache/conftool/dbconfig/20230206-130313-root.json
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43668 and previous config saved to /var/cache/conftool/dbconfig/20230206-130256-root.json
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43667 and previous config saved to /var/cache/conftool/dbconfig/20230206-130250-root.json
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43666 and previous config saved to /var/cache/conftool/dbconfig/20230206-130235-root.json
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43665 and previous config saved to /var/cache/conftool/dbconfig/20230206-130230-root.json
* 12:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43664 and previous config saved to /var/cache/conftool/dbconfig/20230206-125228-root.json
* 12:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43663 and previous config saved to /var/cache/conftool/dbconfig/20230206-125103-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43662 and previous config saved to /var/cache/conftool/dbconfig/20230206-125059-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43661 and previous config saved to /var/cache/conftool/dbconfig/20230206-125042-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43660 and previous config saved to /var/cache/conftool/dbconfig/20230206-125037-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43659 and previous config saved to /var/cache/conftool/dbconfig/20230206-125029-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43658 and previous config saved to /var/cache/conftool/dbconfig/20230206-125025-root.json
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43657 and previous config saved to /var/cache/conftool/dbconfig/20230206-125017-root.json
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43656 and previous config saved to /var/cache/conftool/dbconfig/20230206-124937-root.json
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43655 and previous config saved to /var/cache/conftool/dbconfig/20230206-124924-root.json
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43654 and previous config saved to /var/cache/conftool/dbconfig/20230206-124909-root.json
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43653 and previous config saved to /var/cache/conftool/dbconfig/20230206-124841-root.json
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43652 and previous config saved to /var/cache/conftool/dbconfig/20230206-124814-root.json
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43651 and previous config saved to /var/cache/conftool/dbconfig/20230206-124808-root.json
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43650 and previous config saved to /var/cache/conftool/dbconfig/20230206-124751-root.json
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43649 and previous config saved to /var/cache/conftool/dbconfig/20230206-124745-root.json
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43648 and previous config saved to /var/cache/conftool/dbconfig/20230206-124730-root.json
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43647 and previous config saved to /var/cache/conftool/dbconfig/20230206-124725-root.json
* 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43646 and previous config saved to /var/cache/conftool/dbconfig/20230206-124629-root.json
* 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43645 and previous config saved to /var/cache/conftool/dbconfig/20230206-124617-root.json
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43644 and previous config saved to /var/cache/conftool/dbconfig/20230206-124513-root.json
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43643 and previous config saved to /var/cache/conftool/dbconfig/20230206-124506-root.json
* 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43642 and previous config saved to /var/cache/conftool/dbconfig/20230206-123124-root.json
* 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43641 and previous config saved to /var/cache/conftool/dbconfig/20230206-123112-root.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43640 and previous config saved to /var/cache/conftool/dbconfig/20230206-123007-root.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43639 and previous config saved to /var/cache/conftool/dbconfig/20230206-123001-root.json
* 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43638 and previous config saved to /var/cache/conftool/dbconfig/20230206-121619-root.json
* 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43637 and previous config saved to /var/cache/conftool/dbconfig/20230206-121608-root.json
* 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43636 and previous config saved to /var/cache/conftool/dbconfig/20230206-121503-root.json
* 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43635 and previous config saved to /var/cache/conftool/dbconfig/20230206-121456-root.json
* 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43634 and previous config saved to /var/cache/conftool/dbconfig/20230206-120114-root.json
* 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43633 and previous config saved to /var/cache/conftool/dbconfig/20230206-120103-root.json
* 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43631 and previous config saved to /var/cache/conftool/dbconfig/20230206-115958-root.json
* 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43630 and previous config saved to /var/cache/conftool/dbconfig/20230206-115951-root.json
* 11:58 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host db1108.eqiad.wmnet
* 11:47 jbond: puppetmaster[12]002 reintroduced to services
* 11:46 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host db1108.eqiad.wmnet
* 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43629 and previous config saved to /var/cache/conftool/dbconfig/20230206-114609-root.json
* 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43628 and previous config saved to /var/cache/conftool/dbconfig/20230206-114558-root.json
* 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43627 and previous config saved to /var/cache/conftool/dbconfig/20230206-114453-root.json
* 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43626 and previous config saved to /var/cache/conftool/dbconfig/20230206-114446-root.json
* 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43625 and previous config saved to /var/cache/conftool/dbconfig/20230206-113104-root.json
* 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43624 and previous config saved to /var/cache/conftool/dbconfig/20230206-113053-root.json
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43623 and previous config saved to /var/cache/conftool/dbconfig/20230206-112948-root.json
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43622 and previous config saved to /var/cache/conftool/dbconfig/20230206-112942-root.json
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43621 and previous config saved to /var/cache/conftool/dbconfig/20230206-112900-root.json
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43620 and previous config saved to /var/cache/conftool/dbconfig/20230206-112856-root.json
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43619 and previous config saved to /var/cache/conftool/dbconfig/20230206-112839-root.json
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43618 and previous config saved to /var/cache/conftool/dbconfig/20230206-112832-root.json
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43617 and previous config saved to /var/cache/conftool/dbconfig/20230206-112825-root.json
* 11:28 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on puppetmaster2002.codfw.wmnet,puppetmaster1002.eqiad.wmnet with reason: Decom
* 11:27 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on puppetmaster2002.codfw.wmnet,puppetmaster1002.eqiad.wmnet with reason: Decom
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43616 and previous config saved to /var/cache/conftool/dbconfig/20230206-111356-root.json
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43615 and previous config saved to /var/cache/conftool/dbconfig/20230206-111351-root.json
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43614 and previous config saved to /var/cache/conftool/dbconfig/20230206-111334-root.json
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43613 and previous config saved to /var/cache/conftool/dbconfig/20230206-111327-root.json
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43612 and previous config saved to /var/cache/conftool/dbconfig/20230206-111320-root.json
* 11:03 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 11:03 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 11:03 akosiaris: deploy changeprop 0.10.19, adding wikivoyage to list of domains the mobile-sections get rerendered for. [[phab:T226931|T226931]]
* 11:03 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 11:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 11:01 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 11:01 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:59 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:58 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:58 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43610 and previous config saved to /var/cache/conftool/dbconfig/20230206-105851-root.json
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43609 and previous config saved to /var/cache/conftool/dbconfig/20230206-105846-root.json
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43608 and previous config saved to /var/cache/conftool/dbconfig/20230206-105829-root.json
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43607 and previous config saved to /var/cache/conftool/dbconfig/20230206-105822-root.json
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43606 and previous config saved to /var/cache/conftool/dbconfig/20230206-105815-root.json
* 10:56 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43605 and previous config saved to /var/cache/conftool/dbconfig/20230206-104346-root.json
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43604 and previous config saved to /var/cache/conftool/dbconfig/20230206-104341-root.json
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43603 and previous config saved to /var/cache/conftool/dbconfig/20230206-104324-root.json
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43602 and previous config saved to /var/cache/conftool/dbconfig/20230206-104317-root.json
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43601 and previous config saved to /var/cache/conftool/dbconfig/20230206-104310-root.json
* 10:36 marostegui: Upgrade db1115 (db_inventory master) to 10.6. [[phab:T328408|T328408]]
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43600 and previous config saved to /var/cache/conftool/dbconfig/20230206-102841-root.json
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43599 and previous config saved to /var/cache/conftool/dbconfig/20230206-102837-root.json
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43598 and previous config saved to /var/cache/conftool/dbconfig/20230206-102820-root.json
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43597 and previous config saved to /var/cache/conftool/dbconfig/20230206-102812-root.json
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43596 and previous config saved to /var/cache/conftool/dbconfig/20230206-102806-root.json
* 10:27 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:27 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 10:26 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 10:23 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43595 and previous config saved to /var/cache/conftool/dbconfig/20230206-101336-root.json
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43594 and previous config saved to /var/cache/conftool/dbconfig/20230206-101332-root.json
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43593 and previous config saved to /var/cache/conftool/dbconfig/20230206-101315-root.json
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43592 and previous config saved to /var/cache/conftool/dbconfig/20230206-101308-root.json
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43591 and previous config saved to /var/cache/conftool/dbconfig/20230206-101301-root.json
* 10:10 hashar@deploy1002: Finished deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) (duration: 00m 38s)
* 10:09 hashar@deploy1002: Started deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided)
* 09:05 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 09:05 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]] (duration: 18m 56s)
* 09:05 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 09:04 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:04 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:56 urbanecm@deploy1002: wmde-fisch and urbanecm: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:46 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]]
* 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2094 db2097 db2103 db2104 db2105 db2106 db2121 db2122 db2132 db2133 db2136 db2142 db2145 db2146 db2153 db2154 db2155 db2156 db2157 db2158 db2175 db2176 db2183 [[phab:T327925|T327925]]', diff saved to https://phabricator.wikimedia.org/P43587 and previous config saved to /var/cache/conftool/dbconfig/20230206-073015-root.json
* 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2020 es2024 es2026 es2027 es2028 [[phab:T327925|T327925]]', diff saved to https://phabricator.wikimedia.org/P43586 and previous config saved to /var/cache/conftool/dbconfig/20230206-071913-root.json
* 07:17 hashar: Restarted Gerrit for deployment
* 07:14 hashar@deploy1002: Finished deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json (duration: 00m 05s)
* 07:14 hashar@deploy1002: Started deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json
* 07:07 hashar@deploy1002: Finished deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json {{!}} [[phab:T328134|T328134]] (duration: 00m 10s)
* 07:06 hashar@deploy1002: Started deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json {{!}} [[phab:T328134|T328134]]


== June 21 ==
== 2023-02-05 ==
* 11:28 jynus: restarting apache on mw1110
* 22:28 topranks: Re-enabling peering to Seabone/Telecom Italit AS 6762 on cr2-esams at AMS-IX
* 06:55 gwicke: restarted  bootstrap on restbase1009 earlier today; hardware hasn't died yet
* 14:39 cdanis: silenced NELHigh alert for 20 hours: Telecom Italy issues; alertmanager silence id 3fb3b999-9756-44af-a1e8-{{Gerrit|fd1faae8b9bf}}
* 05:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 21 05:01:07 UTC 2015 (duration 1m 6s)
* 11:49 topranks: Manually deactivating peering to Telecom Italia / Seabone at AMS-IX on cr2-esams as they are having issues
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-21 02:27:13+00:00
* 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 10m 23s)
* 01:39 jgage: restarted gitblit on antimony at 00:43 UTC
* 01:37 Krenair: testing morebots


== June 20 ==
== 2023-02-03 ==
* 22:50 bblack: restarted gitblit java service on antimony
* 21:05 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 04:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 20 04:27:14 UTC 2015 (duration 27m 13s)
* 21:04 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-20 02:21:30+00:00
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 02s)
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 21:02 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 21:00 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 20:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 20:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 19:44 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
* 19:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS bullseye
* 19:00 dzahn@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
* 18:59 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
* 18:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
* 18:49 topranks: Enabling 4x10G channelization for pic 0 QSFP 4 on cr1-codfw
* 18:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
* 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS bullseye
* 18:23 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
* 18:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS bullseye
* 17:57 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
* 17:57 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
* 17:39 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
* 17:36 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1089.eqiad.wmnet with OS bullseye
* 17:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS bullseye
* 17:34 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
* 17:34 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1086.eqiad.wmnet with OS bullseye
* 17:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
* 17:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1086.eqiad.wmnet with OS bullseye
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1089.eqiad.wmnet with OS bullseye
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 16:44 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 16:41 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 16:32 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet
* 16:25 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet
* 15:51 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): test (duration: 00m 26s)
* 15:51 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): test
* 15:23 milimetric@deploy1002: Finished deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection (duration: 00m 15s)
* 15:22 milimetric@deploy1002: Started deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection
* 14:31 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 09s)
* 14:31 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 14:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet
* 14:19 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 23s)
* 14:18 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 14:13 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=ats-be
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=cdn
* 13:51 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1087.eqiad.wmnet with OS bullseye
* 13:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
* 13:25 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
* 13:05 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1087.eqiad.wmnet with OS bullseye
* 12:09 moritzm: installing node-moment security updates
* 12:01 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 13s)
* 12:00 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2010.codfw.wmnet
* 11:58 moritzm: installing node-qs security updates
* 11:50 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2010.codfw.wmnet
* 11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2009.codfw.wmnet
* 11:28 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2009.codfw.wmnet
* 10:44 moritzm: updating perf on buster hosts
* 10:24 stevemunene@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:11 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2008.codfw.wmnet
* 10:07 stevemunene@cumin1001: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:06 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:03 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2008.codfw.wmnet
* 09:51 moritzm: installing ruby-rack security updates
* 09:31 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:31 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:24 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:23 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:23 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:19 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet
* 09:14 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet
* 09:13 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:13 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:07 moritzm: installing modsecurity-crs security updates
* 09:02 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
* 05:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1084.eqiad.wmnet with OS bullseye
* 05:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1085.eqiad.wmnet with OS bullseye
* 04:50 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1084.eqiad.wmnet with OS bullseye
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1085.eqiad.wmnet with OS bullseye
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS bullseye
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1082.eqiad.wmnet with OS bullseye
* 03:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
* 03:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS bullseye
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1083.eqiad.wmnet with OS bullseye
* 03:20 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
* 03:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1080.eqiad.wmnet with OS bullseye
* 02:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
* 02:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=ats-be
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=cdn
* 02:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS bullseye
* 02:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
* 02:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
* 02:00 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
* 01:38 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS bullseye
* 01:31 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
* 00:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye


== June 19 ==
== 2023-02-02 ==
* 23:32 gwicke: upgraded restbase1006 to cassandra 2.1.7
* 22:58 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
* 23:30 gwicke: starting cassandra bootstrap on restbase1009
* 22:15 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
* 21:37 gwicke: upgraded cassandra on 1003 to 2.1.7 (pre-release, likely going out on Monday)
* 22:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS bullseye
* 18:32 godog: stop cassandra on restbase1008
* 22:01 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
* 17:45 logmsgbot: krenair Synchronized private/PrivateSettings.php: sync 4a30446e for wikitech cleanup - T102361 (duration: 00m 12s)
* 22:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
* 17:24 godog: install linux 3.19 on restbase100[789]
* 21:58 zabe@deploy1002: Finished scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] (duration: 07m 58s)
* 17:12 ori: salt -t30 -G 'php:hhvm' cmd.run 'rm -f /usr/local/bin/check_tc_space' (https://gerrit.wikimedia.org/r/#/c/219102/)
* 21:52 zabe@deploy1002: zabe: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 16:54 moritzm: updated/rebooted nescio/maerlant to 3.19
* 21:50 zabe@deploy1002: Started scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]]
* 13:40 andrewbogott: test test test
* 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1078.eqiad.wmnet with OS bullseye
* 02:19 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-19 02:19:33+00:00
* 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
* 02:16 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 05m 08s)
* 21:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
* 00:49 springle: killed storm of research queries on dbstore1002, load avg 90+, replag, likely explosion, etc. emailing analytics@
* 21:30 brennen: end of utc late backport & config window
* 00:13 logmsgbot: ebernhardson Synchronized php-1.26wmf10/extensions/Flow/tests/: no-op sync of flow test cases in wmf10 (duration: 00m 17s)
* 21:30 brennen@deploy1002: Finished scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] (duration: 11m 14s)
* 00:11 logmsgbot: ebernhardson Synchronized php-1.26wmf10/skins/Vector/: Bump Vector submodule in 1.26wmf10 for swat (duration: 00m 12s)
* 21:23 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
* 21:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS bullseye
* 21:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
* 21:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS bullseye
* 21:21 brennen@deploy1002: brennen and nray: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 21:20 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
* 21:19 brennen@deploy1002: Started scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]]
* 21:18 brennen@deploy1002: Finished scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] (duration: 12m 02s)
* 21:07 brennen@deploy1002: brennen and dreamyjazz: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 21:06 brennen@deploy1002: Started scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]]
* 20:59 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
* 20:59 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1078.eqiad.wmnet with OS bullseye
* 20:52 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
* 20:49 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
* 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
* 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS bullseye
* 20:23 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include bullseye-wikimedia /home/rzl/httpbb/bullseye/httpbb_0.0.3-1+deb11u1_amd64.changes  # [[phab:T328280|T328280]]
* 20:21 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/httpbb/buster/httpbb_0.0.3-1_amd64.changes  # [[phab:T328280|T328280]]
* 20:11 zabe@deploy1002: Finished scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] (duration: 09m 39s)
* 20:03 zabe@deploy1002: zabe: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 20:02 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic2037.codfw.wmnet
* 20:01 zabe@deploy1002: Started scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]]
* 19:55 bking@cumin1001: START - Cookbook sre.hosts.reboot-single for host elastic2037.codfw.wmnet
* 19:54 ryankemper: [[phab:T328674|T328674]] [Elastic] With puppet disabled on elastic* fleet, `ryankemper@elastic2037:~$ sudo run-puppet-agent --force` to verify changes in https://gerrit.wikimedia.org/r/886055
* 19:30 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 19:28 zabe@deploy1002: say aborted:  (duration: 00m 03s)
* 18:42 zabe@deploy1002: Finished scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] (duration: 08m 19s)
* 18:36 zabe@deploy1002: zabe: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 18:34 zabe@deploy1002: Started scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]]
* 18:08 aokoth@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0
* 18:08 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:08 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:08 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2043.codfw.wmnet with OS bullseye
* 18:07 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:06 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:05 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:05 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:03 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1037.eqiad.wmnet with OS bullseye
* 17:52 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage
* 17:49 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage
* 17:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage
* 17:45 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage
* 17:33 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2043.codfw.wmnet with OS bullseye
* 17:32 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1037.eqiad.wmnet with OS bullseye
* 17:29 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0
* 17:12 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync
* 17:12 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync
* 16:53 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 16:52 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 16:51 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 16:50 dancy@deploy1002: Installation of scap version "4.34.0" completed for 561 hosts
* 16:50 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 16:50 dancy@deploy1002: Installing scap version "4.34.0" for 561 hosts
* 16:50 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 16:49 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 16:48 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 16:48 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 16:47 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync
* 16:46 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync
* 16:25 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2007.codfw.wmnet
* 16:18 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 16:17 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 16:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2007.codfw.wmnet
* 16:17 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 16:16 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 16:16 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 16:15 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 16:10 volans: uploaded python3-wmflib_1.2.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia
* 16:10 dzahn@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0
* 15:40 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) (duration: 07m 01s)
* 15:38 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:37 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:35 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:35 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:34 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0
* 15:33 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided)
* 15:24 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti3004
* 15:17 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti3004
* 15:06 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2006.codfw.wmnet
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002"
* 15:02 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002"
* 15:00 vgutierrez: rolling restart of varnish in cache::text - [[phab:T315676|T315676]]
* 14:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:59 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2006.codfw.wmnet
* 14:55 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 14:45 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 14:39 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 14:31 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2005.codfw.wmnet
* 14:29 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 14:25 moritzm: installing containerd security updates on codfw k8s nodes
* 14:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2005.codfw.wmnet
* 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=ats-be
* 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=cdn
* 13:10 kharlan:: Deployed security patch for [[phab:T328643|T328643]]
* 13:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1076.eqiad.wmnet with OS bullseye
* 13:04 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:03 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:03 kharlan:: Deployed security patch for [[phab:T328643|T328643]]
* 13:02 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 13:01 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2004.codfw.wmnet
* 13:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 12:55 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2004.codfw.wmnet
* 12:47 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage
* 12:47 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 12:46 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 12:44 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage
* 12:42 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 12:42 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 12:39 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 12:39 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 12:29 btullis@deploy1002: Finished deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade (duration: 00m 42s)
* 12:29 claime: Work ongoing on m2 and m3
* 12:29 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2003.codfw.wmnet
* 12:29 btullis@deploy1002: Started deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade
* 12:23 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1076.eqiad.wmnet with OS bullseye
* 12:22 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2003.codfw.wmnet
* 12:08 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 12:08 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 11:46 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 11:42 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:42 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:41 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:41 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:40 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:39 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:38 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:37 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-4.out # [[phab:T328634|T328634]] – made some progress then errored out again
* 11:32 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-3.out # [[phab:T328634|T328634]] – seemed to finish the first 20 pages and then go into an infinite loop, I Ctrl+Ced it
* 11:28 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-2.out # [[phab:T328634|T328634]] – another error but made more progress
* 11:23 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes.out # [[phab:T328634|T328634]] – failed quickly, details in task
* 11:22 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync
* 11:22 elukey@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync
* 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:02 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 10:27 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet
* 10:19 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet
* 10:17 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 10:11 moritzm: restarting FPM on mw canaries to pick up tiff security updates
* 10:04 moritzm: installing tiff security updates
* 09:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet
* 09:55 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync
* 09:54 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync
* 09:51 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet
* 09:40 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync
* 09:40 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: sync
* 09:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 398143
* 09:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 398143
* 09:16 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6
* 09:13 apergos: UTC morning backport and config training window done
* 09:13 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync
* 09:12 elukey@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-main: sync
* 09:11 elukey: roll restart of eventgate-main pods in wikikube eqiad/codfw to pick up new stream configs - [[phab:T328576|T328576]]
* 08:57 ariel@deploy1002: Finished scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] (duration: 10m 56s)
* 08:48 ariel@deploy1002: ariel and aishik: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:46 ariel@deploy1002: Started scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]]
* 08:39 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6
* 08:37 tgr@deploy1002: Finished scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] (duration: 14m 26s)
* 08:27 tgr@deploy1002: tgr: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 08:23 tgr@deploy1002: Started scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]]
* 06:17 kart_: Updated cxserver to 2023-02-02-004918-production ([[phab:T129470|T129470]], [[phab:T172035|T172035]], [[phab:T327842|T327842]])
* 06:16 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 06:15 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 06:13 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 06:12 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 06:09 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 06:09 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 04:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5024.eqsin.wmnet
* 03:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS bullseye
* 03:21 ejegg: payments-wiki upgraded from {{Gerrit|f20a2208}} to {{Gerrit|53d1a58d}}
* 02:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage
* 02:46 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage
* 02:14 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye
* 02:14 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5024.eqsin.wmnet with OS bullseye
* 01:56 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye
* 01:55 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet
* 01:55 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5023.eqsin.wmnet with OS bullseye
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=cdn
* 01:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1075.eqiad.wmnet with OS bullseye
* 01:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage
* 01:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage
* 01:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage
* 01:18 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage
* 01:07 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1075.eqiad.wmnet with OS bullseye
* 00:44 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5023.eqsin.wmnet with OS bullseye
* 00:06 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5022.eqsin.wmnet
* 00:04 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS bullseye


== June 18 ==
== 2023-02-01 ==
* 23:37 logmsgbot: ebernhardson Synchronized php-1.26wmf9/skins/Vector: Bump Vector in 1.26wmf9 for SWAT (duration: 00m 16s)
* 23:45 zabe@deploy1002: Finished scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] (duration: 08m 07s)
* 23:22 logmsgbot: ebernhardson Synchronized wmf-config/: Actually enable the feedback link on Special:Search (duration: 00m 17s)
* 23:39 zabe@deploy1002: zabe: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
* 23:08 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Enable wgCirrusSearchFeedbackLink on enwiki (duration: 00m 13s)
* 23:37 zabe@deploy1002: Started scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]]
* 21:07 godog: start (bootstrap) cassandra on restbase1008
* 23:31 rzl@cumin2002: dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P43574 and previous config saved to /var/cache/conftool/dbconfig/20230201-233140-rzl.json
* 20:43 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-urd-hin_0.1.0+svn~r60389-1
* 23:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
* 20:17 akosiaris: restarted salt on sca1001, truncate log files. keep a sample in /tmp/
* 23:27 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
* 20:03 chasemp: apache && hhvm restart for mw 1243 1250 1254 1256 1257
* 23:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: security release
* 20:00 chasemp: apache && hhvm restart for mw...1256 1255 1254 1250 1243 1242 1071 1021
* 23:17 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]] (duration: 06m 57s)
* 19:58 mutante: restarting hhvm on mw1021, mw1071
* 23:10 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 19:27 godog: bounce cassandra on restbase1003, new logging configuration
* 23:01 zabe@deploy1002: Finished scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] (duration: 07m 45s)
* 19:26 akosiaris: puppet-merged on strontium
* 22:55 zabe@deploy1002: zabe: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 19:15 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf10
* 22:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye
* 19:06 godog: upgrade cassandra to 2.1.6 on restbase1003
* 22:53 zabe@deploy1002: Started scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]]
* 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd_0.1.0~r57551-1
* 22:49 zabe@deploy1002: Finished scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] (duration: 13m 03s)
* 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hin_0.1.0~r57344-1
* 22:47 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release
* 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cy-en_0.1.1~r57554-1
* 22:40 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS bullseye
* 18:43 legoktm: fixed content model of MediaWiki:Common.css@lrcwiki
* 22:38 zabe@deploy1002: zabe: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 18:18 YuviPanda: restarted nutcracker on wikitech
* 22:36 zabe@deploy1002: Started scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]]
* 18:16 YuviPanda: restarted keystone on labcontrol1001
* 22:32 kindrobot: close UTC late backport window
* 17:13 gwicke: bouncing cassandra on restbase1002
* 22:31 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] (duration: 10m 37s)
* 17:11 godog: restart cassandra on restbase1004
* 22:22 kindrobot@deploy1002: nray and kindrobot: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 15:53 gwicke: updated restbase to 7ffaf94b
* 22:21 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]]
* 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Hovercards: Disable test release on Catalan and Greek Wikipedias [[gerrit:215932]] (duration: 00m 13s)
* 22:14 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] (duration: 18m 14s)
* 15:06 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150618 [[gerrit:218886]] (duration: 00m 14s)
* 21:57 kindrobot@deploy1002: kindrobot and sbailey: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 11:14 akosiaris: powercycling labstore2001
* 21:57 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 09:08 moritzm: added firejail_0.9.26-1~wmfjessie1 and firejail_0.9.26-1~wmftrusty1 to apt.wikimedia.org
* 21:56 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]]
* 08:45 jynus: very brief replication stop for s7, already corrected
* 21:53 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 06:51 Coren: rebooting labstore2001
* 21:52 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] (duration: 14m 53s)
* 06:32 legoktm: live hacking mw1017 for T102915
* 21:43 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye
* 05:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 18 05:26:01 UTC 2015 (duration 26m 0s)
* 21:39 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 02:48 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-18 02:48:44+00:00
* 21:39 kindrobot@deploy1002: dreamyjazz and kindrobot: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 02:46 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 05m 03s)
* 21:37 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]]
* 02:32 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-18 02:32:45+00:00
* 21:32 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] (duration: 13m 56s)
* 02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 56s)
* 21:26 eevans@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 02:04 springle: applied T99941 scema change to all remaining affected (ie, old) wikis
* 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 02:01 tgr: ran https://gerrit.wikimedia.org/r/#/c/159350/7/backend/schema/mysql/developer_agreement.sql on mediawikiwiki
* 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 01:32 ejegg: updated payments from f33d0a8687a120a2057a7e6acad67da63b17f97e to a17ee221db0dbde70c92e24fc188379b6dbad613
* 21:24 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 01:20 logmsgbot: ori Synchronized php-1.26wmf10/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: 0c21a14a6e: Revert StashEdit: Use postWithToken (duration: 00m 13s)
* 21:20 kindrobot@deploy1002: arlolra and kindrobot: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
* 01:06 twentyafterfour: applied hotfix for T102276 and restarted apache on iridium
* 21:19 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 00:00 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf10
* 21:18 kindrobot@deploy1002: Started scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]]
* 21:14 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3065.esams.wmnet
* 21:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS bullseye
* 21:03 kindrobot: start UTC late backport deployment window
* 21:02 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 20:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3065.esams.wmnet with reason: host reimage
* 20:44 eevans@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=sessionstore,name=codfw
* 20:43 urandom: depooling sessionstore —codfw— in preparation for Cassandra restarts — [[phab:T327675|T327675]]
* 20:42 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3065.esams.wmnet with reason: host reimage
* 20:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3064.esams.wmnet
* 20:38 eevans@puppetmaster1001: conftool action : get/pooled; selector: dnsdisc=$SERVICE,name=$DC
* 20:33 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3064.esams.wmnet with OS bullseye
* 20:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS bullseye
* 20:21 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3063.esams.wmnet
* 20:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3064.esams.wmnet with reason: host reimage
* 20:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3063.esams.wmnet with OS bullseye
* 20:08 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3064.esams.wmnet with reason: host reimage
* 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=ats-be
* 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=cdn
* 20:00 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5031.eqsin.wmnet with OS bullseye
* 19:53 dancy: The train is blocked on [[phab:T328601|T328601]]
* 19:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3064.esams.wmnet with OS bullseye
* 19:49 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.20  refs [[phab:T325584|T325584]] (duration: 06m 36s)
* 19:49 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3062.esams.wmnet
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3062.esams.wmnet with OS bullseye
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3063.esams.wmnet with reason: host reimage
* 19:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3063.esams.wmnet with reason: host reimage
* 19:42 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.20  refs [[phab:T325584|T325584]]
* 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=ats-be
* 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=cdn
* 19:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5021.eqsin.wmnet with OS bullseye
* 19:33 dancy@deploy1002: deploy-promote aborted:  (duration: 11m 58s)
* 19:33 dancy@deploy1002: sync-file aborted: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]] (duration: 03m 38s)
* 19:30 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage
* 19:29 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 19:27 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage
* 19:26 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3062.esams.wmnet with reason: host reimage
* 19:24 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3063.esams.wmnet with OS bullseye
* 19:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3061.esams.wmnet
* 19:24 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3062.esams.wmnet with reason: host reimage
* 19:17 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3061.esams.wmnet with OS bullseye
* 19:04 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage
* 19:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3062.esams.wmnet with OS bullseye
* 19:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3060.esams.wmnet
* 19:02 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3060.esams.wmnet with OS bullseye
* 19:01 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage
* 18:56 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3061.esams.wmnet with reason: host reimage
* 18:55 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:55 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye
* 18:52 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3061.esams.wmnet with reason: host reimage
* 18:47 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:46 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye
* 18:39 jbond@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts puppetmaster2003.codfw.wmnet
* 18:38 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3060.esams.wmnet with reason: host reimage
* 18:37 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:35 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3060.esams.wmnet with reason: host reimage
* 18:32 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3061.esams.wmnet with OS bullseye
* 18:31 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3059.esams.wmnet
* 18:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3059.esams.wmnet with OS bullseye
* 18:29 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye
* 18:29 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts puppetmaster2003.codfw.wmnet
* 18:29 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5021.eqsin.wmnet with OS bullseye
* 18:22 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye
* 18:21 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing
* 18:20 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=ats-be
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=cdn
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=ats-be
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=cdn
* 18:13 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3060.esams.wmnet with OS bullseye
* 18:13 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3058.esams.wmnet
* 18:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3058.esams.wmnet with OS bullseye
* 18:10 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS bullseye
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43573 and previous config saved to /var/cache/conftool/dbconfig/20230201-181036-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43572 and previous config saved to /var/cache/conftool/dbconfig/20230201-181031-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43571 and previous config saved to /var/cache/conftool/dbconfig/20230201-181024-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43570 and previous config saved to /var/cache/conftool/dbconfig/20230201-181016-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43569 and previous config saved to /var/cache/conftool/dbconfig/20230201-181011-root.json
* 18:06 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3059.esams.wmnet with reason: host reimage
* 18:03 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3059.esams.wmnet with reason: host reimage
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43568 and previous config saved to /var/cache/conftool/dbconfig/20230201-175531-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43567 and previous config saved to /var/cache/conftool/dbconfig/20230201-175526-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43566 and previous config saved to /var/cache/conftool/dbconfig/20230201-175519-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43565 and previous config saved to /var/cache/conftool/dbconfig/20230201-175511-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43564 and previous config saved to /var/cache/conftool/dbconfig/20230201-175506-root.json
* 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43563 and previous config saved to /var/cache/conftool/dbconfig/20230201-175446-root.json
* 17:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3058.esams.wmnet with reason: host reimage
* 17:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3058.esams.wmnet with reason: host reimage
* 17:41 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3059.esams.wmnet with OS bullseye
* 17:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3057.esams.wmnet
* 17:40 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3057.esams.wmnet with OS bullseye
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43562 and previous config saved to /var/cache/conftool/dbconfig/20230201-174026-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43561 and previous config saved to /var/cache/conftool/dbconfig/20230201-174021-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43560 and previous config saved to /var/cache/conftool/dbconfig/20230201-174015-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43559 and previous config saved to /var/cache/conftool/dbconfig/20230201-174007-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43558 and previous config saved to /var/cache/conftool/dbconfig/20230201-174001-root.json
* 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43557 and previous config saved to /var/cache/conftool/dbconfig/20230201-173941-root.json
* 17:39 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage
* 17:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43555 and previous config saved to /var/cache/conftool/dbconfig/20230201-172521-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43554 and previous config saved to /var/cache/conftool/dbconfig/20230201-172516-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43553 and previous config saved to /var/cache/conftool/dbconfig/20230201-172510-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43552 and previous config saved to /var/cache/conftool/dbconfig/20230201-172502-root.json
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43551 and previous config saved to /var/cache/conftool/dbconfig/20230201-172456-root.json
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43550 and previous config saved to /var/cache/conftool/dbconfig/20230201-172436-root.json
* 17:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3058.esams.wmnet with OS bullseye
* 17:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3056.esams.wmnet
* 17:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3056.esams.wmnet with OS bullseye
* 17:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3057.esams.wmnet with reason: host reimage
* 17:17 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5019.eqsin.wmnet with OS bullseye
* 17:15 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3057.esams.wmnet with reason: host reimage
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43549 and previous config saved to /var/cache/conftool/dbconfig/20230201-171016-root.json
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43548 and previous config saved to /var/cache/conftool/dbconfig/20230201-171011-root.json
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43547 and previous config saved to /var/cache/conftool/dbconfig/20230201-171005-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43546 and previous config saved to /var/cache/conftool/dbconfig/20230201-170957-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43545 and previous config saved to /var/cache/conftool/dbconfig/20230201-170951-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43544 and previous config saved to /var/cache/conftool/dbconfig/20230201-170931-root.json
* 16:57 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:57 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:57 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3056.esams.wmnet with reason: host reimage
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43543 and previous config saved to /var/cache/conftool/dbconfig/20230201-165512-root.json
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43542 and previous config saved to /var/cache/conftool/dbconfig/20230201-165506-root.json
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43541 and previous config saved to /var/cache/conftool/dbconfig/20230201-165500-root.json
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43540 and previous config saved to /var/cache/conftool/dbconfig/20230201-165452-root.json
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3056.esams.wmnet with reason: host reimage
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43539 and previous config saved to /var/cache/conftool/dbconfig/20230201-165446-root.json
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3057.esams.wmnet with OS bullseye
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43538 and previous config saved to /var/cache/conftool/dbconfig/20230201-165426-root.json
* 16:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:42 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43536 and previous config saved to /var/cache/conftool/dbconfig/20230201-164007-root.json
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43535 and previous config saved to /var/cache/conftool/dbconfig/20230201-164002-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43534 and previous config saved to /var/cache/conftool/dbconfig/20230201-163955-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43533 and previous config saved to /var/cache/conftool/dbconfig/20230201-163947-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43532 and previous config saved to /var/cache/conftool/dbconfig/20230201-163941-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43531 and previous config saved to /var/cache/conftool/dbconfig/20230201-163921-root.json
* 16:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:33 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3056.esams.wmnet with OS bullseye
* 16:31 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:29 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage
* 16:26 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage
* 16:25 jynus: reloaded apache on mailman
* 16:25 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:23 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 16:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 16:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
* 16:14 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
* 16:14 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 16:13 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 15:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye
* 15:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5019.eqsin.wmnet with OS bullseye
* 15:31 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye
* 14:56 sukhe: cp1075.eqiad.wmnet for idrac firmware upgrade testing
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=ats-be
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=cdn
* 14:52 awight: EU deployment window complete
* 14:48 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:48 awight@deploy1002: Finished scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] (duration: 08m 25s)
* 14:47 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:41 awight@deploy1002: elukey and awight: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2136 db2158 db2157 es2026 db2106 db2146 [[phab:T327404|T327404]]', diff saved to https://phabricator.wikimedia.org/P43530 and previous config saved to /var/cache/conftool/dbconfig/20230201-144152-root.json
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:40 awight@deploy1002: Started scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]]
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:37 awight@deploy1002: Finished scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] (duration: 09m 22s)
* 14:29 awight@deploy1002: jdrewniak and awight: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 14:28 awight@deploy1002: Started scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]]
* 14:26 awight@deploy1002: Finished scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 09m 07s)
* 14:19 awight@deploy1002: awight and mlitn: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
* 14:17 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]]
* 14:11 awight@deploy1002: backport aborted:  (duration: 06m 09s)
* 14:11 awight@deploy1002: sync-world aborted: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 03m 36s)
* 14:09 awight@deploy1002: mlitn and awight: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 14:07 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]]
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast3005.wikimedia.org
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 14:06 moritzm: updating perf on Bullseye hosts
* 14:05 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:55 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:51 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast3005.wikimedia.org
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5002.wikimedia.org
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:47 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:43 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:36 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5002.wikimedia.org
* 13:21 moritzm: installing curl security updates on bullseye
* 13:00 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
* 12:59 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2003.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 12:40 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 12:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 12:27 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2003.codfw.wmnet
* 12:16 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002
* 12:15 jmm@cumin2002: START - Cookbook sre.puppet.renew-cert for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002
* 11:29 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part III ([[phab:T308932|T308932]]) (duration: 06m 43s)
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2001.codfw.wmnet
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 11:24 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 11:22 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP (duration: 01m 45s)
* 11:21 ladsgroup@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part II ([[phab:T308932|T308932]]) (duration: 07m 04s)
* 11:21 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 11:20 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP
* 11:17 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet
* 11:17 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP (duration: 00m 51s)
* 11:16 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP
* 11:14 ladsgroup@deploy1002: Synchronized wmf-config/ext-CirrusSearch.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part I ([[phab:T308932|T308932]]) (duration: 07m 04s)
* 11:01 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0] (duration: 01m 18s)
* 11:00 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0]
* 10:59 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0] (duration: 00m 05s)
* 10:59 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0]
* 10:58 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0] (duration: 04m 29s)
* 10:54 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0]
* 10:52 steve_munene: Deploying refinery for ops week
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 10:42 zabe: start running migrateRevisionCommentTemp in remaining sections (for now except s3) in screens # [[phab:T275246|T275246]]
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb2002.codfw.wmnet with OS bullseye
* 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage
* 10:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage
* 10:01 godog: upgrade grafana to 8.5.20 on cloudmetrics* - [[phab:T328405|T328405]]
* 09:57 godog: upgrade grafana to 8.5.20 on grafana1002 - [[phab:T328405|T328405]]
* 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host krb2002.codfw.wmnet with OS bullseye
* 09:47 godog: upgrade grafana to 8.5.20 on grafana2001 - [[phab:T328405|T328405]]
* 09:15 urbanecm: Clean sign up throttle for IP 195.113.145.2 (via resetAuthenticationThrottle.php; [[phab:T328521|T328521]])
* 09:14 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]] (duration: 07m 24s)
* 09:07 urbanecm@deploy1002: Started scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]]
* 09:06 urbanecm@deploy1002: backport aborted:  (duration: 00m 01s)
* 09:05 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] (duration: 15m 06s)
* 08:54 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: apply on main
* 08:54 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
* 08:52 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
* 08:50 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]]
* 08:49 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] (duration: 08m 05s)
* 08:45 jayme@cumin1001: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=k8s-ingress-staging
* 08:45 jayme@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=k8s-ingress-staging
* 08:42 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 08:41 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]]
* 08:40 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] (duration: 12m 26s)
* 08:29 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 08:27 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]]
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 08:27 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] (duration: 09m 42s)
* 08:19 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 6 hosts
* 08:19 jayme@cumin1001: START - Cookbook sre.hosts.remove-downtime for 6 hosts
* 08:19 ladsgroup@deploy1002: ladsgroup and krinkle: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:17 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]]
* 08:14 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] (duration: 10m 15s)
* 08:06 ladsgroup@deploy1002: phedenskog and ladsgroup: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 08:05 moritzm: installing libarchive security updates
* 08:04 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]]
* 08:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 55821
* 07:57 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 55821
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43524 and previous config saved to /var/cache/conftool/dbconfig/20230201-073348-ladsgroup.json
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43523 and previous config saved to /var/cache/conftool/dbconfig/20230201-071841-ladsgroup.json
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43522 and previous config saved to /var/cache/conftool/dbconfig/20230201-070335-ladsgroup.json
* 06:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43521 and previous config saved to /var/cache/conftool/dbconfig/20230201-064828-ladsgroup.json
* 06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43520 and previous config saved to /var/cache/conftool/dbconfig/20230201-064311-ladsgroup.json
* 06:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 06:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 00:38 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3055.esams.wmnet
* 00:37 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3055.esams.wmnet with OS bullseye
* 00:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3055.esams.wmnet with reason: host reimage
* 00:12 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3055.esams.wmnet with reason: host reimage
* 00:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3054.esams.wmnet
* 00:01 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3054.esams.wmnet with OS bullseye


== June 17 ==
==Archives ==
* 23:35 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: SWAT (duration: 00m 14s)
See [[Server Admin Log/Archives]].
* 23:35 gwicke: rolled back restbase to 90817c2a
<noinclude>
* 23:24 logmsgbot: catrope Synchronized php-1.26wmf9/extensions/MobileFrontend: SWAT (duration: 00m 15s)
[[Category:SAL]]
* 23:23 logmsgbot: catrope Synchronized php-1.26wmf9/extensions/Flow: SWAT (duration: 00m 15s)
[[Category:Operations]]
* 22:45 gwicke: rolling restart of cassandra nodes
</noinclude>
* 22:09 gwicke: rolling restart of restbase instances to apply puppet change after puppet actually ran on all nodes
* 21:58 gwicke: rolling restart of restbase instances to apply config change
* 21:56 godog: restart nutcracker on mw1145
* 21:35 gwicke: restarting cassandra on restbase1005
* 20:47 mutante: temp. stopped icinga-wm
* 20:37 gwicke: deployed RESTBase 7ffaf94bfc
* 20:24 cscott: updated Parsoid to version 402ddf66
* 20:01 ottomata: resized antimony's / LV from 30G to 100G.  looks like /var/lib/git was getting filled up
* 19:43 jynus: rolling schema changes on hewiki
* 19:29 godog: downgrade and restart cassandra to 2.1.3 on restbase1001, metrics not being pushed to graphite with 2.1.6
* 19:05 godog: bounce cassandra on xenon
* 18:46 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ic03b152de: Make $wgUploadPath for commons https only for benefit instant commons (duration: 00m 14s)
* 18:11 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf10
* 17:45 godog: bounce cassandra on restbase1001
* 17:39 mutante: repooled mw1234
* 17:24 ottomata: starting reinstall of Zookeeper analytics nodes (analytics102[345]): https://phabricator.wikimedia.org/T101713
* 17:16 godog: bounce cassandra on restbase1001
* 17:14 jynus: rolling schema changes on ruwiki master
* 17:13 mutante: running puppet via salt on api appservers in batches, switch to ganglia_new and carbon
* 17:12 godog: cassandra stopped sending graphite metrics after restart, investigating (test cluster works fine tho)
* 16:58 jynus: rolling schema changes on ruwiki slaves
* 16:28 godog: start upgrading restbase1001 to cassandra 2.1.6 T102015
* 16:02 logmsgbot: thcipriani Finished scap: Wikitech-Ldap host record roll-out (duration: 24m 35s)
* 15:37 logmsgbot: thcipriani Started scap: Wikitech-Ldap host record roll-out
* 15:19 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Give patrolmarks right to "*" on dewiki [[gerrit:218901]] (duration: 00m 13s)
* 15:17 logmsgbot: anomie Synchronized wmf-config/throttle.php: SWAT: Add a throttle exception for United Islands of Prague [[gerrit:217413]] (duration: 00m 14s)
* 15:15 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable captcha on labswiki for now [[gerrit:218908]] (duration: 00m 13s)
* 15:10 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add extra namespace aliases for Italian Wikipedia [[gerrit:215708]] (duration: 00m 13s)
* 15:08 anomie: SWAT: Enable anti-abuse features on labswiki [[gerrit:218903]]
* 15:08 jynus: testing some schema changes on testwiki
* 15:00 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on nowiki and plwiki (duration: 00m 13s)
* 13:56 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on fiwiki and idwiki (duration: 00m 13s)
* 13:26 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on bgwiki and eowiki (duration: 00m 13s)
* 10:52 akosiaris: reload pybal on lvs1006
* 10:50 mobrovac: finished deploying mathoid I40ef68 on SCA
* 10:48 akosiaris: repooled mathoid.svc.eqiad.wmnet: sca1002 backend
* 10:44 akosiaris: enable puppet on sca1002
* 10:43 akosiaris: enable puppet
* 10:43 akosiaris: depool sca1002 for mathoid.svc.eqiad.wmnet
* 10:43 akosiaris: reloaded pybal on lvs1003
* 10:28 akosiaris: repool sca1002, depool sca1001
* 10:18 mark: Halting pvmove of md124 on labstore1001
* 09:30 akosiaris: disable puppet on sca1001
* 09:09 akosiaris: depool sca1001, resource: mathoid
* 09:09 akosiaris: puppet disabled on sca1002
* 08:37 YuviPanda: run sudo salt -t 20 -b 100 '*' cmd.run 'sudo service salt-minion restart' on virt1000, attempt to get them to answer on labcontrol1001 instead
* 06:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 17 06:52:58 UTC 2015 (duration 52m 57s)
* 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-17 02:56:49+00:00
* 02:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1045 (duration: 00m 13s)
* 02:54 springle: found wikiversions.json modified on tin since 2015-06-16 23:27 (catrope?); stashed and reapplied the file in order to do a pull
* 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 04m 44s)
* 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-17 02:35:23+00:00
* 02:32 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 06m 12s)
* 02:21 logmsgbot: ori Synchronized php-1.26wmf9/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I480cbc7ad (duration: 00m 12s)
* 02:21 logmsgbot: ori Synchronized php-1.26wmf10/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I480cbc7ad (duration: 00m 12s)
* 00:10 paravoid: draining esams because of upcoming network maintenance window
 
== June 16 ==
* 23:28 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable local upload on fawikivoyage; enable logging for T76305 (duration: 00m 13s)
* 23:28 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Set previous values for password length policies (duration: 00m 16s)
* 23:17 logmsgbot: twentyafterfour Finished scap: testwiki to 1.26wmf10 (duration: 43m 04s)
* 23:02 godog: restore INFO cassandra logging level on restbase1003
* 22:44 godog: start cassandra on restbase1008
* 22:43 godog: enable back some cassandra debugging on restbase1003
* 22:33 logmsgbot: twentyafterfour Started scap: testwiki to 1.26wmf10
* 22:26 urandom: restored default logging level on restbase1003
* 22:22 urandom: enabling even more debugging on restbase1003
* 22:14 urandom: enable (some) debug logging on restbase1003
* 21:57 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.SxGNHsmVYP" ' returned non-zero exit status 1 (duration: 01m 24s)
* 21:56 logmsgbot: twentyafterfour Started scap: testwiki to 1.26wmf10
* 20:34 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents/modules/ext.wikimediaEvents.resourceloader.js: T101806 live hack (duration: 00m 12s)
* 19:24 Coren: labstore1001 pvmove of slice2 to slice 51 started; some bursts of iowait expected but should have minimal enduser impact)
* 18:36 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Fix usage tracking setting (duration: 00m 14s)
* 18:03 godog: bounce statsite on graphite1001, stuck while writing to graphite
* 17:30 ejegg: update SmashPig on listener from e1e925c9fc2a60c1e14ef01d8b653dc09512f51f to 258f2c917b1ae50b01231927bcd6f58ecaa8940b
* 17:23 logmsgbot: krinkle Synchronized php-1.26wmf9/includes/resourceloader/ResourceLoader.php: undo live hack (duration: 00m 13s)
* 17:09 logmsgbot: aude Synchronized arbitraryaccess.dblist: Enable arbitrary access on gomwiki and lrcwiki (duration: 00m 13s)
* 17:09 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on second batch of s3 wikis (duration: 00m 13s)
* 17:03 logmsgbot: bblack Synchronized wmf-config/InitialiseSettings.php: wgCanonicalServer: HTTPS for all (duration: 00m 15s)
* 16:44 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
* 16:43 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
* 16:43 logmsgbot: krenair Synchronized w/static/images/project-logos/gomwiki.png: (no message) (duration: 00m 14s)
* 16:42 logmsgbot: krenair Synchronized langlist: gomwiki (duration: 00m 13s)
* 16:41 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 16:40 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 13s)
* 16:29 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
* 16:27 logmsgbot: krenair Synchronized langlist: (no message) (duration: 00m 14s)
* 16:25 logmsgbot: krenair Synchronized w/static/images/project-logos/lrcwiki.png: (no message) (duration: 00m 13s)
* 16:21 moritzm: updated copper, oxygen, labstore2001 and labnodepool1001 to the 3.19 kernel
* 16:11 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
* 16:10 logmsgbot: krenair Synchronized wmf-config: (no message) (duration: 00m 14s)
* 16:06 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 16:05 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 15s)
* 15:43 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: templateeditor: add templateeditor right in hewiki [[gerrit:218426]] (duration: 00m 13s)
* 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on wgGenerateThumbnailOnParse for wikitech. [[gerrit:218553]] (duration: 00m 12s)
* 15:03 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for CX deployment on 20150616 [[gerrit:218341]] (duration: 00m 12s)
* 14:18 cmjohnson: barium is going down for disk replacement
* 13:38 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on dewiki (duration: 00m 15s)
* 13:18 akosiaris: rebooted etherpad1001 for kernel upgrades
* 12:51 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2005, es2006 and es2007 after maintenance (duration: 00m 13s)
* 12:44 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on cswiki (duration: 00m 14s)
* 12:20 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on ruwiki (duration: 00m 15s)
* 11:21 paravoid: restarting the puppetmaster
* 11:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1073, warm up (duration: 00m 13s)
* 10:36 akosiaris: rebooting ganeti200{1..6}.codfw.wmnet for kernel upgrades
* 09:33 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2005, es2006 and es2007 for maintenance (duration: 00m 14s)
* 09:10 YuviPanda: deleted huge puppet-master.log on labcontrol1001
* 08:05 jynus: added m5-slave to dns servers
* 07:52 paravoid: restarting hhvm on mw1121
* 07:52 moritzm: blacklisted the overlayfs kernel module (prevents a reliable local root exploit on all Ubuntu systems). no systems in the fleet had an overlaysfs mount present or the kernel module loaded, so there should be no impact on existing systems. Note: This is a bandaid, I'll create a Phab task to deploy this via puppet in the future (and to also blacklist additional desktopy kernel modules which increase our attack
* 07:39 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1005 (duration: 00m 14s)
* 06:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 16 06:24:04 UTC 2015 (duration 24m 3s)
* 06:18 godog: restore ES replication throttling to 20mb/s
* 06:13 godog: restore ES replication throttling to 40mb/s
* 06:08 logmsgbot: filippo Synchronized wmf-config/PoolCounterSettings-common.php: unthrottle ES (duration: 00m 14s)
* 05:56 godog: bump ES replication throttling to 60mb/s
* 05:50 manybubbles: ok - we're yellow and recovering. ops can take this from here. We have a root cause and we have things I can complain about to the elastic folks I plan to meet with today anyway. I'm going to finish waking up now.
* 05:49 manybubbles: reenabling puppet agent on elasticsearch machines
* 05:46 manybubbles: I expect them to be red for another few minutes during the initial master recovery
* 05:45 manybubbles: started all elasticsearch nodes and now they are recovering.
* 05:41 godog: restart gmond on elastic1007
* 05:39 logmsgbot: filippo Synchronized wmf-config/PoolCounterSettings-common.php: throttle ES (duration: 00m 13s)
* 05:25 manybubbles: shutting down all the elasticsearch on the elasticsearch nodes against - another full cluster restart should fix it like it did last time...............
* 05:11 godog: restart elasticsearch on elastic1031
* 03:06 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 12s)
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-16 02:27:51+00:00
* 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 52s)
* 00:55 tgr: running extensions/Gather/maintenance/updateCounts.php for gather wikis - https://phabricator.wikimedia.org/T101460
* 00:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057, warm up (duration: 00m 13s)
* 00:46 godog: killed bacula-fd on graphite1001, shouldn't be running and consuming bandwidth (cc akosiaris)
* 00:27 godog: kill python stats on cp1052, filling /tmp
 
== June 15 ==
* 23:42 ori: Cleaning up renamed jobqueue metrics on graphite{1,2}001
* 23:01 godog: killed bacula-fd on graphite2001, shouldn't be running and consuming bandwidth (cc akosiaris)
* 22:54 logmsgbot: hoo Synchronized wmf-config/filebackend.php: Fix commons image inclusion after commons went https only (duration: 00m 14s)
* 22:18 godog: run disk stress-test on restbase1007 / restbase1009
* 22:06 logmsgbot: twentyafterfour Synchronized hhvm-fatal-error.php: deploy: Guard header() call in error page (duration: 00m 15s)
* 22:05 logmsgbot: twentyafterfour Synchronized wmf-config/InitialiseSettings-labs.php: deploy: Never use wgServer/wgCanonicalServer values from production in labs (duration: 00m 12s)
* 20:37 logmsgbot: yurik Synchronized docroot/bits/WikipediaMobileFirefoxOS: Bumping FirefoxOS app to latest (duration: 00m 14s)
* 20:30 godog: bounce cassandra on restbase1003
* 20:18 godog: start cassandra on restbase1008, bootstrapping
* 20:04 godog: sign restbase1008 key, run puppet
* 20:00 godog: powercycle restbase1007, investigate disk issue
* 19:07 logmsgbot: ori Synchronized php-1.26wmf9/includes/jobqueue: 0a32aa3be4: jobqueue: use more sensible metric key names (duration: 00m 13s)
* 16:57 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT:  Grant cloudadmins the 'editallhiera' right [[gerrit:218115]] (duration: 00m 14s)
* 16:48 logmsgbot: thcipriani Synchronized php-1.26wmf9/extensions/OpenStackManager/OpenStackManagerHooks.php: SWAT: refer to user the right way (duration: 00m 13s)
* 16:48 godog: powercycle graphite1002, no ssh, unresponsive console
* 16:19 jynus: upgrading es1005 mysql service while depooled
* 16:12 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT:  Grant cloudadmins the 'editallhiera' right [[gerrit:218115]] (duration: 00m 12s)
* 16:10 bblack: pybal restarts complete, all ok
* 16:09 logmsgbot: thcipriani Finished scap: SWAT: Openstack manager and language updates (duration: 21m 27s)
* 15:47 logmsgbot: thcipriani Started scap: SWAT: Openstack manager and language updates
* 15:46 bblack: starting pybal restart process for config changes ( https://gerrit.wikimedia.org/r/#/c/218285/ ), inactives first w/ manual verification of ok-ness
* 15:11 bblack: rebooting cp3041 (downtimed)
* 15:00 _joe_: ES is green
* 14:38 logmsgbot: aude Synchronized php-1.26wmf9/extensions/Wikidata: Fix property label constraints bug (duration: 00m 24s)
* 14:27 logmsgbot: aude Synchronized arbitraryaccess.dblist: Enable arbitrary access on s7 wikis (duration: 00m 13s)
* 13:47 jynus: enabling puppet on all elastic* nodes, should enable also ganglia
* 13:11 logmsgbot: demon Synchronized wmf-config/PoolCounterSettings-common.php: all the search (duration: 00m 12s)
* 13:04 _joe_: re-scaling down the recovery index bandwidth in ES to 20 mb/s
* 12:52 logmsgbot: demon Synchronized wmf-config/PoolCounterSettings-common.php: partially turn search back on (duration: 00m 13s)
* 11:54 _joe_: raised the ES index replica bandwidth limit to 60mb
* 11:31 akosiaris: migrating etherpad.wikimedia.org to etherpad1001.eqiad.wmnet
* 11:15 _joe_: raised the max bytes for ES recovery to 40mbps
* 10:49 manybubbles: and we're yellow right now.
* 10:49 manybubbles: the initial primaries stage - the red stage of the rolling restart - recovers quick-ish
* 10:48 manybubbles: soon we should see it go yellow and stay that way while the replicas recover
* 10:48 manybubbles: manybubbles is confident his mighty bitch slap of the elasticsearch cluster has set it further to the road to recovery
* 10:46 jynus: disabled puppet on all elasticsearch nodes to avoid restarting services and other magic
* 10:44 _joe_: disabled hot threads logging, ganglia on es nodes
* 10:44 manybubbles: started Elasticsearch on all elasticsearch nodes
* 10:38 manybubbles: stopping all elasticsearch servers - going for a full cluster resstart.
* 10:11 manybubbles: restarting elasticsearch on elasticsearch1021 - that one is in a gc death spiral
* 09:26 logmsgbot: oblivian Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s)
* 09:12 logmsgbot: oblivian Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s)
* 07:35 _joe_: attempting a fast restart of elastic1020
* 07:21 logmsgbot: ori Synchronized php-1.26wmf9/extensions/CirrusSearch/includes/Util.php: I504dac0c3: Add missing 'use \Status;' to includes/Util.php (duration: 00m 13s)
* 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 15 04:56:39 UTC 2015 (duration 56m 38s)
* 03:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 12s)
* 02:22 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-15 02:22:56+00:00
* 02:19 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 46s)
 
== June 14 ==
* 10:39 YuviPanda: running du -d 2 on /srv/project in a screen sesssion on labstore1001
* 04:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 14 04:33:20 UTC 2015 (duration 33m 19s)
* 02:42 logmsgbot: reedy Synchronized wmf-config/extension-list: noop (duration: 00m 13s)
* 02:40 logmsgbot: krenair Synchronized wmf-config/squid-labs.php: sync random labs-only file to test per irc (duration: 00m 13s)
* 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-14 02:21:28+00:00
* 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 47s)
 
== June 13 ==
* 19:30 bblack: repooled cp1071, cp3040
* 18:53 bblack: rebooting cp1071, cp3040 to look at BIOS-level things (depooled, icinga-downed)
* 17:08 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents: T101806 (duration: 00m 12s)
* 15:47 paravoid: labstore1001: stopping manage-nfs-volumes daemon
* 04:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 13 04:41:57 UTC 2015 (duration 41m 56s)
* 03:51 Krinkle: Running deleteEqualMessages.php for sawiki (T45917)
* 03:49 Krinkle: Running deleteEqualMessages.php for cewiki (T45917)
* 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-13 02:20:58+00:00
* 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 19s)
* 00:17 gwicke: restarted cassandra on restbase1001
* 00:13 gwicke: restarted cassandra on restbase1002
 
== June 12 ==
* 22:57 ejegg: rolled back SmashPig on listener from 15acdafef9d9682c417632e5ac5a5f2e5380f92e to e1e925c9fc2a60c1e14ef01d8b653dc09512f51f
* 22:40 ejegg: updated SmashPig on listener from e1e925c9fc2a60c1e14ef01d8b653dc09512f51f to 15acdafef9d9682c417632e5ac5a5f2e5380f92e
* 22:24 godog: upgrade and bounce carbon daemons on graphite2001 to investigate T101572
* 21:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I3694489ba: wgCanonicalServer->https for new HTTPS domains (duration: 00m 14s)
* 20:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 13s)
* 20:32 logmsgbot: krenair Synchronized w/static/images/project-logos/dawiki-200k.png: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 16s)
* 20:15 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217670/ (duration: 00m 12s)
* 19:28 ejegg: updated SmashPig on payments-listener from f9c3eaa99fa0fe8ef098d0fc876091d3676aa039 to 5a463400bc74706ba7bf6256cd0101014e792acb
* 19:28 ejegg: updated SmashPig on payments-listener ccepting New Patients:
* 18:47 ejegg: updated SmashPig on payments-listener from 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510 to f9c3eaa99fa0fe8ef098d0fc876091d3676aa039
* 18:45 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: remove wmgHTTPSBlacklistCountries (duration: 00m 12s)
* 18:45 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: remove CanIPUseHTTPS hook (duration: 00m 13s)
* 17:39 moritzm: updated cerium, xenon and praseodymium to 3.19 kernel
* 17:08 ejegg: enabled queue consumer
* 17:08 ejegg: updated crm from d13aaa4e9e937b0b1ae1f5de61ea7ff1f316d58f to bd8a00196071ddd04efbff7b30567dd9357c9000
* 16:53 ejegg: disabled donations queue consumer
* 15:52 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: hide prefershttps user pref (duration: 00m 13s)
* 15:40 logmsgbot: faidon Synchronized docroot/search.wikimedia.org/index.php: unbreak search.wikimedia.org due to HTTPS (duration: 00m 12s)
* 15:27 jynus: mysql load issues on labsdb1003, investigating
* 13:39 moritzm: updated etcd* to 3.19 kernel
* 12:11 jynus: restarting mariadb at labsdb1003
* 11:58 moritzm: updated rdb200* to 3.19 kernel
* 11:31 jynus: db2068 up but all services and console login unresponsive, powercycling
* 10:06 springle: killed a bunch of queries hammering labsdb1003 for days
* 09:58 moritzm: updated mc2004 to mc2016 to 3.19 kernel
* 06:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 12 06:06:55 UTC 2015 (duration 6m 54s)
* 04:37 logmsgbot: ori Synchronized php-1.26wmf9/extensions/FlaggedRevs: I4cfb47b41: Avoid post-redirect parse for certain edits (duration: 00m 14s)
* 02:40 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-12 02:40:36+00:00
* 02:34 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 10m 00s)
* 00:40 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217759 (duration: 00m 15s)
* 00:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 14s)
 
== June 11 ==
* 23:59 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217753 (duration: 00m 16s)
* 23:54 logmsgbot: ori Synchronized php-1.26wmf9/includes/EditPage.php: cf7df757f2: Instrument edit failures (duration: 00m 14s)
* 23:41 logmsgbot: ebernhardson Synchronized php-1.26wmf9/extensions/MobileFrontend: Bump MobileFrontend in 1.26wmf9 for SWAT (duration: 00m 14s)
* 23:40 ejegg: updated civicrm from 7ffe0cefb019828a09c9369187f14518847b5f41 to d13aaa4e9e937b0b1ae1f5de61ea7ff1f316d58f
* 23:24 logmsgbot: ebernhardson Synchronized php-1.26wmf9/extensions/CirrusSearch/: Fix prefer-recent queries in cirrussearch (duration: 00m 13s)
* 23:02 ejegg: updated SmashPig on the rest of the cluster from 477e8a8be5ea895262031c147330de5a651cc3ac to 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510
* 22:17 godog: temporary bump php memory_limit on magnesium to test T102092
* 22:11 ejegg: updated SmashPig on payments-listener from 477e8a8be5ea895262031c147330de5a651cc3ac to 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510
* 21:54 ori: Widespread TC cache exhaustion again, doing rolling restart of HHVMs
* 21:46 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I3d3ed7647: Test LCStoreStaticArray on test2wiki (duration: 00m 14s)
* 21:01 godog: NPE while trying to make restbase1007 (cassandra 2.1.5) join the cluster, trying matching the same cassandra version (2.1.3)
* 20:57 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: fix last commit, did not have any affect (duration: 00m 16s)
* 20:55 ejegg: updated payments from 43c7952d2a31deaea97e8319f5612d644dce43c8 to f33d0a8687a120a2057a7e6acad67da63b17f97e
* 20:54 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217688/1 (duration: 00m 13s)
* 20:10 godog: sign restbase1007 puppet key and first puppet run
* 19:10 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217591 (duration: 00m 13s)
* 18:58 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: beta only change - https://gerrit.wikimedia.org/r/217560 (duration: 00m 12s)
* 18:55 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents: T101806 (duration: 00m 14s)
* 18:43 logmsgbot: twentyafterfour Synchronized php-1.26wmf9/includes/AjaxResponse.php: Hotfix Iafff9982bbbee893c13f891901dde88f998db7a6 (duration: 00m 14s)
* 18:16 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf9
* 17:44 ejegg: rolled back payments to 43c7952d2a31deaea97e8319f5612d644dce43c8
* 17:41 ejegg: updated payments from 43c7952d2a31deaea97e8319f5612d644dce43c8 to 15f24d24b150d5d774314b0c1b40ae26a73185f2
* 17:00 moritzm: updated mc200[1-3] to linux 3.19
* 16:28 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Use arbitrary access tag (duration: 00m 12s)
* 16:27 logmsgbot: aude Synchronized wmf-config/CommonSettings.php: Add arbitrary access group tag (duration: 00m 13s)
* 16:27 logmsgbot: aude Synchronized arbitraryaccess.dblist: Add dblist for arbitrary access wikis (duration: 00m 13s)
* 16:24 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Use usagetracking tag (duration: 00m 13s)
* 16:23 logmsgbot: aude Synchronized wmf-config/CommonSettings.php: Add usagetracking group tag (duration: 00m 16s)
* 16:23 ori: Scap + deployments exhausted TC cache on Apaches; performed a rolling restart of HHVM
* 16:21 logmsgbot: aude Synchronized usagetracking.dblist: Add dblist for usage tracking wikis (duration: 00m 25s)
* 16:19 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Disable Parsoid update jobs (duration: 00m 14s)
* 16:18 logmsgbot: thcipriani Finished scap: SWAT: Update namespaces and special pages for Northern Luri (lrc) from translatewiki [[gerrit:216533]] [[gerrit:217327]] (duration: 32m 11s)
* 15:46 logmsgbot: thcipriani Started scap: SWAT: Update namespaces and special pages for Northern Luri (lrc) from translatewiki [[gerrit:216533]] [[gerrit:217327]]
* 15:27 logmsgbot: thcipriani Synchronized php-1.26wmf9/extensions/OpenStackManager: SWAT: update OpenStackManager to disable unused sudoer features [[gerrit:217407]] (duration: 00m 13s)
* 15:11 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Make VisualEditor access RESTbase directly on all public wikis [[gerrit:214833]] (duration: 00m 12s)
* 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150611 [[gerrit:217460 ]] (duration: 00m 12s)
* 14:33 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on jawiki (duration: 00m 12s)
* 13:40 _joe_: rolling restart of all the restbase instances
* 13:33 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on frwiki (duration: 00m 12s)
* 13:32 _joe_: running puppet on all restbase hosts
* 13:19 _joe_: running puppet on restbase1001
* 13:16 _joe_: disabling puppet on restbase hosts in anticipation for merging https://gerrit.wikimedia.org/r/217431
* 13:11 paravoid: removing gdnsd from apt: precise-wikimedia (1.9.0-1~precise1/2.1.0-1~precise1), trusty-wikimedia (2.1.0-1), jessie-wikimedia (2.1.2-1~deb8u1)
* 12:13 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on Wikivoyage and Wikiquote (duration: 00m 13s)
* 11:48 YuviPanda: reboot labvirt1005 for kernel upgrade
* 11:46 YuviPanda: installing linux-image-generic-lts-vivid on labvirt1005 to get a 3.19 kernel
* 09:51 akosiaris: uploaded ruby-jsduck_5.3.4 and ruby-rkelly-remix_0.0.6 on apt.wikimedia.org/jessie-wikimedia/main
* 08:18 akosiaris: recreating jessie chroots on copper
* 06:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 11 06:21:53 UTC 2015 (duration 21m 52s)
* 04:44 twentyafterfour: upgraded phabricator at 1:50 UTC (belatedly logged...)
* 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-11 03:01:48+00:00
* 03:00 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057, warm up (duration: 01m 16s)
* 02:59 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 59s)
* 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-11 02:43:34+00:00
* 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 09m 13s)
 
== June 10 ==
* 23:23 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Add www.limis.lt to $wgCopyUploadsDomains (duration: 00m 19s)
* 22:07 logmsgbot: twentyafterfour Synchronized php-1.26wmf9/extensions/MobileFrontend/includes/skins/banners.mustache: Deploying https://gerrit.wikimedia.org/r/#/c/217417/ (duration: 00m 16s)
* 20:38 logmsgbot: ori Synchronized php-1.26wmf8/includes/Hooks.php: d6802ad7d6: Avoid section profiling in Hooks::run due to high overhead (duration: 00m 14s)
* 20:37 logmsgbot: ori Synchronized php-1.26wmf9/includes/Hooks.php: e552f4942d: Avoid section profiling in Hooks::run due to high overhead (duration: 00m 17s)
* 20:36 logmsgbot: ori Synchronized php-1.26wmf9/includes/User.php: 2f4f1e279d: Fixed "wfTimestamp() fed bogus time value" errors (duration: 00m 12s)
* 20:36 logmsgbot: ori Synchronized php-1.26wmf8/includes/User.php: 55e18123ca: Fixed "wfTimestamp() fed bogus time value" errors (duration: 00m 15s)
* 18:07 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf9
* 16:14 godog: reboot ms-be2008 to check disk swap config
* 15:50 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: retry (duration: 01m 08s)
* 15:34 Krenair: sync failed to something like 25 hosts, cannot directly log into any of them either
* 15:17 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/215030/ - no code change, just docs - should not have to wait 9 days for this (duration: 01m 08s)
* 13:16 moritzm: installed curl security updates on elastic*, wtp*, db*, virt*, labs*, labmon*, labstore*, es*
* 12:38 paravoid: zirconium: rm -rf /var/log2 (last log there from Mar 20th 2014)
* 10:55 jynus: disruption for maintenance starting on labsdb1002 https://lists.wikimedia.org/pipermail/labs-l/2015-June/003766.html
* 03:02 logmsgbot: ori Synchronized php-1.26wmf8/includes/User.php: 55e18123ca: Fixed "wfTimestamp() fed bogus time value" (duration: 01m 07s)
* 03:01 logmsgbot: ori Synchronized php-1.26wmf9/includes/User.php: 2f4f1e279d: Fixed "wfTimestamp() fed bogus time value" (duration: 01m 08s)
* 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-10 02:35:44+00:00
* 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 20s)
* 01:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 01m 08s)
* 01:13 logmsgbot: ori Synchronized php-1.26wmf8/extensions/FlaggedRevs: 433fae7f23: Update FlaggedRevs for cherry-picks (duration: 01m 09s)
* 01:10 logmsgbot: ori Synchronized php-1.26wmf9/extensions/FlaggedRevs: 2cfc8c9f2b: Update FlaggedRevs for cherry-picks (duration: 01m 09s)
 
== June 9 ==
* 23:57 logmsgbot: catrope Synchronized php-1.26wmf8/includes/: Avoid parser cache miss that often occurs post-save (duration: 01m 14s)
* 23:29 logmsgbot: catrope Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: touch (duration: 01m 08s)
* 23:23 logmsgbot: catrope Synchronized php-1.26wmf9/includes/resourceloader/ResourceLoaderOOUIImageModule.php: Fix OOUI image variants (duration: 01m 08s)
* 23:22 ori: Deleting unused metrics on graphite2001 (sum_sq and stddev) as well
* 23:21 logmsgbot: catrope Synchronized php-1.26wmf9/resources/src/mediawiki/mediawiki.js: Add logging for T101806 private modules (duration: 01m 08s)
* 23:20 ori: Deleting unused  metrics in graphite1001 (sum_sq and stddev)
* 23:19 logmsgbot: catrope Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: Add logging for T101806 private modules (duration: 01m 08s)
* 23:16 logmsgbot: catrope Synchronized wmf-config/CirrusSearch-common.php: fix total breakage of search in wmf9 (duration: 01m 08s)
* 22:44 andrewbogott: moving labs-ns0 from virt1000 to labcontrol1001
* 22:43 andrewbogott: stopping almost everything on virt1000
* 20:31 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf9
* 20:27 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf9 and rebuild l10n cache (duration: 29m 24s)
* 19:58 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf9 and rebuild l10n cache
* 19:42 mutante: einsteinium - no console output after reboot command, powercycled, booting again
* 19:36 mutante: rebooting einsteinium
* 19:28 mutante: restarted apache on mw1227
* 17:30 mutante: wikitech-static: installing bunch of package upgrades on the external wikitech-static VM
* 17:13 cmjohnson1: db1058 replacing failed disk 7
* 16:20 cmjohnson1: analytics1028 going down for troubleshooting
* 16:17 kart_: updated cxserver to 4a71145
* 15:37 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Wikidata: SWAT: Update Wikidata - forward compat for usage tracking [[gerrit:216967]] (duration: 01m 17s)
* 15:20 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT take II: Enabled Guided Tour on th.wikipedia [[gerrit:216950]] (duration: 01m 08s)
* 15:19 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enabled Guided Tour on th.wikipedia [[gerrit:216950]] (duration: 01m 08s)
* 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150609 [[gerrit:216622]] (duration: 01m 09s)
* 11:09 Krenair: Email set for User:GifTagger@commonswiki per [[phab:T100889]]
* 09:05 akosiaris: uploaded etherpad-lite_1.5.6-2 on apt.wikimedia.org/jessie-wikimedia/main component
* 08:22 akosiaris: upload etherpad-lite_1.5.6-1 on apt.wikimedia.org, jessie-wikimedia dist, main component
* 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun  9 04:34:08 UTC 2015 (duration 34m 7s)
* 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-09 02:27:30+00:00
* 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 12s)
* 01:42 godog: stop icinga-wm on neon
 
== June 8 ==
* 23:43 bblack: repooled cp3030/cp1065 in pybal
* 23:11 logmsgbot: ebernhardson Synchronized php-1.26wmf8/extensions/UploadWizard/: Bump UploadWizard in 1.26wmf8 for evening SWAT (duration: 01m 09s)
* 22:21 bblack: depooled cp3030, cp1065 in pybal for ipsec
* 20:17 subbu: deployed parsoid sha 131554ba
* 19:18 jynus: RAID degradation (disk failure) on s5 master (db1058), no production impact, replacement on the way
* 17:13 ottomata: restarted eventlogging services on eventlog1001 after disabling kafka pieces
* 16:13 _joe_: powercycling tmh1001, console blank, unresponsive to pings
* 16:00 logmsgbot: thcipriani Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia, for real [[gerrit:216719]] (duration: 01m 07s)
* 15:58 logmsgbot: thcipriani Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia [[gerrit:216719]] (duration: 01m 08s)
* 15:40 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Cite: SWAT: Revert Do all of Cite's real work during unstrip and followup [[gerrit:216715]] (duration: 01m 08s)
* 15:19 Coren: T96063: process halted for now as store/backup is unmovable and on slice5
* 15:17 logmsgbot: thcipriani Synchronized w/static/images/project-logos/pflwiki.png: SWAT: Fix transparency of pflwiki logo [[gerrit:216595]] (duration: 01m 08s)
* 15:15 akosiaris: disabled ircecho on neon for a while
* 14:53 Coren: T96063: starting pvmove from slice5 to slice2
* 14:48 Coren: T96063: dropped volume slice1 from vg store
* 14:46 Coren: T96063: dropped store/project
* 14:44 Coren: starting https://phabricator.wikimedia.org/T96063 on labstore1001
* 14:24 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1005 (duration: 01m 08s)
* 14:23 Coren: rsync in progress between labstore1001:store/backup and labstore1002:backup/backup (at ionice idle)
* 14:13 Coren: created store/backup snapshot on labstore1001 for backup copy
* 13:03 moritzm: added strongswan_5.3.0-1+wmf2 to jessie-wikimedia on carbon
* 11:42 _joe_: purging squid cache on carbon
* 11:26 moritzm: updated mc2* to 2:2.8.17-1+deb8u1
* 10:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1007 (duration: 01m 08s)
* 10:27 akosiaris: disabled puppet on uranium, investigating ganglia problems
* 10:05 akosiaris: ganglia gmetad problems
* 05:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun  8 05:24:08 UTC 2015 (duration 24m 7s)
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-08 02:25:12+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 07s)
 
== June 7 ==
* 23:27 godog: reboot ms-be2008 sdg failed, xfs unhappy
* 07:03 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1073, warm up (duration: 01m 09s)
* 05:16 andrewbogott: we did a whole lot of things to labstore1001 while morebots was away
* 05:14 andrewbogott: service nfs-kernel-server restart on labstore1001
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-07 02:25:13+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s)
 
== June 6 ==
* 23:46 subbu: deployed parsoid 5172a446 (cherry-pick of 719c736f) -- hotfix for T101599
* 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun  6 05:47:40 UTC 2015 (duration 47m 39s)
* 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-06 02:30:24+00:00
* 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 10s)
 
== June 5 ==
* 22:42 godog: powercycle graphite2001, no console no ssh
* 22:06 andrewbogott: restarted apache on virt1000
* 20:49 ori: Upgrading hhvm-fss on application servers to 1.1.7; expect brief 5xx spike.
* 20:14 logmsgbot: demon Synchronized php-1.26wmf8: live hack (duration: 02m 32s)
* 20:10 mutante: apt-get upgrade on terbium
* 19:52 godog: bounce redis on rdb1001/rdb1003 to pick up new slave limits
* 19:51 mutante: chown root:root / on terbium
* 19:50 godog: bounce redis on rdb1002/rdb1004 to pick up new slave limits
* 19:29 godog: bounce redis again on rdb1003 after increasing the slave limits more
* 19:17 godog: bounce redis on rdb1003 after bumping slave limits
* 19:07 godog: redis master logs shows periodic 'cmd=sync scheduled to be closed ASAP for overcoming of output buffer limits.' indicating the slave fails to sync
* 18:40 godog: spike in redis network starting at ~15.00 UTC, correlates with ocg failures
* 18:01 moritzm: restarted gerrit on ytterbium for java update
* 14:43 jynus: short lag period on db1049, traffic automatically redirected to other slave and back to normal
* 14:07 moritzm: added ubuntu-meta-1.325+wmf1 for trusty-wikimedia to apt.wikimedia.org (T100004)
* 14:07 moritzm: added ubuntu-meta-1.267.1+wmf1 for precise-wikimedia to apt.wikimedia.org (T100004)
* 12:44 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1007 (duration: 01m 08s)
* 12:08 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1009 (duration: 01m 08s)
* 11:30 _joe_: uploaded new HHVM package, installing on mw1025 for testing
* 09:17 moritzm: added redis_2.6.13-1+wmf1 to precise-wikimedia on apt.wikimedia.org
* 06:24 moritzm: added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org
* 05:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun  5 05:22:50 UTC 2015 (duration 22m 49s)
* 04:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 01m 08s)
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-05 02:25:20+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s)
* 01:27 tgr: deploying schema changes for Gather on enwiki, enwikivoyage, hewiki (T98490, T101460)
* 00:08 logmsgbot: catrope Synchronized php-1.26wmf8/vendor/oojs/oojs-ui/php/Tag.php: Fix OOUI fatals (T99210) (duration: 00m 13s)
 
== June 4 ==
* 23:40 logmsgbot: catrope Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT (duration: 00m 13s)
* 23:28 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Disable VE A/B test for new accounts on enwiki (duration: 00m 13s)
* 22:39 ejegg: updated payments from d22e44e3fab2b937707c2776384cb93a49b4cfd3 to 43c7952d2a31deaea97e8319f5612d644dce43c8
* 22:21 ottomata: doing controlled restart of kafka brokers services to apply auto create topic config
* 21:48 jgage: analyics1013 crashed, rebooted
* 21:42 logmsgbot: ori Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: 1b20d62c26: Revert "awful hack: disable fss on zhwiki only, except on mw1017" (duration: 00m 13s)
* 21:34 ori: performing rolling restart of HHVMs for hhvm-fss upgrade
* 21:27 bd808: restarted logstash and elasticsearch on logstash100[1-3] to pick up latest jre updates
* 18:48 mutante: restarted apache on silver/wikitech
* 18:20 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1009 and master-slave switchover (duration: 00m 13s)
* 18:01 awight: Enabling PayPal audit parser job
* 17:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1008 (duration: 00m 15s)
* 17:44 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2008 and its slaves (duration: 00m 13s)
* 17:21 ori: Disabling Puppet and nutcracker on mw1017 to control for parser cache
* 17:18 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2008 and its slaves (duration: 00m 13s)
* 17:17 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1008 (duration: 00m 12s)
* 16:33 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 09m 17s)
* 16:23 logmsgbot: kartik Started scap: Update ContentTranslation
* 15:54 moritzm: added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org
* 15:48 logmsgbot: anomie Synchronized php-1.26wmf8/includes/jobqueue/: SWAT: jobqueue: Record stats on how long it takes before a job is run [[gerrit:215748]] (duration: 00m 14s)
* 15:38 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ApiFeatureUsage everywhere [[gerrit:215901]] (duration: 00m 19s)
* 15:36 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Remove obsolete 'ValidateExtendedMetadataCache' hook [[gerrit:215900]] (duration: 00m 12s)
* 15:35 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Added staff-recommender campaign [[gerrit:215865]] (duration: 00m 12s)
* 15:30 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150406 [[gerrit:215281]] (duration: 00m 12s)
* 15:12 logmsgbot: ori Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: Ia5f3dc84605: awful hack: disable fss on zhwiki only, except on mw1017 (duration: 00m 17s)
* 15:09 _joe_: puppet disabled, fss disabled on mw1017
* 14:42 YuviPanda: running sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on all labs nodes
* 14:36 awight: Disable PayPal audit parsing job
* 12:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1072, warm up (duration: 00m 13s)
* 05:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun  4 05:11:32 UTC 2015 (duration 11m 31s)
* 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-04 02:28:54+00:00
* 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 22s)
 
== June 3 ==
* 23:42 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing ImportSource change for meta (duration: 00m 13s)
* 23:34 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing config change for mediawiki logo on mobile, take 2 (duration: 00m 12s)
* 23:26 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing config change for mediawiki logo on mobile (duration: 00m 12s)
* 23:25 logmsgbot: kaldari Synchronized images/mobile/mediawiki.png: syncing mediawiki logo for mobile (duration: 00m 12s)
* 22:02 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on ukwiki and viwiki (duration: 00m 15s)
* 21:58 mutante: restarted gitblit
* 21:53 logmsgbot: ori Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoader.php: 7f49853fc9: ResourceLoader::filter: use APC when running under HHVM (did not sync correct file previously) (duration: 00m 12s)
* 21:20 andrewbogott: restarting pdns on virt1000 and labcontrol1001
* 21:05 Jamesofur: decryption key for Board Election insert into voteWiki
* 20:58 bblack: repooling ns0 -> radon AuthDNS
* 20:55 bblack: depooling ns0 -> radon AuthDNS (rebooting for kernel update)
* 20:50 hashar: restarted zuul entirely to remove some stalled jobs
* 20:29 paravoid: kafka preferred-replica-election on an1021
* 20:28 hashar: Restarting Jenkins to release a deadlock
* 20:23 logmsgbot: ori Synchronized php-1.26wmf8/resources/Resources.php: 7f49853fc9: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 13s)
* 20:19 subbu: deployed parsoid sha ab675400
* 19:08 bblack: changed ops/puppet repo to ff-only in gerrit config, feel free to scream/revert if necc!
* 18:46 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: All wikis to 1.26wmf8, no new branch until next Tuesday, June 9th
* 18:42 logmsgbot: twentyafterfour Finished scap: Delete stale branch symlinks (1.26wmf1,1.26wmf2) (duration: 07m 14s)
* 18:35 logmsgbot: twentyafterfour Started scap: Delete stale branch symlinks (1.26wmf1,1.26wmf2)
* 15:16 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Remove references to $wgEchoCohortInterval (duration: 00m 12s)
* 15:16 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Change default extension distributor branch to REL1_25 (duration: 00m 15s)
* 15:15 bblack: repooling ns1->baham DNS traffic
* 15:07 bblack: depooling ns1->baham DNS traffic for kernel update
* 15:00 moritzm: added linux 3.19.3-5 for jessie-wikimedia on apt.wikimedia.org
* 14:46 bblack: restarted hhvm on mw1195, seems to be a case of https://phabricator.wikimedia.org/T89912
* 14:32 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on huwiki (duration: 00m 12s)
* 14:29 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2008, es2009 and es2010 (duration: 00m 14s)
* 14:10 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on eswiki (duration: 00m 13s)
* 13:38 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2008, es2009 and es2010 (duration: 00m 14s)
* 13:12 paravoid: reimaging rubidium with trusty, as spare
* 13:02 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on arwiki and cawiki (duration: 00m 15s)
* 12:56 paravoid: permanently switching ns0 to radon instead of rubidium
* 12:53 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2009 (duration: 00m 15s)
* 11:04 paravoid: kafka preferred-replica-election on an1021
* 10:55 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2009 (duration: 00m 13s)
* 10:43 paravoid: powercycling ms-be1005
* 10:28 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool es2010 (duration: 00m 14s)
* 10:24 moritzm: added linux-meta 1.2 for jessie-wikimedia on carbon.wikimedia.org
* 10:09 hashar: Jenkins: refreshing all jobs to get rid of an obsolete http notification to Zuul {{bug|T93321}}
* 09:48 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1008 (duration: 00m 13s)
* 09:00 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: depool es2010 (duration: 00m 13s)
* 08:51 moritzm: removed fuse/ntfs-3g from wtp*
* 07:47 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1008 (duration: 00m 14s)
* 05:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun  3 05:41:31 UTC 2015 (duration 41m 30s)
* 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-03 02:48:55+00:00
* 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 06m 37s)
* 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-03 02:27:38+00:00
* 02:25 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1072 (duration: 00m 12s)
* 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 07m 13s)
* 01:57 springle: replicate m3 to codfw dbstore2001
* 01:37 springle: start sync m4 eventlogging to codfw dbstore2002
* 00:35 logmsgbot: mattflaschen Synchronized php-1.26wmf8/extensions/Calendar/: Sync Calendar 1.26wmf8 for module position (duration: 00m 12s)
* 00:20 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/User.php: Fixed $flags bit operation precedence fail in User::loadFromDatabase() (duration: 00m 14s)
 
== June 2 ==
* 23:56 logmsgbot: mattflaschen Synchronized php-1.26wmf8/extensions/Flow/: Sync Flow 1.26wmf8 for import fix (duration: 00m 15s)
* 23:43 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Disable WikiGrok (duration: 00m 13s)
* 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoaderStartUpModule.php: Don't cache minification of user.tokens (duration: 00m 15s)
* 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoader.php: Don't cache minification of user.tokens (duration: 00m 13s)
* 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/OutputPage.php: Don't cache minification of user.tokens (duration: 00m 14s)
* 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/resourceloader/ResourceLoaderStartUpModule.php: Don't cache minification of user.tokens (duration: 00m 13s)
* 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/resourceloader/ResourceLoader.php: Don't cache minification of user.tokens (duration: 00m 14s)
* 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/OutputPage.php: Don't cache minification of user.tokens (duration: 00m 13s)
* 21:44 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I263aa9542: Set $wgExtDistUseEventLogging = true; (duration: 00m 13s)
* 21:43 logmsgbot: ori Synchronized php-1.26wmf8/extensions/ExtensionDistributor: cdd033e7d8: Update ExtensionDistributor for cherry-picks (duration: 00m 13s)
* 19:24 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I7810b72d5: Sample profiling data at 1:10,000 (duration: 00m 12s)
* 19:19 logmsgbot: ori Synchronized wmf-config: I35255f357 and I026dfdbf68 (duration: 00m 12s)
* 19:15 logmsgbot: aude Synchronized wmf-config/Wikibase.php: bump cache epoch for wikidata (duration: 00m 13s)
* 19:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: wgMaxCredits to 0 (duration: 00m 13s)
* 18:53 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf8
* 18:46 robh: sodium has resumed normal service. all items on https://phabricator.wikimedia.org/T100711 addressed
* 17:56 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1010 (duration: 00m 12s)
* 17:18 robh: mailing list traffic halted for list renames
* 17:07 robh: lists.wikimedia.org is now sha256 cert
* 17:04 robh: starting the lists.wikimedia.org certificate update, archives will offline during this process
* 15:44 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1010 (duration: 00m 13s)
* 15:03 logmsgbot: thcipriani Synchronized wmf-config/wikitech.php: SWAT: No longer set use_dnsmasq for new instances. [[gerrit:215317]] (duration: 00m 12s)
* 12:31 twentyafterfour: merged https://gerrit.wikimedia.org/r/#/c/214288/ and deployed scap
* 12:18 moritzm: installed linux-tools-3.19.8-1 for jessie-wikimedia on carbon
* 07:36 logmsgbot: nikerabbit Synchronized wmf-config/InitialiseSettings.php: Fixed wiki id for fiu_vro for CX beta feature (duration: 00m 13s)
* 05:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun  2 05:39:57 UTC 2015 (duration 39m 56s)
* 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-02 02:48:23+00:00
* 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 45s)
* 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-02 02:27:42+00:00
* 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 26s)
* 02:06 logmsgbot: krinkle Synchronized php-1.26wmf7/resources/src/mediawiki/mediawiki.js: backport rl-fix I717b86573 (duration: 00m 14s)
* 00:33 ejegg: updated payments-wiki from a4fef65ec1dd3db1fb1d7ceb797b2c7485c722d2 to d22e44e3fab2b937707c2776384cb93a49b4cfd3
* 00:07 ori: Updated jobrunner for I1d351d8d1: Made periodictasks stats calls more useful
* 00:02 logmsgbot: ori Synchronized php-1.26wmf8/extensions/RSS/RSSParser.php: Ice44740fb: Don't rely on strip marker uniqueness (T10104) (duration: 00m 14s)
* 00:01 logmsgbot: ori Synchronized php-1.26wmf7/extensions/RSS/RSSParser.php: Ice44740fb: Don't rely on strip marker uniqueness (T10104) (duration: 00m 13s)
 
== June 1 ==
* 23:36 mutante: restarted gitblit ..
* 23:15 ori: Deployed jobchron / jobrunner change Icab05090b and restarted jobchron / jobrunner on job queue runners.
* 22:51 ejegg: updated payments from 60c160110a20cf763b82677ff1501e9ce0c919bc to a4fef65ec1dd3db1fb1d7ceb797b2c7485c722d2
* 21:36 godog: doing some local testing on carbon for T100636 fwiw, thus puppet disabled
* 21:35 ejegg: update paymentswiki from aa66797553fbcfb63f7cf29abccc44d060b65db0 to 60c160110a20cf763b82677ff1501e9ce0c919bc
* 21:13 logmsgbot: ori Synchronized php-1.26wmf7/languages/LanguageConverter.php: 1d054ce6d3: Use a fixed marker prefix string in the Parser and MWTidy (duration: 00m 14s)
* 20:40 logmsgbot: ori Synchronized php-1.26wmf8/languages/LanguageConverter.php: 1d054ce6d3: Use a fixed marker prefix string in the Parser and MWTidy (duration: 00m 13s)
* 20:29 twentyafterfour: disabled several no-longer-existent repositories in phabricator which apparently have been deleted in gerrit
* 20:26 subbu: deployed parsoid sha 73445bfd
* 20:05 twentyafterfour: restarted apache2 and phd on iridium (phabricator)
* 19:52 MaxSem: Repopulated gis.spatial_ref_sys on labsdb1004 with postgis 2.1 data, old contents backed up as spatial_ref_sys_bak
* 18:55 logmsgbot: ori Synchronized php-1.26wmf7/extensions/SemanticForms/includes/SF_FormUtils.php: I7ed3996a1: Stop using StripState (duration: 00m 13s)
* 18:55 logmsgbot: ori Synchronized php-1.26wmf8/extensions/SemanticForms/includes/SF_FormUtils.php: I7ed3996a1: Stop using StripState (duration: 00m 15s)
* 17:46 yurik: deployed graphoid service update - grafana logging cleanup
* 16:40 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1003 (duration: 00m 15s)
* 16:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: T99491, T100925: Sysops to add users to import group on maiwiki, newiki (duration: 00m 14s)
* 15:47 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/CodeReview: SWAT: Backport CodeReview module position fix [[gerrit:215043]] (duration: 00m 13s)
* 15:24 logmsgbot: thcipriani Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoaderWikiModule.php: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 15s)
* 15:23 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/WikiEditor: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 13s)
* 15:22 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/VectorBeta: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 15s)
* 15:21 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/SyntaxHighlight_GeSHi: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 14s)
* 15:20 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 13s)
* 15:18 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Gather: SWAT: Make ResourceLoaderWikiModule support custom position [[gerrit:214741]] (duration: 00m 13s)
* 14:42 cmjohnson1: powering down analytics1028 to swap the bad DIMM
* 14:38 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 12s)
* 13:48 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on wikisource and itwiki, and make other projects sidebar feature default for ptwiki (for real) (duration: 00m 12s)
* 13:45 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on wikisource and itwiki, and make other projects sidebar feature default for ptwiki (duration: 00m 15s)
* 13:31 logmsgbot: aude Synchronized php-1.26wmf8/extensions/Wikidata: css compatibility fixes for wmf8 (duration: 00m 24s)
* 13:00 logmsgbot: krenair Synchronized php-1.26wmf8/extensions/WikimediaMessages/WikimediaMessages.hooks.php: https://gerrit.wikimedia.org/r/#/c/215011/ - fix EditPageCopyrightWarning (duration: 00m 16s)
* 12:22 moritzm: added firmware-nonfree 0.44~wmf1 for jessie-wikimedia on carbon
* 09:32 yurik: deployed latest graphoid service to sca100x
* 08:18 hashar: Jenkins: upgrading git plugin from 1.5.0 to latest
* 08:12 mobrovac: restbase restart cassandra on restbase1006
* 08:09 mobrovac: restbase restart cassandra on restbase1005
* 08:07 mobrovac: restbase restart cassandra on restbase1004
* 08:05 mobrovac: restbase restart cassandra on restbase1003
* 08:00 mobrovac: restbase restart cassandra on restbase1002
* 07:59 mobrovac: restbase restart cassandra on restbase1001
* 05:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun  1 05:18:18 UTC 2015 (duration 18m 17s)
* 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-01 02:46:32+00:00
* 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 37s)
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-01 02:26:03+00:00
* 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 35s)
 
== May 31 ==
* 22:35 jgage: graphite2001 keeps falling off the net due to OOM; swap 100% in use. dist-upgraded & rebooted. dmesg in ~gage/dmesg.2015-05-31
* 18:37 logmsgbot: krinkle Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: rl live fix - I717b86573 (duration: 00m 12s)
* 17:36 Krinkle: Confirmed RL problem solved. The jquery|mediawiki&version=bizqqnC request was cached with an old mw.loader implementation somehow. After the touch and sync, the version is now dQAzAsdU and the implementation is up to date.
* 17:33 logmsgbot: krinkle Synchronized php-1.26wmf7/resources: touch mediawiki.js (duration: 00m 13s)
* 17:20 Krinkle: Investigating RL issues (clients are loading mediawiki.notification&version=19700101T000000Z, mw.loader.moduleRegistry contains NaN for versions)
* 17:12 gwicke: performed a rolling restart of RESTBase Cassandra nodes to address elevated request error rates apparently related to schema disagreement
* 05:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 31 05:34:36 UTC 2015 (duration 34m 35s)
* 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-31 02:46:41+00:00
* 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 51s)
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-31 02:25:44+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 41s)
 
== May 30 ==
* 21:07 bd808: Upgraded Elasticsearch cluster to 1.3.9 on logstash100[1-6]
* 18:35 logmsgbot: hoo Synchronized php-1.26wmf7/extensions/UploadWizard/: Touch js… (duration: 00m 18s)
* 17:06 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/WikiEditor/extension.json: Explicitly define module position (duration: 00m 13s)
* 05:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 30 05:31:02 UTC 2015 (duration 31m 1s)
* 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-30 02:55:22+00:00
* 02:52 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 40s)
* 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-30 02:34:55+00:00
* 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 50s)
* 01:15 ori: Deployed rcstream I797bc1244: Handle invalid JSON gracefully
* 00:08 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/212436/ - docs only, no code change (how was this waiting 10 days?) (duration: 00m 14s)
 
== May 29 ==
* 23:56 logmsgbot: ori Synchronized w/static/images/project-logos: Ic62747f37: Optimise project logos added since I8c9a6a56 (duration: 00m 13s)
* 21:21 logmsgbot: ori Synchronized wmf-config/throttle.php: Ife45684c5: Add another IP address for Santiago edit-a-thon (duration: 00m 13s)
* 20:43 logmsgbot: ori Synchronized robots.txt: I7b321b62d: allow robots to use RL on domains (duration: 00m 14s)
* 17:18 mutante: fix client_max_body_size syntax error in nginx config of payments1001
* 15:19 logmsgbot: anomie Synchronized php-1.26wmf8/extensions/ConfirmEdit/: Update ConfirmEdit to fix API breakage [[gerrit:214620]] (duration: 00m 14s)
* 14:52 paravoid: re-redirecting ns0 traffic back to rubidium
* 14:17 jynus: Moving pdns and designate databases from m1 to m5
* 13:30 logmsgbot: aude Synchronized php-1.26wmf8/extensions/Wikidata: touch js and css files to try to fix issues on test.wikidata (duration: 00m 26s)
* 13:17 godog: roll-restart cassandra on cerium / xenon / praseodymium following java upgrade
* 11:53 paravoid: reimaging rubidium
* 11:45 _joe_: restart nutcracker on mw1150
* 11:41 paravoid: redirecting ns0 traffic to baham (= ns1) in preparation for rubidium upgrade
* 06:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 29 06:51:45 UTC 2015 (duration 51m 44s)
* 06:13 logmsgbot: ori Synchronized php-1.26wmf7/includes/deferred/SiteStatsUpdate.php: Icc12c07ab: Update context stats in SiteStatsUpdate (duration: 00m 13s)
* 06:12 logmsgbot: ori Synchronized php-1.26wmf8/includes/deferred/SiteStatsUpdate.php: Icc12c07ab: Update context stats in SiteStatsUpdate (duration: 00m 14s)
* 06:03 apergos: salt keys regenerated on all production hosts (minions, not master key)
* 03:09 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-29 03:08:15+00:00
* 03:02 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 10m 08s)
* 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-29 02:35:10+00:00
* 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 54s)
* 00:07 logmsgbot: ori Synchronized php-1.26wmf7/includes/diff/UnifiedDiffFormatter.php: d95cac90c7: Make the output of UnifiedDiffFormatter match diff -u (duration: 00m 14s)
* 00:06 logmsgbot: ori Synchronized php-1.26wmf7/extensions/Echo/includes/DiffParser.php: 41d27c4a26: Update Echo for cherry-picks (duration: 00m 13s)
 
== May 28 ==
* 23:33 jgage: restarted nutcracker on mw1056 due to errors, per bd808
* 23:18 logmsgbot: catrope Synchronized php-1.26wmf7/includes/EditPage.php: Fix regression with URL-specified edit tags (duration: 00m 13s)
* 23:18 logmsgbot: catrope Synchronized php-1.26wmf6/includes/EditPage.php: Fix regression with URL-specified edit tags (duration: 00m 13s)
* 23:04 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable A/B test of VE for new accounts on enwiki (duration: 00m 13s)
* 22:48 logmsgbot: hoo Synchronized php-1.26wmf7/: Touching some JS, re-syncing resource definitions to rule out causes for Wikidata JS problem. (duration: 01m 00s)
* 21:52 logmsgbot: ori Synchronized php-1.26wmf7/resources/src/mediawiki/mediawiki.toc.js: Touching file on unconfirmed suspicion of stale cache (duration: 00m 16s)
* 21:51 logmsgbot: ori Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.toc.js: Touching file on unconfirmed suspicion of stale cache (duration: 00m 15s)
* 20:24 mutante: killed nodejs on wtp1023,wtp1016
* 20:11 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on Wikivoyage (duration: 00m 13s)
* 20:03 cscott: updated Parsoid to version 497da30e ; canary restart of wtp1001; observed network TX spike (possibly UDP, possibly logging); reverted to 8ed6fd0b and restarted all parsoids.
* 19:33 mutante: temp. stopped icinga-wm
* 19:05 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/Gadgets/: Explicitly define module position (duration: 00m 14s)
* 18:32 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/GlobalCssJs/: Explicitly define module position (duration: 00m 12s)
* 18:24 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/GlobalCssJs/: Explicitly define module position (duration: 00m 13s)
* 18:22 logmsgbot: krenair Synchronized php-1.26wmf6/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214397/ - in case we have to go back to wmf6 again for whatever reason (duration: 00m 15s)
* 18:20 logmsgbot: krenair Synchronized php-1.26wmf8/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214396/ (duration: 00m 13s)
* 18:17 logmsgbot: krenair Synchronized php-1.26wmf7/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214395/ (duration: 00m 14s)
* 17:29 logmsgbot: twentyafterfour Finished scap: Group0 to 1.26wmf8, everything else to 1.26wmf7 (duration: 28m 16s)
* 17:01 logmsgbot: twentyafterfour Started scap: Group0 to 1.26wmf8, everything else to 1.26wmf7
* 16:59 paravoid: reimaging baham
* 16:52 paravoid: redirecting ns1 traffic to rubidium (= ns0) in preparation for baham upgrade
* 15:54 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 03m 19s)
* 15:50 logmsgbot: kartik Started scap: Update ContentTranslation
* 15:47 logmsgbot: thcipriani Synchronized wmf-config/abusefilter.php: SWAT: Modify AbuseFilter block configuration on eswikibooks [[gerrit:206510]] (duration: 00m 15s)
* 15:40 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Prevent indexing of User: namespace on ukwiki [[gerrit:210680]] (duration: 00m 14s)
* 15:35 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on sa.wikipedia [[gerrit:212724]] (duration: 00m 13s)
* 15:28 godog: set operations/debs/python-statsd as hidden in gerrit -- deprecated
* 15:24 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT:  Enable Extension:NewUserMessage on ta.wikipedia [[gerrit:213841]] (duration: 00m 12s)
* 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink for cswiki [[gerrit:214247]] (duration: 00m 15s)
* 15:11 godog: set operations/debs/txstatsd as hidden in gerrit -- deprecated
* 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for CX deployment on 20150528 [[gerrit:213992]] (duration: 00m 15s)
* 15:00 bblack: merged up https://gerrit.wikimedia.org/r/214345 - look here if IPv6 problems!
* 14:37 cmjohnson1: powering down dataset1001 to add disk array
* 14:17 bblack: deploying https://gerrit.wikimedia.org/r/214341 - keep in mind if ipv6-related issues arise!
* 13:50 akosiaris: started ircecho (icinga-wm) on neon
* 13:46 hashar: upgrading Jenkins git plugin from 1.4.6+wmf1 to 1.7.1 {{bug|T100655}}  and restarting Jenkins
* 13:25 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1003 (not to confuse with db1003) after warmup (duration: 00m 15s)
* 13:11 akosiaris: killed ircecho service on neon
* 09:48 _joe_: depooling the HHVM appserver. 503s reduced slightly but still non-irrelevant
* 09:37 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool pc1003 (duration: 00m 15s)
* 09:35 _joe_: pooling mw1152 into the imagescalers pool after fixes made in Lyon
* 06:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 28 06:09:56 UTC 2015 (duration 9m 55s)
* 04:22 springle: reload dbstore1002 s7
* 02:41 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-28 02:40:00+00:00
* 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 46s)
* 02:20 springle: set global read_only=0 on pc1001 pc1002. this config broke in the recent upgrade
* 00:59 logmsgbot: legoktm Synchronized php-1.26wmf8/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 17s)
* 00:58 logmsgbot: legoktm Synchronized php-1.26wmf7/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 13s)
* 00:07 logmsgbot: twentyafterfour Synchronized rpc/RunJobs.php: deploy I98b8a4ddbcdd58d1f2f23e4b1bf154f10b6b279e (duration: 00m 17s)
 
== May 27 ==
* 23:46 awight: updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0
* 23:31 logmsgbot: twentyafterfour Finished scap: scap, now with 10% less fail (duration: 22m 07s)
* 23:26 awight: payments rolled back to 858b87319daa3d66f62eb32e08cefc6b061748d1
* 23:24 awight: updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0
* 23:09 logmsgbot: twentyafterfour Started scap: scap, now with 10% less fail
* 22:57 logmsgbot: ori rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 21:49 mutante: restarted hhvm on mw1250,mw1254,mw1256
* 21:47 mutante: restarted hhvm on mw1017,mw1243,mw1244
* 21:42 bblack: restarting hhvm everywhere on 30s intervals between hosts
* 21:10 logmsgbot: twentyafterfour Synchronized php-1.26wmf8: Fix ConfirmEdit fatal Change-Id: I22353669a85391c3d9760a5253cac1263e895cf9 (duration: 01m 08s)
* 20:46 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf6
* 20:45 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf8
* 20:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf7
* 20:36 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf8 and rebuild l10n cache (duration: 67m 53s)
* 19:40 akosiaris: removed operations/puppet/varnish from gerrit, git.wikimedia.org and github. The repo was used as a git submodule but the workflow turned out to be cumbersome approximately a year ago and was no longer updated. Up to a few minutes ago, it only served as a source of confusion. It no longer does.
* 19:28 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache
* 19:22 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_1863397713" --threads=4 --lang en  --quiet' returned non-zero exit status 255 (duration: 03m 38s)
* 19:18 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache
* 18:12 moritzm: Uploaded gridengine_6.2u5-4+wmf2 for precise-wikimedia to apt.wikimedia.org
* 17:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1002 (duration: 00m 13s)
* 17:42 paravoid: rebooting asw-d2-eqiad
* 17:41 ottomata: initiating controlled shutdown of kafka broker analytics1018 in anticipation of switch reboot
* 15:33 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1002 (duration: 00m 13s)
* 15:02 cmjohnson1: powering down cp1069 to relocate within the same rack
* 14:47 cmjohnson1: powering down cp1070 to relocate within the same rack
* 13:30 hashar: All Jenkins slaves are disconnected due to some ssh error. CI is down.
* 13:27 hashar: restarting Jenkins for java upgrade
* 13:13 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 13s)
* 11:16 akosiaris: rebooting ganeti100{1..4} for bridge networking configuration
* 09:59 paravoid: powercycling ms-be1001; dead, console unresponsive
* 06:35 springle: clone dbstore2001 data to dbstore2002
* 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 27 05:47:25 UTC 2015 (duration 47m 24s)
* 02:53 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-27 02:52:25+00:00
* 02:48 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 52s)
* 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-27 02:28:34+00:00
* 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 45s)
 
== May 26 ==
* 18:21 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf7
* 17:13 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 15s)
* 17:10 logmsgbot: krenair Synchronized multiversion/MWMultiVersion.php: open cnwikimedia (duration: 00m 13s)
* 16:27 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
* 16:12 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: add cnwikimedia
* 16:08 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s)
* 16:07 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 15s)
* 16:07 logmsgbot: krenair Synchronized w/static/images/project-logos/cnwikimedia.png: (no message) (duration: 00m 19s)
* 15:52 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 14s)
* 15:32 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (warm period) (duration: 00m 13s)
* 15:24 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/213652/ (duration: 00m 15s)
* 15:23 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/213257/ (duration: 00m 14s)
* 14:54 bblack: restarted ganglia-monitor on all cp* (many were obviously-broken, probably most recently from bad startup after the reboots last week)
* 14:14 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 12s)
* 08:24 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 13s)
* 05:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 26 05:52:50 UTC 2015 (duration 52m 49s)
* 03:02 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-26 03:01:12+00:00
* 02:55 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 09m 31s)
* 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-26 02:28:08+00:00
* 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 44s)
* 01:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1026, warm up (duration: 00m 14s)
 
== May 25 ==
* 16:36 jynus: running diagnostics on mariadb@pc1001: a very small amount of requests may experience extra latency
* 14:17 duh: intentionally not scapping right now, will let l10nupdate sync it out
* 14:16 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/WikimediaMessages/i18n/: ExtensionDistributor message updates (duration: 00m 17s)
* 13:53 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/ExtensionDistributor: Update ExtensionDistributor to master (duration: 00m 13s)
* 13:38 logmsgbot: jynus Synchronized wmf-config/InitialiseSettings-labs.php: restbase change from yurik (duration: 00m 14s)
* 13:37 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1018 (warm cache) (duration: 00m 13s)
* 13:09 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1018 (duration: 00m 14s)
* 10:31 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 13s)
* 08:36 YuviKTM: running du -d 1 -h > du-may-25-2015 on /exp/project/tools on labstore1001 to audit tools' NFS usage
* 05:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 25 05:11:47 UTC 2015 (duration 11m 46s)
* 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-25 02:49:45+00:00
* 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 32s)
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-25 02:26:39+00:00
* 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 36s)
 
== May 24 ==
* 17:18 springle: stop mysqld db1002 db1003 db1004 db1005 db1006 db1007
* 10:00 ^d: gerrit: manually gc'd all repos to help with clone times
* 08:55 godog: resize existing whisper files with new retention on graphite2001
* 05:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 24 05:41:35 UTC 2015 (duration 41m 34s)
* 02:58 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-24 02:57:17+00:00
* 02:53 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 57s)
* 02:34 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-24 02:33:23+00:00
* 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 34s)
 
== May 23 ==
* 23:30 logmsgbot: ori Synchronized php-1.26wmf7/extensions/Gadgets: b592efa5fe: Update Gadgets for I6da3eede0: Conversion to using WAN cache (duration: 00m 13s)
* 12:54 godog: remove MediaWiki.xhprof to pick up new retention schema
* 12:53 godog: bounce carbon on graphite1001 to pick up new retention schema
* 11:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ic258d01a7: Revert "Change StatsD port to another value temporarily" (duration: 00m 13s)
* 10:22 ori: Metrics from MediaWiki to graphite are temporarily suspended while xhprof profiling work is ongoing.
* 10:21 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Exclude xhprof.run_init from being reported (duration: 00m 13s)
* 10:03 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 13s)
* 09:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Ia7549d45: Re-enable xhprof profiling (duration: 00m 14s)
* 09:52 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I311c989e9: Change StatsD port to another value temporarily (duration: 00m 14s)
* 05:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 23 05:12:44 UTC 2015 (duration 12m 43s)
* 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-23 02:44:48+00:00
* 02:41 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 05m 56s)
* 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-23 02:23:36+00:00
* 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 02s)
* 00:33 mutante: adding cwdent to WMF LDAP group per https://www.mediawiki.org/wiki/User:CDentinger_%28WMF%29
* 00:04 logmsgbot: ori Synchronized php-1.26wmf6/includes: 9bf0236c20, 2d3c9233ed (duration: 00m 17s)
 
== May 22 ==
* 20:59 logmsgbot: ori Synchronized php-1.26wmf7/includes: 4632aff034 (duration: 00m 18s)
* 19:19 logmsgbot: ori Synchronized php-1.26wmf6/includes/profiler: 0d9c4dd8fe, ec22d6e6c3, 4127b1a315: Profiler improvements (duration: 00m 16s)
* 19:18 logmsgbot: ori Synchronized php-1.26wmf7/includes/profiler: a69ee4a0f7, a3773b4d8b, ab19be9d99: Profiler improvements (duration: 00m 15s)
* 17:16 yuvipanda: rebooted labvirt1005 from mgmt see what's up with disk array
* 16:53 yuvipanda: rebooted labvirt1005 for T99738
* 15:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/211696/ - disable VE A/B test (duration: 00m 12s)
* 13:57 jynus: schema change on x1 shard https://phabricator.wikimedia.org/T94427 No downtime expected
* 10:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1036 (duration: 00m 12s)
* 07:58 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1036 (duration: 00m 13s)
* 06:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 22 06:47:25 UTC 2015 (duration 47m 23s)
* 05:50 springle: upgrade db1026 trusty mariadb 10, mydumper reload
* 03:09 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-22 03:08:51+00:00
* 03:02 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 10m 14s)
* 02:43 logmsgbot: hoo Synchronized php-1.26wmf6/extensions/Wikidata/: Update Wikidata: Make wbmergeitems respect the bot parameter (duration: 00m 19s)
* 02:38 logmsgbot: hoo Synchronized php-1.26wmf7/extensions/Wikidata/: Update Wikidata from wmf4 to wmf6 branch. (duration: 00m 22s)
* 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-22 02:35:33+00:00
* 02:32 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 56s)
 
== May 21 ==
* 23:50 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Re-enable subpages for the template namespace on officewiki (duration: 00m 13s)
* 23:35 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on hif.wikipedia (duration: 00m 14s)
* 23:30 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Configure import sources for hif.wikipedia (duration: 00m 12s)
* 23:26 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Site name configuration on ast.wiktionary (duration: 00m 12s)
* 23:08 logmsgbot: ori Synchronized php-1.26wmf6/includes: 7238213e6d: Defer some updates in doEditUpdates() (duration: 00m 16s)
* 23:07 logmsgbot: ori Synchronized php-1.26wmf7/includes: da79b19b88: Defer some updates in doEditUpdates() (duration: 00m 16s)
* 17:01 mutante: mw1123: apt-get autoclean, rebooting for kernel upgrade
* 16:57 mutante: dist-upgrade on mw1123
* 16:34 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 23m 25s)
* 16:10 logmsgbot: kartik Started scap: Update ContentTranslation
* 16:04 mutante: armed keyholder on mira
* 15:56 kart_: Updated cxserver
* 15:32 Tim: removed max-registration properties from 2015 board elections on metawiki and votewiki per my comment on T97924
* 15:09 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/212281/ (duration: 00m 10s)
* 15:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/211116/ (duration: 00m 16s)
* 15:00 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/205778/ - enable VE A/B test (duration: 00m 14s)
* 14:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/205778/ - VE A/B test on enwiki (duration: 00m 11s)
* 14:37 bblack: enabling puppet on caches for varnish retries changes...
* 11:51 logmsgbot: twentyafterfour Finished scap: 1.26wmf7 symlinks (duration: 05m 16s)
* 11:49 twentyafterfour: I'm investigating some inconsistencies in symlinks in /srv/mediawiki, ref https://phabricator.wikimedia.org/T99886
* 11:46 logmsgbot: twentyafterfour Started scap: 1.26wmf7 symlinks
* 11:31 paravoid: troubleshooting analytics1036, includes reboots
* 07:49 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia distribution jessie-wikimedia: php-luasandbox_2.0.9
* 07:21 _joe_: cleaning the bytecode cache database everywhere
* 06:43 _joe_: cleaning up the bytecode caches of a few appservers
* 06:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 21 06:27:09 UTC 2015 (duration 27m 8s)
* 04:55 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia5239c1e: Unset $wgDiff, so we stop shelling out to diff (duration: 00m 12s)
* 03:10 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-21 03:09:49+00:00
* 03:06 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 13s)
* 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-21 02:44:18+00:00
* 02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 09m 36s)
* 00:38 logmsgbot: ori Synchronized php-1.26wmf7/includes/MediaWiki.php: adacd7b35c: Pass a message key to MalformedTitleException constructor (duration: 00m 11s)
* 00:37 logmsgbot: ori Synchronized php-1.26wmf6/includes/MediaWiki.php: b13721b5cb: Pass a message key to MalformedTitleException constructor (duration: 00m 12s)
* 00:20 logmsgbot: ori Synchronized php-1.26wmf6/includes/jobqueue/JobQueueGroup.php: 1e43c05283: Revert "Undefer push() in lazyPush() temporarily" (duration: 00m 12s)
 
== May 20 ==
* 23:07 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/SyntaxHighlight_GeSHi/: https://gerrit.wikimedia.org/r/212456 (duration: 00m 14s)
* 23:05 logmsgbot: legoktm Synchronized wmf-config/: Disable WikiGrok in WMF production (duration: 00m 13s)
* 22:14 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf5
* 21:51 logmsgbot: ori Synchronized php-1.26wmf6/includes: I32a3cfabc: Made pushLazyJobs() handle all queue groups (duration: 00m 18s)
* 21:25 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/SyntaxHighlight_GeSHi: https://gerrit.wikimedia.org/r/#/c/212450/ (duration: 00m 13s)
* 21:18 logmsgbot: twentyafterfour Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
* 21:01 cscott: updated OCG to version ca4f64852de5b1de782b292b50038fbd2dd84266
* 20:59 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf7
* 20:58 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf6
* 20:50 logmsgbot: twentyafterfour Finished scap: retry: testwiki to php-1.26wmf7 and rebuild l10n cache (duration: 26m 02s)
* 20:42 ebernhardson: restarted gmond on elastic10{01..31}.eqiad.wmnet
* 20:24 logmsgbot: twentyafterfour Started scap: retry: testwiki to php-1.26wmf7 and rebuild l10n cache
* 20:12 subbu: deployed parsoid version 8ed6fd0b
* 19:35 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3448528422" --threads=4 --lang en  --quiet' returned non-zero exit status 255 (duration: 03m 22s)
* 19:32 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf7 and rebuild l10n cache
* 17:41 bblack: esams+eqiad upload varnish caches will be downtimed+rebooted today, experimenting with depool effects as well (next several hours)
* 16:03 logmsgbot: manybubbles Synchronized php-1.26wmf5/extensions/Flow/: SWAT update flow for wmf5 to fix two issues (duration: 00m 14s)
* 15:54 godog: rolling restart restbase on restbase1003-1006
* 15:52 mobrovac: restbase restarted on restbase1002
* 15:47 godog: restbase restarted on restbase1001
* 15:35 logmsgbot: manybubbles Synchronized php-1.26wmf6/extensions/Flow/: SWAT update flow for wmf6 to fix two issues (duration: 00m 12s)
* 15:22 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT new namespaces for ptwikinews (duration: 00m 11s)
* 15:18 logmsgbot: manybubbles Synchronized wmf-config/throttle.php: SWAT clean old throttle rule and add a new one for an upcoming festival (duration: 00m 13s)
* 15:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT update urwikiquote logo 2/2 (duration: 00m 11s)
* 15:13 logmsgbot: manybubbles Synchronized w/static/images/project-logos/urwikiquote.png: SWAT update urwikiquote logo 1/2 (duration: 00m 13s)
* 15:06 springle: db1045 pt-osc reindexing (should be low load, ~2hr)
* 14:36 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on itwiki and wikiquote (duration: 00m 16s)
* 14:25 milimetric: Deployed Event Logging Server with better batch insertion on Monday, May 18 (apologies for late notice)
* 13:13 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1045; depool db1026 (duration: 00m 13s)
* 10:18 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 11s)
* 09:43 _joe_: stopping puppet, fiddling with HHVM parameters on mw1114
* 09:37 Coren: tools kicked grrrit-wm in the diodes.
* 09:35 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 12s)
* 06:45 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 for maintenance (duration: 00m 11s)
* 06:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 20 06:42:22 UTC 2015 (duration 42m 21s)
* 03:13 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-20 03:12:31+00:00
* 03:06 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 09m 40s)
* 02:41 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-20 02:40:07+00:00
* 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 30s)
* 01:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1045 (duration: 00m 15s)
* 00:43 logmsgbot: ebernhardson Synchronized wmf-config/: Per-user poolcounter triggered many more times than expected (duration: 00m 15s)
* 00:42 logmsgbot: ebernhardson Synchronized wmf-config/PoolCounterSettings-common.php: Enable per-user poolcounter in CirrusSearch on all wikis (duration: 00m 14s)
* 00:41 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Enable per-user poolcounter in CirrusSearch on all wikis (duration: 00m 12s)
* 00:40 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/NavigationTiming/: Update NavigationTiming for cherry-picks in 1.26wmf5 (duration: 00m 12s)
* 00:39 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/NavigationTiming/: Update NavigationTiming for cherry-picks in 1.26wmf6 (duration: 00m 12s)
* 00:36 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf5 for poolcounter error message updates (duration: 00m 11s)
* 00:35 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf6 for poolcounter error message updates (duration: 00m 13s)
* 00:34 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf5 for poolcounter error message updates (duration: 00m 12s)
* 00:32 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf6 for poolcounter error message updates (duration: 00m 12s)
 
== May 19 ==
* 23:35 gwicke: deployed RESTBase 90817c2a
* 23:20 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: logstash: Exclude jobrunner debug messages (duration: 00m 12s)
* 23:10 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on maiwiki and pawiki (duration: 00m 12s)
* 22:06 ejegg: updated payment from e89d18ee20abcb1ca3c455e6a298bf8a6aa84442 to  858b87319daa3d66f62eb32e08cefc6b061748d1
* 21:16 logmsgbot: kaldari Synchronized php-1.26wmf6/extensions/MobileFrontend: syncing MobileFrontend for 1.26wmf6 (duration: 00m 11s)
* 21:15 logmsgbot: kaldari Synchronized php-1.26wmf6/extensions/Gather: syncing Gather for 1.26wmf6 (duration: 00m 12s)
* 21:07 robh: merging fixes to sodium, mailing list outage fixed
* 20:51 andrewbogott: rebooting/reimaging virt1005, virt1006, 1007
* 20:22 mutante: mailman: killed processes by user "list". started mailman
* 19:40 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia6a2cb7: Removed "refreshLinks" from $wgJobBackoffThrottling (duration: 00m 12s)
* 19:37 logmsgbot: anomie Finished scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time (duration: 44m 34s)
* 19:25 robh: mailman permission errors abound!  had to take it offline again and fixing
* 19:02 robh: mailman is back to routing mail normally (still testing rename parts)
* 18:53 logmsgbot: anomie Started scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time
* 18:51 logmsgbot: anomie Finished scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data (duration: 05m 39s)
* 18:46 logmsgbot: anomie Started scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data
* 18:38 yuvipanda: issuing start command for all hosts on labvirt1006, just to make sure
* 18:35 yuvipanda: labvirt1006 rebooting, long POST
* 18:31 yuvipanda: restarted labvirt1006
* 18:20 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.26wmf6
* 18:15 robh: stopping mailman again for further planned work T99098
* 17:43 robh: mailing lists still down, scrubbing list archives is painful and error prone
* 17:33 ottomata: starting reboots of analytics worker nodes in order to enable hyperthreading Bug: https://phabricator.wikimedia.org/T90640
* 17:04 robh: puppet stopped on sodium (dont need it restarting mailman while im working)
* 17:04 robh: starting mailman downtime window to scrub content off list archive per T99098
* 16:58 bblack: automated reboots of esams/eqiad non-upload caches starting (should auto-downtime, should be no real impact)...
* 15:51 logmsgbot: anomie Synchronized php-1.26wmf5/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch [[gerrit:211743]] (duration: 00m 12s)
* 15:50 logmsgbot: anomie Synchronized php-1.26wmf6/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch [[gerrit:211744]] (duration: 00m 10s)
* 15:31 logmsgbot: anomie Synchronized php-1.26wmf5/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" [[gerrit:211893]] (duration: 00m 12s)
* 15:28 logmsgbot: anomie Synchronized php-1.26wmf6/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" [[gerrit:211894]] (duration: 00m 13s)
* 15:16 logmsgbot: anomie Synchronized php-1.26wmf5/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211948] (duration: 00m 12s)
* 15:15 logmsgbot: anomie Synchronized php-1.26wmf6/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211947] (duration: 00m 12s)
* 14:43 jynus: back to read/write after virt1000 database migration - migration seems ok
* 14:41 godog: purge cassandra system CF metrics from graphite1001
* 14:29 jynus: temporarily going read-only for virt1000 for database migration
* 14:24 mobrovac: enabled puppet on restbase1001
* 14:19 mobrovac: restbase group1 wiki keyspaces created
* 14:15 mobrovac: starting manually RB with group1 wikis enabled on restbase1001
* 14:11 mobrovac: restbase100x: removed superfluous keyspaces by hand from Cassandra
* 13:47 bblack: done with cp40xx reboot process
* 13:32 bblack: rebooting ulsfo caches (cp40xx - currently depooled from all traffic + downtimed in icinga)
* 13:09 mobrovac: disabled puppet on restbase100x
* 12:51 godog: bounce hhvm on mw1152
* 08:26 _joe_: restarting a few HHVM instances with a full TC space
* 05:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 19 05:03:56 UTC 2015 (duration 3m 55s)
* 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-19 02:45:17+00:00
* 02:43 logmsgbot: krinkle Synchronized php-1.26wmf6/includes/resourceloader/ResourceLoader.php: Ic0df4fb5cff (duration: 00m 12s)
* 02:42 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 43s)
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-19 02:25:05+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 11s)
* 00:37 logmsgbot: ebernhardson Synchronized php-1.26wmf5/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s)
* 00:36 logmsgbot: ebernhardson Synchronized php-1.26wmf6/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s)
 
== May 18 ==
* 23:49 yuvipanda: restarted nutcracker on mw1053 and mw1107 for bd808
* 23:47 bd808: nutcracker needs restart on mw1053 and mw1107
* 23:37 yuvipanda: restarting hhvm on mw1123
* 23:36 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Revert "Removed "refreshLinks" from $wgJobBackoffThrottling" (duration: 00m 14s)
* 23:29 logmsgbot: ebernhardson Synchronized wmf-config/CommonSettings.php: removed refreshlinks from #wgJobBackoffThrottling (duration: 00m 14s)
* 23:21 hoo: Reverting my changes to the sites and site_identifiers tables from earlier on... apparently the export/importSites.php maintenance scripts don't work as advertised
* 23:03 logmsgbot: ori Synchronized php-1.26wmf6/extensions/Echo: 8609cb6b90: Update Echo for cherry-picks (duration: 00m 30s)
* 23:02 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Echo: 8c619b99a6: Update Echo for cherry-picks (duration: 00m 57s)
* 22:46 hoo: Updating the sites table on all wikis to reflect the language code change of bhwiki (from bh to bho). I have a backup of the old table from Wikidata in my home, should things go wrong.
* 20:38 mforns: upgraded and restarted EventLogging server: 19b5b7ae719321c4b8fb112890b574051b090571
* 20:12 subbu: deployed parsoid version 8ed3e503
* 19:42 yurik: restarted graphoid service to pick up the new config https://gerrit.wikimedia.org/r/#/c/211450/
* 19:35 ori: restarted statsv on hafnium
* 18:29 logmsgbot: ori Synchronized php-1.26wmf6/includes: 335f8a257d, e3b2255d9c (for UBN! T99468) (duration: 00m 28s)
* 18:28 logmsgbot: ori Synchronized php-1.26wmf5/includes: 335f8a257d, e3b2255d9c (for UBN! T99468) (duration: 01m 26s)
* 18:06 ori: restarted HHVM on mw1107 with libjemalloc heap profiling enabled
* 17:55 ori: Enabling heap profiling on mw11107 to troubleshoot T99525
* 17:08 andrewbogott: starting all instances on labvirt1001 (well, the ones that were running before)
* 16:59 andrewbogott_: dist-upgrading labvirt1001 since it’s down anyway and we may be due for kernel updates.
* 16:53 andrewbogott_: rebooting labvirt1001, and frowning a lot
* 15:59 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209286/ and https://gerrit.wikimedia.org/r/#/c/211407/ - should be no-ops (duration: 00m 20s)
* 15:36 logmsgbot: marktraceur Synchronized php-1.26wmf6/includes/: [SWAT] [wmf6] resourceloader: Don't cache minification of user.tokens (duration: 00m 19s)
* 15:24 logmsgbot: marktraceur Synchronized php-1.26wmf6/includes/Title.php: [SWAT] [wmf6] Log callers that trigger Title::newFromText $text type warning (duration: 00m 46s)
* 15:23 logmsgbot: marktraceur Synchronized php-1.26wmf5/includes/Title.php: [SWAT] [wmf5] Log callers that trigger Title::newFromText $text type warning (duration: 00m 15s)
* 15:07 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add wikis for deployment on 2015-05-18 (duration: 00m 29s)
* 14:35 andrewbogott: disabling puppet on labnet1001 to debug dnsmasq
* 14:07 _joe_: restarting HHVM on mw1107 - memory leak probably happening
* 13:38 logmsgbot: aude Synchronized wmf-config/InitialiseSettings-labs.php: Remove beta-specific Graph settings (duration: 01m 46s)
* 13:34 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on enwikivoyage, fawiki, and hewiki, and graph extension everywhere (duration: 00m 57s)
* 13:31 logmsgbot: aude Synchronized php-1.26wmf6/extensions/Wikidata: Fix rdf dump script (duration: 03m 23s)
* 13:27 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 after warmup period (duration: 01m 01s)
* 13:01 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 17s)
* 11:13 yurik: deployed graphoid update to fix https://phabricator.wikimedia.org/T99349
* 11:10 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 01m 00s)
* 11:07 jynus: depooling db1063 from cluster for maintenance
* 09:02 godog: loss on ulsfo-eqiad, depooled ulsfo
* 05:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 18 05:17:50 UTC 2015 (duration 17m 49s)
* 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-18 02:45:52+00:00
* 02:42 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 35s)
* 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-18 02:25:54+00:00
* 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 24s)
 
== May 17 ==
* 05:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 17 05:05:16 UTC 2015 (duration 5m 15s)
* 02:44 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-17 02:43:13+00:00
* 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 18s)
* 02:25 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-17 02:24:09+00:00
* 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 10s)
 
== May 16 ==
* 13:27 manybubbles: that was the last server in the elasticsearch rolling restart. all done. now we have new versions of the plugins. Lets try not to do that again.
* 13:25 manybubbles: es-tool restart-fast on elastic1031
* 09:15 godog: bounce hhvm on mw1196
* 09:10 godog: bounce hhvm on mw1141
* 07:49 godog: restart hhvm on mw1234, still pushing xhprof metrics
* 06:03 _joe_: killed nrpe on labvirt1003 - see T99341
* 05:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 16 05:01:02 UTC 2015 (duration 1m 1s)
* 04:11 andrewbogott: restarting sshd and generally poking around on labvirt1003
* 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-16 02:46:08+00:00
* 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 55s)
* 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-16 02:28:37+00:00
* 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 55s)
 
== May 15 ==
* 22:35 ejegg: updated crm from 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18 to 7ffe0cefb019828a09c9369187f14518847b5f41
* 19:44 manybubbles: elastic1027 es-tool restart-fast
* 19:37 awight: update crm from 2a2336655737a2cd1d3cc24624d1e8475e4cf039 to 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18
* 18:29 manybubbles: elastic1026 es-tool restart-fast
* 18:28 godog: bounce hhvm on mw1118
* 17:55 jynus: migrating of db service from virt1000 to m5-master aborted, service continues on virt1000
* 17:44 manybubbles: rolling restart almost done on elastic1025 - 1026 is next!
* 17:33 andrewbogott: updating qemu binaries on labvirt1001
* 17:29 godog: clean up remaining xhprof metrics from graphite1001
* 17:19 godog: bounce hhvm on mw1017
* 17:07 godog: still seeing metrics from xhprof creating, looking for source
* 16:29 godog: bounce carbon on graphite1001
* 16:23 manybubbles: elastic1023 and elastic1024 (skipped one log) es-tool restart-fast
* 16:16 godog: bounce statsdlb on graphite1001
* 14:49 jynus: migrating mariadb service from virt1000 to m5-master
* 14:37 manybubbles: elastic1021 es-tool restart-fast
* 14:26 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1053 in s1, warm up (duration: 00m 13s)
* 12:21 manybubbles: elastic1020 es-tool restart-fast
* 10:19 godog: bounce statsite and uwsgi on graphite1001
* 09:29 godog: restart carbon on graphite1001
* 09:15 godog: restart hhvm on mw1018, straggling
* 09:07 godog: rm MediaWiki.run_init from graphite1001 / graphite2001
* 09:04 ori: restarted hhvm / jobrunner on jobrunners to force them to pick up I6a516a0da ; re-cleared /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001
* 08:49 kart_: Updated cxserver to 1cb6cec
* 08:21 jynus: reenabling icinga check for MySQL on db1009
* 08:15 logmsgbot: oblivian Synchronized wmf-config/StartProfiler.php: Null-sync to touch the file (duration: 00m 12s)
* 07:20 ori: rm -rf /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001, as follow-up cleanup for I6a516a0da
* 07:14 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I6a516a0da: Don't send profiling data to graphite for now (duration: 00m 11s)
* 06:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 15 06:22:19 UTC 2015 (duration 22m 18s)
* 05:38 jynus: temporarily opening mysql port on firewall from db1009 to virt1000
* 04:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1018, warm up (duration: 00m 11s)
* 02:58 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-15 02:56:59+00:00
* 02:55 springle: xtrabackup clone db1057 to db1053
* 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 37s)
* 02:42 springle: upgrade db1053 trusty
* 02:34 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-15 02:33:18+00:00
* 02:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1019; depool db1053 (duration: 00m 13s)
* 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 39s)
* 02:12 manybubbles|away: elastic1019 es-tool restart-fast
* 01:12 manybubbles|away: elastic1018 es-tool restart-fast
* 00:07 manybubbles|away: elastic1017 es-tool restart-fast
 
== May 14 ==
* 23:35 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
* 23:20 ori: Depooled mw1169; HHVM deadlock à la T89912. Leaving it depooled to investigate.
* 23:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
* 23:05 logmsgbot: demon Synchronized w/static/images/project-logos/urwikiquote.png: (no message) (duration: 00m 14s)
* 23:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 17s)
* 22:26 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Icbf826a7: 1:1000 request profiling via xhprof (duration: 00m 12s)
* 22:23 gwicke: deployed RESTBase v0.6.3 (fd942ac38ad)
* 22:20 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 15s)
* 21:39 manybubbles: I'm going to be done doing rolling restarts for a couple of hours. If someone wants to pick them up and do the next one after the cluster goes green again then be my guest.
* 21:35 manybubbles: es-tool restart-fast on elastic1016
* 21:27 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s)
* 21:27 logmsgbot: ori Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s)
* 21:14 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 13s)
* 21:14 logmsgbot: ori Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 15s)
* 21:11 manybubbles: elastic1015 es-tool restart-fast
* 19:43 robh: mass unsubcription in listadmins list, resulting in unsupressed mass unsubscribe notices to all listadmin email address (sorry about the emails!)
* 19:24 logmsgbot: legoktm Synchronized php-1.26wmf5/skins/Nostalgia/skin.json: touch (duration: 00m 17s)
* 19:15 legoktm: debugging on tin / mw1017 for nostalgiawiki issue
* 16:59 ^d: elasticsearch: set transient cluster.routing.allocation.node_concurrent_recoveries on prod cluster to 8 (default: 2) to speed up recoveries.
* 16:52 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 44m 07s)
* 16:28 andrewbogott: disabling puppet on labnet1001 for testing
* 16:13 godog: es-tool restart-fast on elastic1014
* 16:08 logmsgbot: kartik Started scap: Update ContentTranslation
* 15:46 logmsgbot: thcipriani Synchronized php-1.26wmf5/extensions/Translate: SWAT update translate to a6f0a63 [[gerrit:210919]] (duration: 00m 15s)
* 15:12 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT enable new article campaign except bawiki [[gerrit:210916]] (duration: 00m 12s)
* 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Open external links on votewiki in new tab [[gerrit:210849]] (duration: 00m 12s)
* 15:00 godog: es-tool restart-fast on elastic1013
* 14:48 logmsgbot: andyrussg Synchronized php-1.26wmf6/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
* 14:34 paravoid: reimaging multatuli
* 14:34 jynus: migrating data db from virt1000 to db1009
* 14:23 bblack: restarted ganglia-monitor on eeden
* 14:21 logmsgbot: andyrussg Synchronized php-1.26wmf5/extensions/CentralNotice/: Update CentralNotice (duration: 00m 12s)
* 14:16 godog: es-tool restart-fast on elastic1012
* 14:12 paravoid: switching ns2 back to eeden
* 13:56 cmjohnson1: upgrading tellurium to trusty
* 13:41 cmjohnson1: power cycling barium
* 13:40 godog: es-root restart-fast on elastic1011
* 13:21 paravoid: reimaging eeden with jessie
* 12:59 paravoid: switching ns2 to multatuli
* 12:53 jynus: disabling temporarily Ichinga check for MySQL running on db1009 until data is migrated from virt1000 and host sent to production
* 12:40 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r60358-1
* 12:36 godog: es-tool restart-fast on elastic1010
* 11:40 manybubbles: restarting elasticsearch on elastic1009
* 05:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 14 05:06:09 UTC 2015 (duration 6m 8s)
* 02:55 manybubbles: restarting elasticsearch on elastic1008
* 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-14 02:49:53+00:00
* 02:47 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 16s)
* 02:44 springle: xtrabackup clone db1056 to db1019
* 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-14 02:28:02+00:00
* 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 51s)
* 01:48 manybubbles: sorry - restarting elasticsearch on elastic1007
* 01:47 manybubbles: restarting elastic1007
* 01:33 logmsgbot: springle Synchronized wmf-config/db-codfw.php: pool new codfw slaves (duration: 00m 11s)
* 01:28 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1060, warm up (duration: 00m 14s)
* 00:49 manybubbles: restarting elasticsearch on elastic1006
* 00:03 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/Gather/: SWAT Submodule bump for Gather extension (duration: 00m 12s)
 
== May 13 ==
* 23:52 awight: payments config: correct memcache location
* 23:40 logmsgbot: ebernhardson Synchronized wmf-config/CirrusSearch-common.php: SWAT deploy cirrus config change (duration: 00m 12s)
* 22:26 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf4
* 22:25 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group 0 to 1.26wmf6
* 22:21 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.26wmf5
* 22:17 twentyafterfour: restarted phd on iridium (phabricator) to sync the daemons' configuration
* 21:28 manybubbles: restarting elasticsearch on elastic1005
* 21:12 cscott: updated OCG to version c7c75e5b03ad9096571dc6dbfcb7022c924ccb4f
* 21:03 awight: updated payments from f97f8f99268974cfdb0182f178955bd627137842 to e89d18ee20abcb1ca3c455e6a298bf8a6aa84442
* 20:28 subbu: deployed parsoid version a8108fe6
* 20:15 manybubbles: restarted elasticsearch on elastic1004
* 20:12 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf6 and rebuild l10n cache (duration: 47m 24s)
* 20:11 manybubbles: cancel that - I just realized I can't do that.
* 20:10 manybubbles: elastic1003 restarted elasticsearch just fine. the cluster restart is going awesome. I'm going to rig the other 28 to restart via a script, one after the other. Expect nagios to complain about them some.
* 20:03 bblack: restarting hhvm on mw1190
* 19:25 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf6 and rebuild l10n cache
* 19:11 awight: paymens rolled back to f97f8f99268974cfdb0182f178955bd627137842
* 19:10 awight: payments updated from f97f8f99268974cfdb0182f178955bd627137842 to 5c326a521120a904a2012654e9287757dc5a8ca2
* 19:00 manybubbles: elastic1002 restart went well - starting elastic1003
* 18:45 awight: rolled back payments to f97f8f99268974cfdb0182f178955bd627137842
* 18:43 awight: update payments from f97f8f99268974cfdb0182f178955bd627137842 to 5c326a521120a904a2012654e9287757dc5a8ca2
* 18:05 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: undo all the nostalgia (duration: 00m 10s)
* 17:21 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: something something skins are broken (duration: 00m 11s)
* 17:14 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: because sometimes moving code helps (duration: 00m 15s)
* 17:10 manybub|lunch: elastic1002 restarted and rejoined the cluster - now the cluster is repaining. hurray.
* 17:08 manybub|lunch: elastic1001 restarted and rejoined the cluster hapilly while I was at lunch. it looks good - no errors beyond the ones we have fixes in flight for. So I'm going to do elastic1002
* 17:03 hashar: Zuul clone failures solved. Was due to network traffic being interrupted between labs and prod.
* 16:53 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209967/ (duration: 00m 14s)
* 16:51 hashar: Zuul clone failure https://phabricator.wikimedia.org/T98980
* 16:49 andrewbogott: re-enabling puppet on labnet1001
* 16:46 mutante: es2010 failed disk, reopening ticket for last fail in January
* 16:41 jynus: Enabling puppet agent in db1009.eqiad after reinstall
* 16:40 logmsgbot: ori Synchronized php-1.26wmf4/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 11s)
* 16:38 logmsgbot: ori Synchronized php-1.26wmf5/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 14s)
* 16:28 andrewbogott: disabling puppet on labnet1001 to tinker with nova config
* 15:44 mark: Disregard cr2-knams:xe-0/0/0; we're working on it
* 15:21 manybubbles: I think the elasticsearch cluster got stuck with alloation disabled after the rolling restart. Funky. Haven't seen that one before. Probably a problem with our instructions. Anyway, unstuck it and recovery is going faster now
* 15:17 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: didn't work, undoing previous sync (duration: 00m 12s)
* 15:15 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: trying something (duration: 00m 12s)
* 14:53 manybubbles: elasticsearch restart on elastic1001 going well. cluster still in recovering state as expect. I'll give it an hour to soak.
* 14:48 manybubbles: ok - time to start the rolling restart. I'm going to to elastic1001 first non-automated and watch it
* 14:36 manybubbles: s/gitfit/gitfat/ oh well
* 14:35 manybubbles: first attempt at syncing elasticsearch plugins didn't work 100%. syncing again. gitfit/gitdeploy is betraying me
* 14:32 manybubbles: syncing new versions of elsaticsearch plugins to prod. no restarts yet.
* 14:04 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking for Wikisource (duration: 00m 14s)
* 13:57 aude: added wbc_entity_usage table on all Wikibase Client wikis
* 13:56 jynus: jcrespo Disabling puppet agent in db1009.eqiad in preparation for reinstall
* 13:45 logmsgbot: aude Synchronized php-1.26wmf5/extensions/Wikidata: Update maintenance script (duration: 00m 20s)
* 12:45 springle: xtrabackup clone db1060 to db1018
* 12:39 springle: upgrade and restart db1060
* 09:20 jamesofur: inserting FDC election encryption key
* 06:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 13 06:19:59 UTC 2015 (duration 19m 58s)
* 05:53 springle: reinstall db1018
* 04:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 12s)
* 03:11 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-13 03:10:31+00:00
* 03:07 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 43s)
* 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-13 02:45:28+00:00
* 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 10m 08s)
* 01:56 damagecat: Started 'jobs' screen in tin to drain refreshLinks for enwiki using --nothrottle (T98621)
* 01:29 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Hardcode UploadWizard max upload size - T98933 (duration: 00m 12s)
* 01:23 logmsgbot: legoktm Synchronized php-1.26wmf5/extensions/GWToolset/:  Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s)
* 01:21 logmsgbot: legoktm Synchronized php-1.26wmf4/extensions/GWToolset/:  Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s)
* 01:07 gwicke: added commons to supported projects in RESTBase API
* 00:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I5ebedfdfb: Set $wgGadgetsCacheType to CACHE_ACCEL (duration: 00m 12s)
* 00:13 logmsgbot: ori Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: 914d71f3cc: Temporary hack to drain excess refreshLinks jobs (duration: 00m 14s)
* 00:12 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Gadgets: 7539873979: Update Gadgets for cherry-pick (duration: 00m 12s)
* 00:10 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Gadgets: cbb9b1e475: Update Gadgets for cherry-pick (duration: 00m 12s)
 
== May 12 ==
* 23:40 ori: Upgraded all Apaches to HHVM 3.6.1+dfsg1-1+wm2 and Apache 2.4.7-1ubuntu4.4
* 23:26 logmsgbot: demon Synchronized php-1.26wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
* 23:24 logmsgbot: demon Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 11s)
* 23:23 logmsgbot: demon Synchronized php-1.26wmf5/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 12s)
* 23:23 logmsgbot: demon Synchronized php-1.26wmf5/includes/media/DjVu.php: (no message) (duration: 00m 12s)
* 23:18 ori: Upgrading more HHVMs; DPKG alerts likely but they will be transient.
* 23:10 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 11s)
* 23:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: swat (duration: 00m 12s)
* 21:48 logmsgbot: kaldari Finished scap: updating i18n for Gather (1.26wmf5) (duration: 23m 17s)
* 21:25 logmsgbot: kaldari Started scap: updating i18n for Gather (1.26wmf5)
* 21:24 logmsgbot: kaldari Synchronized php-1.26wmf5/extensions/Gather: Updating Gather for 1.26wmf5 (duration: 00m 12s)
* 21:06 apergos: manually installed trigger-trebuchet update on tin after accidental salt upgrade there woops :-D
* 20:56 mutante: upgrading salt packages on tin
* 19:50 ori: Upgrading several app servers to new version of HHVM, expect transient 'DPKG CRITICAL' alerts
* 18:19 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf5
* 17:38 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ie4641b6e4: Set $wgWMEStatsdBaseUri to host-relative beacon/ path (duration: 00m 12s)
* 16:24 yurik: graphoid service synced, now supports Cache Control headers
* 16:19 ori: restarted HHVM on mw1061; T89912
* 15:20 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT Add *.sl.nsw.gov.au to wgCopyUploadsDomains [[gerrit:210356]] (duration: 00m 11s)
* 15:15 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT Namespaces configuration on or.wiktionary [[gerrit:210350]] (duration: 00m 12s)
* 15:10 hashar: mediawiki-phpunit-hhvm Jenkins job is broken due to an hhvm upgrade {{bug|T98876}}
* 15:07 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT enable NewUserMessage on bh.wikipedia [[gerrit:209146]] (duration: 00m 13s)
* 13:55 akosiaris: temporarily blocked an IP on uranium firewall. It was the cause of requests causing CPU load. http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=uranium.wikimedia.org&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Miscellaneous+eqiad
* 11:06 twentyafterfour: restarted apache on iridium to clear php opecode cache
* 09:53 akosiaris: restarted gitblit on antimony
* 06:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 12 06:57:17 UTC 2015 (duration 57m 16s)
* 06:15 springle: pt-kill on 3600s running on dbstore1002 until repl streams recover
* 06:05 springle: killed 100+ 3-day unindexed research queries on dbstore1002, all repl streams lagging and /tmp unhappy
* 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-12 03:00:22+00:00
* 02:57 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 47s)
* 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-12 02:34:30+00:00
* 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 33s)
* 00:39 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Update Wikipedia word mark and related config (duration: 00m 11s)
* 00:38 logmsgbot: mattflaschen Synchronized images/mobile/wikipedia-wordmark-en.png: Update Wikipedia word mark and related config (duration: 00m 13s)
* 00:30 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Add www.jacar.go.jp to wgCopyUploadsDomains (duration: 00m 11s)
* 00:30 yuvipanda: restarted nutcracker on silver
* 00:28 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Deploy Catalan Wikinews flood group (duration: 00m 13s)
* 00:19 logmsgbot: mattflaschen Synchronized php-1.26wmf5/includes/page/WikiPage.php: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)
* 00:18 logmsgbot: mattflaschen Synchronized php-1.26wmf5/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)
* 00:17 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 13s)
* 00:15 yuvipanda: restarted apache on silver
* 00:01 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/page/WikiPage.php: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 11s)
* 00:00 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)
 
== May 11 ==
* 23:46 logmsgbot: mattflaschen Synchronized wmf-config: Sync wmf-config for CirrusSearch PoolCounter change; applies to group 0 initially (duration: 00m 12s)
* 23:37 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings-labs.php: sync InitialiseSettings-labs.php for Browse experiment in mobile (duration: 00m 13s)
* 23:34 logmsgbot: mattflaschen Synchronized php-1.26wmf5/extensions/Flow/: Deploy Flow metadataonly fix (duration: 00m 14s)
* 23:32 yuvipanda: andrewbogott_afk playing around with upgrading virt*** boxes, which are non-live labs boxen.
* 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf4/extensions/Flow/: Deploy Flow metadataonly fix (duration: 00m 13s)
* 23:17 logmsgbot: mattflaschen Synchronized wmf-config/CommonSettings.php: Make VE default editor for Flow (duration: 00m 13s)
* 23:13 legoktm: manually renamed and migrated User:~~@nlwiki --> User:~~-~nlwiki@global (T98155)
* 22:55 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Josa: dd2db67d9b: Update Josa for cherry-picks (duration: 00m 13s)
* 22:54 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Josa: a0b561da25: Update Josa for cherry-picks (duration: 00m 11s)
* 22:05 twentyafterfour: removed /var/run/phab_repo_lock_libext_Sprint on iridium to allow sprint repo sync
* 22:01 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings-labs.php: Add common wikitag for all beta cluster wikis (duration: 00m 12s)
* 21:54 ori: Restarting HHVM on mw1036; threads stuck on HPHP::StatCache::refresh
* 21:48 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I45c1c76d4: Deploy Josa extension to production (enabling) (duration: 00m 13s)
* 21:47 logmsgbot: ori Finished scap: I45c1c76d4: Deploy Josa extension to production (but not enabling yet) (duration: 46m 54s)
* 21:43 ori: Restarting HHVM on mw1110; threads stuck on HPHP::StatCache::refresh
* 21:00 logmsgbot: ori Started scap: I45c1c76d4: Deploy Josa extension to production (but not enabling yet)
* 20:49 hoo: Resolved T98695 by setting the email of the global account to the former enwiki email address.
* 19:37 hoo: Updated Wikidata's property suggester with data from today's json dump
* 18:49 legoktm: renamed a bunch more invalid usernames (https://phabricator.wikimedia.org/T5507)
* 18:41 ori: Deployed I4e3f42ea7, which increases jobrunner::runners_basic from 14 -> 20
* 18:41 logmsgbot: yurik Synchronized wmf-config: patch 210111 - Cleaned Graph, enabled wmgGraphImgServiceAlways (duration: 00m 13s)
* 18:15 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph: Bump Graph to master (duration: 00m 11s)
* 18:14 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/Graph: Bump Graph to master (duration: 00m 14s)
* 17:16 logmsgbot: manybubbles Finished scap: SWAT js config vargs changes (duration: 14m 55s)
* 17:01 logmsgbot: manybubbles Started scap: SWAT js config vargs changes
* 17:01 logmsgbot: manybubbles scap aborted: SWAT js config vargs changes (duration: 27m 58s)
* 16:33 logmsgbot: manybubbles Started scap: SWAT js config vargs changes
* 15:59 manybubbles: waiting a few minutes after that last set of patches before we're sure that the load is down and then, hopefully, we'll scap to get the core changes that are already merged and sitting on tin that we had to ignore while we handled the trafic spike.
* 15:53 logmsgbot: manybubbles Synchronized php-1.26wmf4/includes/media/DjVu.php: SWAT: 10 mb djvu files are expensive to thumbnail (wmf4) (duration: 00m 13s)
* 15:52 logmsgbot: manybubbles Synchronized php-1.26wmf5/includes/media/DjVu.php: SWAT: 10 mb djvu files are expensive to thumbnail (wmf5) (duration: 00m 11s)
* 15:33 manybubbles: stopping SWAT due to some incident that just picked up. Right now Ib990f00ebe974008cea4dccbaa212ec20c846674 and Ida3fd5f8808202892001f66c4a534c1725e769a6 are merged awaiting a scap.
* 15:26 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT cleanup wgGraphImgServiceAlways 3/3 (duration: 00m 12s)
* 15:26 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT cleanup wgGraphImgServiceAlways 2/3 (duration: 00m 12s)
* 15:25 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings-labs.php: SWAT cleanup wgGraphImgServiceAlways 1/3 (duration: 00m 12s)
* 15:05 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT: send all mediawiki events from all wikis to logstash (duration: 00m 12s)
* 15:03 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: enable graph extension in beta. this should be a noop (duration: 00m 13s)
* 14:01 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary Wikibase access for nlwiki and frwikisource (duration: 00m 16s)
* 13:49 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix interaction with AbuseFilter (duration: 00m 20s)
* 13:46 logmsgbot: aude Synchronized php-1.26wmf5/extensions/Wikidata: Fix interaction with AbuseFilter (duration: 00m 19s)
* 05:10 ori: upgrading canary appservers to 3.6.1+dfsg1-1+wm2
* 04:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 11 04:53:58 UTC 2015 (duration 53m 57s)
* 04:17 springle: restarted hhvm on mw1020. lots of fatal noise about N4HPHP13DataBlockFullE
* 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-11 02:42:42+00:00
* 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 37s)
* 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-11 02:22:25+00:00
* 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 19s)
 
== May 10 ==
* 17:45 ori: App server traffic coincides with spike on S4 dbs, lots of commons sleeper queries, fatal log contains many references to User:Richenza/gallery, so nuking.
* 17:20 ori: Inbound app server traffic more than doubled over the past 12 hrs: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report
* 05:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 10 05:16:10 UTC 2015 (duration 16m 9s)
* 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-10 02:44:48+00:00
* 02:41 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 26s)
* 02:25 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-10 02:24:40+00:00
* 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 16s)
 
== May 9 ==
* 20:55 logmsgbot: krenair Synchronized php-1.26wmf4/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWEditModeTool.js: https://gerrit.wikimedia.org/r/#/c/209950/ (duration: 00m 12s)
* 20:53 logmsgbot: krenair Synchronized php-1.26wmf5/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWEditModeTool.js: https://gerrit.wikimedia.org/r/#/c/209949/ (duration: 00m 11s)
* 05:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May  9 05:05:16 UTC 2015 (duration 5m 15s)
* 02:44 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-09 02:43:07+00:00
* 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 21s)
* 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-09 02:23:15+00:00
* 02:19 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 11s)
 
== May 8 ==
* 23:45 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: beta: switch $wmfUdp2logDest to deployment-fluorine.eqiad.wmflabs (duration: 00m 12s)
* 22:11 mutante: gzipping some user data on lutetium
* 21:17 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
* 21:14 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
* 21:02 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Sync out change that only affects Beta Cluster (duration: 00m 11s)
* 19:18 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 13s)
* 19:18 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6236f5e2c: Use $wgServer to construct static-asset URLs (duration: 00m 12s)
* 19:12 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 12s)
* 18:42 csteipp: deployed patch for T98313 for wmf4/5
* 18:14 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph/: Bumping graph (duration: 00m 14s)
* 18:14 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/Graph/: Bumping graph (duration: 00m 14s)
* 16:53 logmsgbot: bd808 Synchronized w/static/images/project-logos/labswiki.png: Add missing labswiki.png (duration: 00m 13s)
* 15:37 Krenair: restarted apache on silver -again- to deal with reports of session errors
* 15:28 greg-g: wikitech's session data errors are transient, hitting save multiple times will eventually work
* 15:26 greg-g: multiple independent reports of wikitech wiki having session data errors
* 14:13 logmsgbot: bblack Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 13s)
* 13:17 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
* 13:17 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
* 13:14 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: revert bits.wm.org change (duration: 00m 12s)
* 13:14 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: revert bits.wm.org change (duration: 00m 12s)
* 13:03 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 15s)
* 13:03 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 14s)
* 11:49 godog: deploy librenms 2fa805ff
* 09:39 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz_0.1.0~r60155-1
* 09:39 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan-nor_1.0.0~r48173-1
* 05:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May  8 05:13:23 UTC 2015 (duration 13m 22s)
* 04:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I4c70ce4d0: Fix wikiname: roa-rupwiki -> roa_rupwiki (duration: 00m 12s)
* 03:33 logmsgbot: legoktm Synchronized w/static/images/project-logos/wikimania2015wiki.png: Use png for wikimania2015wiki logo (duration: 00m 12s)
* 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-08 02:48:15+00:00
* 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 47s)
* 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-08 02:28:07+00:00
* 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 06s)
* 00:00 logmsgbot: rmoen Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
 
== May 7 ==
* 23:54 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/VisualEditor/: Update VE with Cherry-picks (duration: 00m 12s)
* 23:51 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/VisualEditor/: Update VE for cherry-picks (duration: 00m 11s)
* 23:41 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Flow/: Bump flow with cherry-picks (duration: 00m 13s)
* 23:39 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/Flow: Bump Flow with cherry-picks (duration: 00m 14s)
* 23:31 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Gather: Update Gather with cherry-picks (duration: 00m 14s)
* 23:20 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/Gather/: Update Gather with Cherry-picks (duration: 00m 15s)
* 22:58 andrewbogott: restarting all instances on labvirt1008, crossing fingers
* 22:38 andrewbogott: rebooting labvirt1008, running dist-upgrade, rebooting again
* 21:29 awight: updated payments from 3ab89e2b14eb449f7ceddf2325493d6235395ecd to f97f8f99268974cfdb0182f178955bd627137842
* 21:25 gwicke: deployed RESTBase 6043e3ada (v0.6.2)
* 21:01 apergos: dumps are interrupted on snapshot1004 while I do a manual run for testing/debugging purposes. please let it run and don't start any other processes on the box, thanks
* 20:53 bd808: Updated kibana to bb9fcf6 (Merge remote-tracking branch 'upstream/kibana3')
* 20:36 legoktm: renaming users with invalid usernames (https://phabricator.wikimedia.org/T5507)
* 20:18 logmsgbot: ori Synchronized wmf-config: I3846e34ed, I1fcb3f17d, I8c9a6a567, I1a73c83f7, and Iacbd92931: serve optimized, cacheable logos from /static (duration: 00m 19s)
* 20:14 bd808: updated scap to 5d681af (Better handling for php lint checks)
* 20:14 bd808: Trebuchet checkout failed for scap/scap on mw1222.eqiad.wmnet, mw1113.eqiad.wmnet, mw1104.eqiad.wmnet
* 20:13 bd808: Trebuchet fetch for scap/scap failed on mw1222.eqiad.wmnet
* 19:17 logmsgbot: legoktm Synchronized php-1.26wmf4/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/209538 and https://gerrit.wikimedia.org/r/209539 (duration: 00m 16s)
* 19:16 logmsgbot: legoktm Synchronized php-1.26wmf5/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/209538 and https://gerrit.wikimedia.org/r/209539 (duration: 00m 16s)
* 16:56 bd808: sync-common on snapshot1004 finished in 12:36
* 16:49 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Enable shortURL on saprojects [[gerrit:201216]] (duration: 00m 14s)
* 16:43 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on newiki [[gerrit:206736]] (duration: 00m 21s)
* 16:37 bd808: Running sync-common manually on snapshot1004.eqiad.wmnet
* 16:36 thcipriani: create shorturl table in sawiki, sawikisource, sawikiquote, sawiktionary, sawikibooks
* 16:36 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 16m 21s)
* 16:23 thcipriani: populateShortUrlTable on newiki
* 16:20 thcipriani: creating newiki shorturl table
* 16:19 logmsgbot: kartik Started scap: Update ContentTranslation
* 15:48 logmsgbot: thcipriani Synchronized php-1.26wmf4/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: Update CentralAuth [[gerrit:209493]] (duration: 00m 21s)
* 15:34 logmsgbot: thcipriani Synchronized php-1.26wmf5/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: Update CentralAuth [[gerrit:209492]] (duration: 00m 17s)
* 15:27 springle: db connection EINTR noise in logs, see T98489
* 15:16 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: CX enable content translations [[gerrit:209207]] (duration: 00m 12s)
* 14:39 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1019 (duration: 00m 14s)
* 13:55 moritzm: uploaded to apt.wikimedia.org jessie-wikimedia: linux-meta_1.1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-tat_0.1.0~r57462-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r57551-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-es_1.0.6~r60161-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-ca_1.0.6~r60158-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fr-es_0.9.2~r27040-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eus_0.1.0-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-es_0.3.3~r56159-1
* 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-en_0.3.1~r60155-1
* 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-gl_1.0.8~r57542-1
* 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ast_1.1.0~r60158-1
* 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-an_0.3.0~r60158-1
* 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-gl_0.5.2~r57551-1
* 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan_0.1.0-1
* 12:30 bblack: rebooting cp1070
* 12:26 godog: bounce uwsgi on graphite1001
* 12:25 godog: bounce uwsgi on graphite1001
* 10:26 godog: bounce uwsgi on graphite1001
* 10:01 mark: Decreased labstore1001 md125 sync_speed_min from 80000 to 40000
* 09:35 mark: Increased /sys/block/md125/md/sync_speed_min from 4000 to 40000
* 09:29 mark: Increased /sys/block/md125/md/sync_speed_min from 1000 to 4000
* 05:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May  7 05:39:36 UTC 2015 (duration 39m 35s)
* 03:03 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-07 03:02:50+00:00
* 02:59 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 08m 35s)
* 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-07 02:35:43+00:00
* 02:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1054 in s2, warm up (duration: 01m 09s)
* 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 09m 27s)
* 02:14 logmsgbot: krenair Synchronized wmf-config: update interwiki.cdb, T98429 (duration: 00m 24s)
* 01:50 bblack: we're still hitting cap on Zayo as of shortly-ago in graphs and seeing smokeping loss, moved california to eqiad
* 00:13 mutante: running refreshLinks.php for s2
* 00:11 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MobileFrontend/: SWAT (duration: 00m 42s)
* 00:11 gwicke: deployed RESTBase 8865b9c48
 
== May 6 ==
* 23:43 logmsgbot: catrope Synchronized php-1.26wmf5/extensions/VisualEditor: SWAT (duration: 00m 18s)
* 23:43 logmsgbot: catrope Synchronized php-1.26wmf5/extensions/MobileFrontend: SWAT (duration: 00m 34s)
* 23:19 RoanKattouw: Running populateShortUrl.phg on knwiki
* 23:16 RoanKattouw: Running namespaceDupes.php on tewikiquote
* 23:15 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 17s)
* 23:12 RoanKattouw: Created shorturls table on knwiki
* 20:39 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf3
* 20:37 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf5
* 20:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf4
* 20:29 apergos: salt upgraded to 2014.7.5 on all precise/trusty/jessie hosts in production except for: labcontrol2001, tin, virt1000 (deferred) and dysprosium/labvirt1005/labstore1002 (down)
* 20:15 logmsgbot: twentyafterfour Synchronized php-1.26wmf5/extensions/MobileFrontend/javascripts/modules/search/init.js: Temporarily disable MobileWebSearch logging (duration: 00m 36s)
* 20:14 twentyafterfour: ignore all rumors of scap failures, the scaps were successful, with the exception of snapshot1004.eqiad.wmnet which hangs every time
* 20:14 logmsgbot: twentyafterfour Synchronized php-1.26wmf4/extensions/MobileFrontend/javascripts/modules/search/init.js: Temporarily disable MobileWebSearch logging (duration: 00m 37s)
* 20:12 logmsgbot: twentyafterfour scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 27m 49s)
* 19:44 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf5 and rebuild l10n cache
* 18:39 mutante: restarting apache on rhodium
* 18:34 bblack: rebooting cp3030
* 18:14 andrewbogott: restarted gmetad on uranium
* 17:41 andrewbogott: powering down virt1005 and virt1006
* 17:38 andrewbogott: depuppeting and decommissioning virt1005 and virt1006
* 17:24 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on enwikivoyage, fawiki and hewiki (duration: 00m 18s)
* 17:03 jgage: hadoop active namenode switched back to analytics1001 after rack C4 switch replacement
* 16:43 apergos: done with all trusty salt updates in pro except for labcontrol1002 (?), doing jessie now in very tiny batches, it's being trouble
* 15:29 bd808: Stashed uncommitted change to scap on tin that disabled php opening tag check for sync-file
* 15:27 bd808: Updated scap to 57036d2 (Update statsd events)
* 15:27 bd808: trebuchet checkout for scap/scap failed for mw1113.eqiad.wmnet, mw1222.eqiad.wmnet, mw1104.eqiad.wmnet
* 15:25 bd808: trebuchet fetch for scap/scap failed on mw1222.eqiad.wmnet
* 15:04 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Send group0 + group1 MediaWiki events to logstash {{gerrit|209170}} (duration: 00m 16s)
* 14:32 cmjohnson1: shutting down db1054 for maintenance
* 14:22 _joe_: depooling the HHVM imagescaler
* 14:20 Nemo_bis: phabricator went down again for some minutes, seems ok now?
* 14:17 _joe_: pooling the HHVM imagescalers to test if the issues are solved now.
* 14:15 andrewbogott: rebooting labvirt1009 one last time
* 13:53 _joe_: upgrading the hhvm imagescaler (mw1152) to HHVM 3.6.1
* 13:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1021 in s2, warm up (duration: 00m 27s)
* 13:42 apergos: all precise hosts are upgraded to salt except for tin and virt1000; in the middle of trusty updates now, in batches
* 13:38 _joe_: uploading HHVM 3.6.1 and all the related extensions to apt.wikimedia.org
* 13:01 paravoid: replacing asw-c4-eqiad (T93730)
* 12:45 logmsgbot: krenair Synchronized php-1.26wmf4/extensions/SemanticMediaWiki/specials/QueryPages/SMW_QueryPage.php: https://gerrit.wikimedia.org/r/#/c/209212/ (duration: 00m 21s)
* 08:12 logmsgbot: legoktm Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 24s)
* 07:20 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I019944f42: Change EventLogging endpoint to /beacon/event (duration: 00m 14s)
* 06:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May  6 06:50:27 UTC 2015 (duration 50m 26s)
* 03:14 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-06 03:13:28+00:00
* 03:09 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 08m 46s)
* 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-06 02:45:26+00:00
* 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 10m 46s)
* 02:27 springle: xtrabackup clone db1060 to db1021
* 02:04 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I83ad6d060: Remove wmgUseBits setting, now that the migration is complete (duration: 00m 18s)
* 02:02 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix Wikibase api error output bug - update submoduled (duration: 00m 28s)
* 01:59 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix Wikibase api error output bug (duration: 01m 08s)
* 01:52 logmsgbot: ori Synchronized multiversion/MWWikiversions.php: Ib08e36901: MWWikiversions::readDbListFile: allow single-line ("#" or "//") comments (duration: 00m 18s)
* 01:40 springle: upgrade db1021 trusty
* 00:51 springle: schema change running T95179 wikidata, bit unusual, dropping a not-null field
* 00:46 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: Add AffCom user group application contact page on meta {{gerrit|207332}} (duration: 00m 20s)
* 00:45 logmsgbot: bd808 Synchronized docroot/noc/createTxtFileSymlinks.sh: Add AffCom user group application contact page on meta {{gerrit|207332}} (duration: 00m 17s)
* 00:45 logmsgbot: bd808 Synchronized docroot/noc/conf/AffComContactPages.php.txt: Add AffCom user group application contact page on meta {{gerrit|207332}} (duration: 00m 15s)
* 00:44 logmsgbot: bd808 Synchronized wmf-config/AffComContactPages.php: Add AffCom user group application contact page on meta {{gerrit|207332}} (duration: 00m 33s)
* 00:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/Flow: SWAT (duration: 00m 23s)
* 00:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikiEditor: SWAT (duration: 00m 33s)
* 00:13 bd808: Aborted sync-common on snapshot1004; host is starved for RAM and using swap heavily
* 00:06 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/CirrusSearch: SWAT (duration: 00m 28s)
* 00:06 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/Flow: SWAT (duration: 00m 52s)
* 00:04 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/WikiEditor: SWAT (duration: 00m 42s)
 
== May 5 ==
* 23:57 bd808: aborted and restarted sync-common on snapshot1004.eqiad.wmnet manually after waiting 24 minutes with no progress
* 23:49 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Use Wiki.svg for wikimania2015wiki logo (duration: 00m 19s)
* 23:47 jgage: switched hadoop active namenode from analytics1001 to analytics1002 for rack C4 switch replacement tomorrow morning (T93730)
* 23:39 logmsgbot: rmoen Finished scap: Updates for Gather and MobileFrontend (duration: 41m 11s)
* 23:33 bd808: running sync-common on snapshot1004.eqiad.wmnet manually after it was aborted in scap by rmoen
* 23:30 bd808: snapshot1004.eqiad.wmnet hanging scap yet again
* 23:23 mutante: deleted 8G recurring_blocked.tsv from lutetium
* 22:58 logmsgbot: rmoen Started scap: Updates for Gather and MobileFrontend
* 22:54 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/Gather/: Update Gather to master (duration: 00m 36s)
* 22:53 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/MobileFrontend/: Update MobileFrontend (duration: 00m 31s)
* 22:52 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Gather/: Update Gather to master (duration: 00m 25s)
* 22:52 mutante: gzip lutetium-slow.log on lutetium to save disk space
* 22:52 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/MobileFrontend/: Update MobileFrontend (duration: 00m 39s)
* 22:23 mutante: apt-get clean on lutetium to free disk space
* 19:53 twentyafterfour: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf4 (actual time 18:12 UTC)
* 19:44 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix usage tracking issue on Wikidata - with submodule update (duration: 00m 33s)
* 19:41 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix usage tracking issue on Wikidata (duration: 00m 40s)
* 19:35 bblack: rebooting cp3030 ...
* 19:23 yuvipanda: disabled puppet on zookeeper hosts
* 18:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I5978a3910: Update $wgULSFontRepositoryBasePath for post-bits world (duration: 00m 18s)
* 18:43 logmsgbot: ori Synchronized wmf-config: Ia98fc4c5d: wmgUseBits: false for enwiki (duration: 00m 17s)
* 18:33 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I2ee277293: wmgUseBits: false for all but enwiki (duration: 00m 13s)
* 17:50 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Enable graph extension on all wikis except wikidata (duration: 00m 19s)
* 17:43 logmsgbot: yurik Synchronized php-1.26wmf3/extensions/Graph: Cherrypicked Graph ext 209004 (duration: 00m 16s)
* 17:42 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph: Cherrypicked Graph ext 209004 (duration: 00m 20s)
* 17:00 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: Enable graphoid noscript fallback for graph ext (duration: 00m 20s)
* 16:50 yurik_: deployed latest graphoid 0.1.3 service
* 15:16 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Add medialib.naturalis.nl to wgCopyUploadsDomains [[gerrit:208634]] (duration: 00m 26s)
* 14:07 godog: shut fluorine to replace sdb
* 13:13 akosiaris: restarted apache2 on palladium
* 13:04 Tim: updating voter list for the FDC election for T97924
* 08:47 paravoid: repooling ulsfo
* 07:59 godog: test reboot fluorine with new disk
* 05:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May  5 05:50:01 UTC 2015 (duration 50m 0s)
* 05:07 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll/cli/wm-scripts/bv2015/voterList.php: (no message) (duration: 00m 16s)
* 04:43 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll/cli/wm-scripts/bv2015/voterList.php: (no message) (duration: 00m 19s)
* 02:59 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-05 02:57:54+00:00
* 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 06s)
* 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-05 02:30:45+00:00
* 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 20s)
* 01:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1021, move s5 api to db1049 (duration: 00m 15s)
* 01:20 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1070, warm up (duration: 00m 19s)
* 00:32 yuvipanda: restarted hhvm on mw1197
* 00:24 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable Wikibase subscription tracking (duration: 00m 12s)
 
== May 4 ==
* 23:59 logmsgbot: catrope Finished scap: (no message) (duration: 24m 34s)
* 23:34 logmsgbot: catrope Started scap: (no message)
* 23:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/MassMessage/: SWAT (duration: 00m 12s)
* 23:14 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MassMessage/: SWAT (duration: 00m 12s)
* 23:14 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/VisualEditor/: SWAT (duration: 00m 12s)
* 23:13 logmsgbot: catrope Synchronized php-1.26wmf4/includes/skins/SkinTemplate.php: SWAT (duration: 00m 11s)
* 22:37 Krenair: silver: apache2ctl restart for T98084
* 22:26 Tim: on terbium: running voterList.php again, with corrected edit counts
* 21:55 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Id56e33263: wmgUseBits: false for ru and eswiki (duration: 00m 12s)
* 21:40 logmsgbot: bd808 Finished scap: Update 1.26wmf4 ContactPage and WikimediaMessages for AffCom contact form (duration: 22m 11s)
* 21:34 paravoid: cr{1,2}-{eqiad,ulsfo}: swapping metrics for ulsfo's transport links
* 21:18 logmsgbot: bd808 Started scap: Update 1.26wmf4 ContactPage and WikimediaMessages for AffCom contact form
* 21:03 Coren: checking raid consistency from labstore1002
* 21:03 ottomata: rebooting analytics1037
* 20:27 Coren: Starting NFS server switch - graceful labstore1001 shutdown.
* 20:11 gwicke: deployed restbase v0.6.0 / 76583a07
* 19:56 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I62dffd271: wmgUseBits: false for nl and dewiki (duration: 00m 11s)
* 19:24 logmsgbot: ori Synchronized w/5xx.php: (no message) (duration: 00m 14s)
* 19:12 awight: update crm from 514e7ea41acd14e1565b31b76621ea840d209e07 to 2a2336655737a2cd1d3cc24624d1e8475e4cf039
* 19:12 logmsgbot: ori Synchronized multiversion: I2d93ede75: Remove FormatJson from mediawiki-config (duration: 00m 13s)
* 18:51 logmsgbot: ori Synchronized multiversion/FormatJson.php: Ice8f1796c: Update FormatJson to 532337e6ff from mediawiki/core (duration: 00m 12s)
* 18:44 cscott: updated Parsoid to version b53a7272
* 18:26 logmsgbot: ori Synchronized wmf-config: I81df3a614, I02b06f8e2, I366561a0f: Use MWWikiversions::readDbListFile to read dblist files; Allow computed dblist expressions; Add group1.dblist (duration: 00m 14s)
* 17:53 legoktm: running delete-wmf-tags (https://phabricator.wikimedia.org/P531) on all extension repos
* 16:58 andrewbogott: reimaging/renaming virt1011 -> labvirt1007
* 15:40 logmsgbot: thcipriani Synchronized php-1.26wmf4/extensions/ContentTranslation: Update ContentTranslation to 0bd91b6 [[gerrit:208607]] (duration: 00m 30s)
* 15:32 logmsgbot: thcipriani Synchronized php-1.26wmf3/extensions/ContentTranslation: Sync-dir for ContentTranslation to 6f81619 [[gerrit:208605]] (duration: 00m 18s)
* 15:23 logmsgbot: thcipriani Synchronized php-1.26wmf3/extensions/ContentTranslation/modules/tools/ext.cx.tools.formatter.js: Update ContentTranslation to 6f81619 [[gerrit:208605]] (duration: 00m 25s)
* 15:17 ottomata: starting upgrade of Analytics Cluster to CDH 5.4: https://phabricator.wikimedia.org/T97453
* 15:05 andrewbogott: halting virt1011 pending its rename to labvirt1007
* 14:51 godog: halt fluorine to fix console and swap sda
* 14:50 paravoid: draining ulsfo, network troubles (internal network packet loss)
* 13:49 paravoid: draining all traffic from the Giglinx/Zayo link to ulsfo
* 05:56 Tim: on terbium: running populateEditCount-fixup.php on all wikis
* 05:53 logmsgbot: tstarling Synchronized php-1.26wmf4/extensions/SecurePoll: Iae874c0403a8362929362ca645f4aca18feb0269 (duration: 00m 19s)
* 05:52 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll: Iae874c0403a8362929362ca645f4aca18feb0269 (duration: 00m 22s)
* 05:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May  4 05:35:29 UTC 2015 (duration 35m 28s)
* 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-04 02:48:16+00:00
* 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 33s)
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-04 02:26:00+00:00
* 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 58s)
* 01:13 bd808: Started logstash cluster relocating indices off of logstash100[1-3] to logstash100[4-6]
 
== May 3 ==
* 19:28 yuvipanda:  chown www-data: /var/log/mediawiki/refreshLinks/s3@3.log and s2@2.log for Reedy
* 16:23 logmsgbot: hoo Synchronized wmf-config/: Re-enable global renames (duration: 00m 12s)
* 15:17 _joe_: restarted jobchron, not jobcron, this time for real
* 14:37 bblack: dewiki jobqueue:*:rootjob wipe complete
* 14:37 bblack: enwiki + commonswiki jobqueue:*:rootjob wipe complete
* 14:19 bblack: deleting :rootjob: entries for enwiki from redis too
* 14:16 bblack: deleting :rootjob: entries for commonswiki from redis
* 13:54 _joe_: restarting jobcron on the jobrunners
* 13:27 logmsgbot: hoo Synchronized wmf-config/: Temporary disable global renames (duration: 00m 16s)
* 12:47 _joe_: restarting redis server on rdb1001, lagging on the most basic queries
* 12:38 _joe_: deploying I969fe8d329c1bcbb919a54cb225200ba0e006a03 to the jobrunners trying to make them work again
* 05:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May  3 05:13:13 UTC 2015 (duration 13m 12s)
* 04:28 springle: xtrabackup clone db1049 to db1070
* 04:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1070 (duration: 00m 16s)
* 02:48 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-03 02:47:30+00:00
* 02:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1068, warm up (duration: 00m 15s)
* 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 11s)
* 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-03 02:26:02+00:00
* 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 11s)
 
== May 2 ==
* 22:16 ori: Deployed change I3bc87f3a5 to fix UBN! bug T97912. Bug was affecting ability to translate messages needed for running upcoming board election.
* 22:16 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Translate/api/ApiQueryMessageGroups.php: I3bc87f3a5: ApiQueryMessageGroups: mark '_canchange' and '_name' as non-API-metadata (duration: 00m 30s)
* 22:09 logmsgbot: ori Synchronized php-1.26wmf3/extensions/Translate/api/ApiQueryMessageGroups.php: I3bc87f3a5: ApiQueryMessageGroups: mark '_canchange' and '_name' as non-API-metadata (duration: 00m 31s)
* 20:25 windowcat: Updated jobrunners to c95d565e242e6fa3706c088ddab1cc6f716408e1
* 19:31 springle: xtrabackup clone db2048, db2049, db2050, db2051, db2052, db2053, db2054 from codfw masters
* 19:09 springle: upgrade db1068 trusty, xtrabackup clone from db1056
* 19:02 ottomata: resinstalling analytics1004 and analytics1010 as trusty
* 06:08 yuvipanda: signed puppet certs manually on virt1000
* 05:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May  2 05:18:29 UTC 2015 (duration 18m 28s)
* 03:24 ori: Granted self admin rights on metawiki temporarily to debug a CentralNotice issue.
* 02:53 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-02 02:52:36+00:00
* 02:48 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 01s)
* 02:32 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikiEditor: Fix data gathering bug (duration: 00m 25s)
* 02:32 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-02 02:31:00+00:00
* 02:27 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 11s)
* 02:15 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/WikiEditor: Fix data gathering bug (duration: 00m 15s)
* 00:02 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 16s)
 
== May 1 ==
* 23:53 logmsgbot: aaron Synchronized php-1.26wmf4/includes/media/DjVu.php: caa2efc0e76c2ba849d465006600d131dc2f78b5 (duration: 00m 21s)
* 23:52 logmsgbot: aaron Synchronized php-1.26wmf3/includes/media/DjVu.php: 6cdb23c5d662151a2b578c2acc8823bc975fc22a (duration: 00m 15s)
* 23:40 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I02e28db61: Update apple-touch to use static (duration: 00m 23s)
* 21:08 matt_flaschen: Ran FlowUpdateWorkflowPageId.php for all production Flow wikis for https://phabricator.wikimedia.org/T96888
* 20:37 logmsgbot: andyrussg Synchronized php-1.26wmf4/extensions/EducationProgram/: Update EducationProgram (duration: 00m 21s)
* 20:01 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: less realm stuff (duration: 00m 17s)
* 20:00 logmsgbot: andyrussg Synchronized php-1.26wmf3/extensions/EducationProgram/: Update EducatiDonProgram (duration: 00m 30s)
* 18:54 logmsgbot: legoktm Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/208170/ no-op (duration: 00m 25s)
* 18:53 logmsgbot: legoktm Synchronized all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/208170/ no-op (duration: 00m 18s)
* 18:11 logmsgbot: legoktm Synchronized all-labs.dblist: https://gerrit.wikimedia.org/r/208154 - no-op (duration: 00m 19s)
* 15:58 logmsgbot: anomie Synchronized php-1.26wmf3/includes/: Deploy [[gerrit:208109]] to reduce the complaining about the new feature (duration: 00m 28s)
* 15:50 logmsgbot: anomie Synchronized php-1.26wmf4/includes/: Deploy [[gerrit:208109]] to reduce the complaining about the new feature (duration: 00m 24s)
* 15:29 gwicke: finished restarting cassandra nodes on restbase100*.eqiad
* 15:21 ottomata: doing java security update on kafka brokers, doing rolling restarts
* 14:50 gwicke: slowly restarting restbase100*.eqiad to apply new gen size change
* 10:47 godog: bounce apache2 on strontium
* 10:47 godog: bounce apache2 on palladium, mod_passenger died
* 05:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May  1 05:44:23 UTC 2015 (duration 44m 22s)
* 03:05 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-01 03:04:21+00:00
* 03:01 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 45s)
* 02:38 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-01 02:37:20+00:00
* 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 09m 46s)
* 00:18 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/PageTriage/: SWAT (duration: 00m 30s)
* 00:13 logmsgbot: ori Synchronized wmf-config: Iae2e55a11: wmgUseBits: false for itwiki (duration: 00m 19s)