Server Admin Log

2015-11-03

01:05 awight: enabled CRM module amazon_audit
01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s)
01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s)
00:37 logmsgbot: ebernhardson@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s)
00:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
00:34 logmsgbot: ebernhardson@tin Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
00:28 ebernhardson: sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615
00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s)
00:17 awight: update CRM from 77cd64317404b2fcec9500cfbb1c0b19e5846705 to 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7
00:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250590/ (duration: 00m 18s)
00:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: add www.webarchive.co.uk and *.unesco.org to server side upload whitelist (duration: 00m 18s)
00:06 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/246169/ (duration: 00m 18s)

2015-11-02

23:11 RoanKattouw: Running convertAllLqtPages.php on ptwikibooks
22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Flow on ptwikibooks with English-only Topic namespace name (duration: 00m 18s)
22:38 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgFlowEnglishNamespaceOnly (duration: 00m 18s)
21:56 cwd: updated worldpay request timeout to 30s
21:31 _joe_: restarting parsoids in codfw
21:25 subbu: deployed parsoid f0d77afc
20:49 mutante: ms-be2017 - initial puppet run, fresh install
20:38 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.4/includes/: (no message) (duration: 00m 22s)
20:38 robh: lawrencium having puppet stopped and reclaiming into spares per T117477 disregard any alerts
20:06 mutante: ms-be2016 - signing puppet certs, salt-key, initial run
20:05 mutante: started morebots (production-logbot job was gone)
19:46 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/includes/MobileFrontend.skin.hooks.php: Fix changing the license message key via the "MobileLicenseLink" hook (duration: 00m 17s)
19:14 gwicke: restbase cassandra: switching local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ to DTCS
19:00 matt_flaschen: Completed FlowFixLinks run on all Flow wikis
18:53 csteipp: deployed patches for T97897 & T115522
18:53 awight: updated CRM from 60c4a8e5fc4efb6ac38a520e652afe187e8c177b to 77cd64317404b2fcec9500cfbb1c0b19e5846705
18:48 andrewbogott: dist-upgrade and rebooting labvirt1005
18:41 Coren: manually starting service replicate-tools on labstore1001
18:10 awight: removed failing SmashPig job runner task
18:01 jynus: performing some fixes on labsdb1004 database. Slightly higher load on labsdb1005.
17:59 dcausse: restarting elastic on nobelium
17:43 awight: disabled Donations queue consume job
17:39 gwicke: restbase cassandra: switched local_group_wikisource_T_parsoid_html and local_group_wikibooks_T_parsoid_html to DTCS
17:09 bd808: Decreased replica count to 1 for logstash-2015.10.04 thru logstash-2015.10.12 to free cluster disk space; see T117438
17:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/249762/ (duration: 00m 19s)
17:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247088/ (duration: 00m 17s)
16:48 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
16:47 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
16:45 logmsgbot: krenair@tin Synchronized dblists/closed.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ - close wikimania2014.wikimedia.org (duration: 00m 18s)
16:36 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249794/ - should be a noop for prod (duration: 00m 17s)
16:30 bd808: Deleted logstash-2015.10.03 index to free disk space on logstash1004; see T113571
16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/249940/1 (duration: 00m 20s)
16:26 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249935/ (duration: 00m 18s)
16:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250401/ (duration: 00m 18s)
14:43 ottomata: shutting down analytics1038 to run fsck on root filesystem. Currently it is read only
14:36 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-br-fr_0.5.0+svn~57870-1
14:35 mobrovac: citoid deployed 70abb90
13:34 paravoid: draining ulsfo, T116928
12:19 mobrovac: mobileapps deploying 59e8c30
10:52 hashar: Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion"
10:10 akosiaris: powering off pollux. migrating to a VM
09:39 _joe_: moved tor exit node, update_special_pages off of terbium to mw1152
09:24 mobrovac: citoid restarted zotero on sca100x
09:07 jynus: deleting user_daily_contribs view on labs (it points to a non-existent table)
08:57 mobrovac: restbase deploying 8036232
05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Nov 2 05:33:07 UTC 2015 (duration 33m 6s)
02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-02 02:25:42+00:00
02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 44s)

2015-11-01

16:59 gwicke: restarting cassandra on aqs1002; was out of heap space
05:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Nov 1 05:25:49 UTC 2015 (duration 25m 48s)
02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-01 02:24:46+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 36s)

2015-10-31

08:57 dcausse: elastic1008 deleting /var/log/elasticsearch/production-search-eqiad_index_indexing_slowlog.log.[2-7]
05:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 31 05:15:43 UTC 2015 (duration 15m 42s)
02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda4768 (duration: 00m 19s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-31 02:26:22+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 26s)

2015-10-30

23:53 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ifd3543d99: Re-enabled sidebar cache per 47eb083a0fe4 (duration: 00m 17s)
23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/cache/MessageCache.php: Id9a27ba2bbd3 (duration: 00m 17s)
23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/skins/Skin.php: Id9a27ba2bbd3 (duration: 00m 17s)
23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Collection: b89cff0a1d: Updated mediawiki/core Project: mediawiki/extensions/Collection 092204bf333e65b2c749e4bc4a32fd2b0254089b (duration: 00m 18s)
23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/DismissableSiteNotice: a20aa8a544: Updated mediawiki/core Project: mediawiki/extensions/DismissableSiteNotice 5b8f1cfac48704fd740eb61f873aabb600c4c5fb (duration: 00m 17s)
23:03 gwicke: fixed restbase labs install by deploying it; the code had gone missing
23:03 ori: Added MaxSem and gwicke to Gerrit Project Creators group
22:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/objectcache/ObjectCache.php: If12aedae7f: objectcache: Use singleton cache in newAccelerator() (duration: 00m 18s)
22:05 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
22:01 mutante: labvirt1010: salt-minion always in "stop/waiting", doesn't restart
21:54 mutante: labvirt1010 - start salt
21:41 ori: Removing MediaWiki.xhprof.* from graphite{1,2}001
21:41 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s)
19:07 K4-713: antifraud tweak on payments wiki (again)
18:56 K4-713: antifraud tweak on payments wiki
18:56 jynus: disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact
17:33 robh: bismuth has no console output, appears locked up in OS, rebooting via mgmt
17:22 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 18s)
16:56 cmjohnson: reinstalling mw1083 https://phabricator.wikimedia.org/T116184
15:35 moritzm: updated db205[6-8] to the 3.19 kernel
14:38 chasemp: nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable
13:47 Krenair: revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin
13:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s)
11:57 paravoid: upgrading tor to latest stable on radium
11:31 paravoid: ignoring asw-d-eqiad on librenms
11:25 _joe_: powercycling dataset1001, stuck in some kernel task, unable to login from console
10:42 dcausse: elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki
10:30 dcausse: elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181)
09:18 dcausse: elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1
08:46 _joe_: removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008
06:24 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s)
05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s)
04:19 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s)
04:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s)
04:03 awight: update CRM from e0cad7215192464941d0ec282a70cb608619cf2f to 60c4a8e5fc4efb6ac38a520e652afe187e8c177b
03:42 awight|parentoid: update CRM from 367cd85954697c6ad4311deb2799fe9ba08b5d9d to e0cad7215192464941d0ec282a70cb608619cf2f
02:47 awight|parentoid: donations queue consumer reenabled
02:45 awight|parentoid: updated CRM from 61c8c13efb31f7e75564d9a01fa879db0690fb78 to 367cd85954697c6ad4311deb2799fe9ba08b5d9d
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s)
00:48 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s)
00:34 mutante: labvirt1010 - started salt-minion (no such process)
00:34 mutante: mw1193 - restarted HHVM (socket timeout)
00:34 mutante: wdqs1001 - restarted NTP (unknown offset)
00:09 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s)
00:08 mutante: radium - back in service
00:01 bd808: Restarted logstash on logstash1003 again. The first try apparently didn't take

2015-10-29

23:44 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s)
23:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s)
23:32 bd808: Restarted logstash on logstash1003; died with OOM error
23:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s)
23:21 mutante: radium: scheduled downtime, reinstalling
23:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s)
23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s)
23:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s)
23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s)
23:09 awight: set dashboard cache interval to 60m
22:33 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s)
22:31 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s)
22:30 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s)
22:29 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s)
22:05 awight: CiviCRM upgrade to 4.6.9 succeeded; new data is backed up
22:02 gwicke: restbase: switched local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS)
21:40 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s)
21:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s)
21:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s)
20:56 gwicke: restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS)
20:47 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
19:50 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I06879b6e6e: Enable ScribuntoGatherFunctionStats (duration: 00m 17s)
19:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.3
19:39 twentyafterfour: rolling back to 1.27.0-wmf.3 due to increase in log errors
19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.4
18:59 cmjohnson1: powering down aqs1001 for h/w maintenance
17:53 AaronSchulz: Touched/synced PrivateSettings.php symlink via touch -h
17:47 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 18s)
17:36 AaronSchulz: Did touch of InitialiseSettings.php
17:33 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
17:32 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 17s)
17:17 logmsgbot: kartik@tin Synchronized wmf-config/PrivateSettings.php: Really sync right file this time (duration: 00m 17s)
17:10 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Retry syncing for CX token (duration: 00m 17s)
17:04 awight: restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC
17:02 awight: update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78
16:57 YuviPanda: restart etherpad on etherpad1001
16:22 moritzm: removed obsolete mysql 5.5 packages on mw102[2-9], mw1032, mw1053, mw1114, mw1163
16:19 _joe_: moved purge_abusefilter from terbium to mw1152
15:48 _joe_: moving purge_checkuser from terbium to mw1152
15:43 _joe_: moving purge_securepoll from terbium to mw1152
15:15 moritzm: removed openjdk-7 on restbase100[1-9] (it's using openjdk-8 for a while)
13:17 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Fix ContentTranslationCXServerAuth for CX (duration: 00m 18s)
13:14 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Fix name of JWT token (duration: 00m 18s)
12:33 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Set ContentTranslationCXServerAuth for CX (duration: 00m 17s)
11:24 mobrovac: restbase rolling-restart after config change https://gerrit.wikimedia.org/r/249465
10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 at 100% load. Reduce db1035 weight. (duration: 00m 17s)
10:34 _joe_: migrated jobqueue_stats_reporter to mw1152
10:21 hashar: restarting Jenkins (java upgrade)
10:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance (duration: 00m 18s)
10:14 hashar: Upgrading java on gallium and restarting Jenkins
10:04 moritzm: removed openjdk-7 from cassandra test hosts (now using openjdk-8)
10:01 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable nearby on wikidata (duration: 00m 18s)
10:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Config for fetching labels on mobile wikidata (duration: 00m 18s)
09:51 jynus: shutdown mw1083 to avoid querying the mysql servers with an outdated config/spaming the error logs
09:13 jynus: restarting db1065 for regular maintenace
08:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 17s)
07:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 29 07:04:04 UTC 2015 (duration 4m 3s)
06:43 YuviPanda: disabled two fr-tech related catchpoint tests per awight's request
06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 29ccbd248 (duration: 00m 17s)
05:46 ebernhardson: finished manually running 3M enwiki/enqueue, enwiki/wikibase-addUsagesForPage, and wikidatawiki/cirrusSearchLinksUpdatePrioritized jobs from mw1011 and mw1012
05:38 ebernhardson: restarting elasticsearch on nobelium to attempt to clear up extra log GC pauses in the old generation (50s+)
05:37 ebernhardson: restarting elasticsearch on nobelium
03:55 gwicke: cassandra *staging*: testing DateTieredCompactionStrategy (https://labs.spotify.com/2014/12/18/date-tiered-compaction/) on wikipedia html and data-parsoid tables
03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-29 03:05:52+00:00
03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 03s)
02:59 YuviPanda: start pybal on lvs1007
02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-29 02:46:27+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 47s)
02:05 AaronSchulz: Restarted stuck hhvm on mw1016; dump at /tmp/hhvm.25097.bt
01:06 gwicke: lowered gc_grace on wikipedia parsoid html and data-parsoid keyspaces to 24 hours

2015-10-28

23:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Translate/tag/PageTranslationHooks.php: I0e5f2d3b2 (duration: 00m 18s)
23:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249642 (duration: 00m 17s)
23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 17s)
23:22 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 17s)
23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 18s)
23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
23:15 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/249585 (duration: 00m 19s)
23:08 logmsgbot: ebernhardson@tin Synchronized portals: Switch www portals to be deployed from Git, but not being served from anywhere yet (duration: 00m 18s)
23:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I0e5f2d3b2 (duration: 00m 18s)
22:57 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/includes/Hooks.php: I0e5f2d3b2 (duration: 00m 18s)
22:38 urandom: Cassandra cleanup on restbase-test2001-a
22:31 awight: Fundraising CRM db migration started, estimated completion 10:15 UTC (tomorrow)
22:11 awight: update fundraising CRM to civi-4.6.9-deployment branch, from f2fa7b942625b34ede520e11f20e7e0835ecb17d to 23d2020448f343c8c1b2f4d779be66f57552f935
21:11 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.4/includes/changes/EnhancedChangesList.php: Fix diff/history links not showing up for ungrouped enhanced RC - https://gerrit.wikimedia.org/r/#/c/249556/ (duration: 00m 19s)
21:08 YuviPanda: enable puppet on labstore1001
20:55 YuviPanda: reverted changes to nsswitch.conf from puppet run manually on labstore2001
20:29 YuviPanda: disable puppet on labstore1001
20:20 awight: Disabled fundraising CiviCRM
19:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.4
18:50 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiversity
18:42 logmsgbot: legoktm@tin Synchronized wmf-config/mc.php: https://gerrit.wikimedia.org/r/#/c/249463/ (duration: 00m 17s)
18:41 ostriches: rolled scap back to master@62a250a, needs puppet changes before new code goes live
18:34 gwicke: restbase deploy done
18:34 yurik: updated kartotherian & tilerator services
18:23 gwicke: rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster
18:18 gwicke: canary deploy of restbase deploy 3b1f6488f2 to restbase1001
18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hiwiki
18:12 Krinkle: mwscript deleteEqualMessages.php --wiki fawiki
17:59 godog: rolling-restart cassandra-metrics-collector, staggered this time
17:23 ostriches: deployed scap master@f823129 to cluster
17:16 ostriches: deployed scap master@abe1973 to cluster
16:54 godog: unblacklist 'max' cassandra metrics and restart cassandra-metrics-collector
16:54 moritzm: installed openjdk security updates on zookeeper hosts
16:18 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: add settings for displaying labels on test.wikidata in mobile (duration: 00m 18s)
16:15 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: enable wikibase descriptions on test.wikidata (duration: 00m 17s)
15:46 andrewbogott: testing the bot after the nfs move
15:39 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s)
14:22 kart_: T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki)
14:21 moritzm: repooled restbase1006
14:09 moritzm: depooled restbase1006 for kernel update
14:05 moritzm: repooled restbase1005
13:54 moritzm: depooled restbase1005 for kernel update
13:47 moritzm: repooled restbase1004
13:35 moritzm: depooled restbase1004 for kernel update
13:34 moritzm: repooled restbase1003
13:16 moritzm: depooled restbase1003 for kernel update
13:12 moritzm: repooled restbase1002
12:56 moritzm: depooled restbase1002 for kernel update
11:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s)
10:39 moritzm: updated kernel on restbase1001 to latest 3.19
10:37 _joe_: manually removed crontab from mw1152, erroneously created by puppet
10:11 _joe_: reimaging mw1152
09:15 _joe_: preparing to reimage mw1152, disabling puppet, scheduling downtime.
06:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s)
05:15 legoktm: ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium
03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s)
02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s)
00:55 yurik: deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/
00:37 logmsgbot: tgr@tin Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s)
00:35 mutante: mw1135 temp. unresponsive - OOM killer killing hhvm
00:27 mutante: powercycling unresponsive mw1127

2015-10-27

23:53 logmsgbot: tgr@tin Started scap: Updating MediaViewer with r246112
23:13 ori: running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7)
23:12 gwicke: ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state
23:11 gwicke: rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies
23:06 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s)
23:01 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s)
22:39 mutante: powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248
21:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s)
21:50 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s)
21:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4
21:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s)
19:46 logmsgbot: twentyafterfour@tin Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s)
19:44 bd808: Ran sync-common on mw2187 to rebuild l10n caches
19:38 ejegg: enabled donation queue consumer
19:15 logmsgbot: twentyafterfour@tin Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch
19:09 awight: updated DjangoBannerStats from 57a0392b3f43b65050b01a0465e120ed609a769e to 12e819a04a40ee6fab5dd55fcaf072661df31106
17:56 paravoid: updating jessie debian-installer to 20150422+deb8u2
17:50 Jeff_Green: add fundraising-banner-logger hosts to icinga/nsca
17:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s)
17:37 ejegg: disabled donation queue consumer to build up queue for benchmark
17:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s)
16:07 Jeff_Green: round of fundraising OS updates, occasional icinga noise is expected
16:02 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language gerrit:249081 (duration: 00m 17s)
15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia gerrit:248910 (duration: 00m 17s)
15:15 godog: reenable puppet on graphite1001
15:01 kart_: TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki)
13:00 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s)
12:59 akosiaris: disabling puppet and bringing down OTRS service on mendelevium
12:21 jynus: Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables.
11:56 godog: reimage restbase-test2003
11:23 godog: cassandra OOM'd on restbase1007, restarting
10:57 godog: downtime restbase endpoints health for restbase1* while investigating
10:56 hashar: Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-(
10:50 hashar: stopping Jenkins due to an unclean state
10:11 jynus: disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs
09:55 godog: convert restbase-test2003 to cassandra multi-instance
09:08 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s)
05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s)
03:42 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00
02:24 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s)
01:02 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s)
00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s)
00:11 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s)

2015-10-26

23:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248371/ (duration: 00m 20s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/extension-list-labs: https://gerrit.wikimedia.org/r/#/c/248632/ (duration: 00m 17s)
23:19 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/vecwiki.png: https://gerrit.wikimedia.org/r/#/c/248633/ (duration: 00m 18s)
23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248478/ (duration: 00m 17s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
23:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248871/3 (duration: 00m 18s)
22:39 ejegg: updated payments from 7fd1e880b6a45d79fc305b39f3fee2f324324136 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
21:48 cmjohnson1: powering off wdqs1001 to update idrac settings
21:33 ejegg: update payments from 71d2d927f4efac1a639aaf7627c765f48d1b129c to 7fd1e880b6a45d79fc305b39f3fee2f324324136
20:50 subbu: deployed parsoid version 660c59a9
20:21 ebernhardson: started copy of eqiad elasticsearch indices to noeblium
20:19 YuviPanda: stress test on nobelium complete, CPU temperature didn't go above 65C
20:13 paravoid: deactivating ulsfo<->NTT BGP peering due to upcoming network migration
20:05 YuviPanda: running stress on nobelium
19:36 cmjohnson1: swapped bad disk on db1030
15:44 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s)
15:31 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s)
15:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s)
15:06 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s)
15:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s)
14:58 bblack: repooling cp1059 varnish mobile frontend (wiped)
14:18 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s)
14:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s)
13:32 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s)
13:27 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s)
09:42 dcausse: deleting unused elasticsearch indices in eqiad (T112863)
09:20 _joe_: restarting etcd on conf1001
09:16 jynus: rebooting and installing jessie on db2060-db2070
05:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)

2015-10-25

05:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 25 05:36:33 UTC 2015 (duration 36m 32s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-25 02:28:08+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 59s)

2015-10-24

19:34 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium
06:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s)
05:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s)
05:11 hoo: Set an email address for user "Ymnes", after request. Confirmed by several, including.
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00
02:33 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s)
00:21 ebernhardson: started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads

2015-10-23

23:31 Krinkle: mwscript deleteEqualMessages.php --wiki hewikisource
23:31 Krinkle: mwscript deleteEqualMessages.php --wiki ukwiki
23:09 hoo: Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer
22:23 robh: powercycled nobelium
21:24 mutante: nobelium - powercycled, no console output
20:04 jynus: rename user job had created lag on almost all enwiki dbs, things should be better now
19:08 urandom: starting nodetool cleanup on restbase-test2001-b
19:07 urandom: starting nodetool cleanup on restbase-test2001-a
18:16 godog: bounce grafana-server on krypton
17:58 YuviPanda: moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch
17:57 godog: delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries
17:53 YuviPanda: stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856
17:13 akosiaris: enable_notifications=0 in neon's icinga for a few mins while the storm dies down
17:04 cwd: updated payments from 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c to 71d2d927f4efac1a639aaf7627c765f48d1b129c
16:47 godog: stop puppet on graphite1001 and graphite-index cron, suspected root cause
16:26 godog: uwsgi timing out while serving requests, bounce also carbon daemons
16:20 godog: bounce uwsgi for graphite-web on graphite1001
16:14 urandom: starting rebuild of restbase-test2002-b
15:42 jynus: copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org
15:42 urandom: bouncing restbase-test2002-a Just To See
15:06 urandom: running nodetool cleanup on restbase-test2003
14:47 godog: remove outdated cassandra metrics from graphite2001
14:27 godog: remove restbase-test2001 restbase-test2002 cassandra metrics
14:12 mobrovac: restbase rolling-restarting after applying aab840f
13:55 jynus: Rebooting and installing jessie on db2055-db2059
13:45 godog: remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
13:34 godog: remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
11:51 godog: roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313
10:27 jynus: retrying schema change on EventLogging (db1046) after failure
10:21 jynus: End of online schema change to geo_tags; all wikis on dblist have been updated
09:34 godog: reimage restbase-test2002.codfw.wmnet
09:23 jynus: restarting schema change on geo_tags for all wikis
09:17 mobrovac: restbase rolling-restart after config changes
06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s)
06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s)
05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s)
03:15 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s)
00:10 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s)

2015-10-22

23:58 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s)
23:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s)
23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
23:12 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
22:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s)
22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
21:11 logmsgbot: reedy@tin Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s)
21:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s)
21:05 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s)
20:39 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s)
20:38 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s)
20:26 bd808: Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview
20:12 bd808: Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover)
19:53 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner: Deploying ZeroBanner T116309 patch 248116 (duration: 00m 18s)
19:28 bd808: Forced ELK Elasticsearch to allocate replica of logstash-2015.10.22 shard 0 on logstash1004
18:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.27.0-wmf.3
16:48 bd808: Restarted Elasticsearch on logstash1004
16:39 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: T116134: Set CX JWT token (duration: 00m 17s)
16:19 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f690589: Increase abusefilter emergency disable threshold on MediaWiki.org (duration: 00m 17s)
15:30 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf/highlight.php: SWAT: Remove urlencode from phabricator links gerrit:248049 (duration: 00m 17s)
15:13 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf: SWAT: noc: change Gitblit links to Diffusion gerrit:248027 (duration: 00m 17s)
15:11 moritzm: uploaded openjdk-8 8u66-b17 for jessie-wikimedia to carbon
14:46 jynus: performing schema change on eventlogging database on db1046
14:41 mutante: mw1083 - Error: Could not run Puppet configuration client: Read-only file system
14:25 jynus: setting thread_pool_size to 32 dynamically on all MariaDB hosts
14:01 jynus: performing schema change on officewiki-flow (s3)
13:19 jynus: performing schema change on x1-master (flowdb)
12:31 jynus: Rolling schema change for GeoData on all wikis (geo_tags)
11:35 mobrovac: restbase deployed 2bc05f40
07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 00m 17s)
07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 07m 33s)
06:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 22 06:17:39 UTC 2015 (duration 17m 38s)
06:14 AaronSchulz: Restarted hhvm on mw1011, it was stuck doing nothing at 100 cpu
05:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksUpdate.php: fe323f9b68bbb (duration: 00m 17s)
03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-22 03:05:21+00:00
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 52s)
02:37 AaronSchulz: Started running 5 threads of enwiki refreshLinks jobs on tin
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-22 02:35:41+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 44s)
01:10 logmsgbot: krenair@mira Synchronized README: testing sync from mira (duration: 00m 17s)
01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
01:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
00:56 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
00:55 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 17s)
00:49 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
00:20 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibf752b832: $wgMathCheckFiles = false (duration: 00m 18s)
00:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
00:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)

2015-10-21

23:33 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 19s)
23:32 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 18s)
23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Provide category suggestions from the right wiki (duration: 00m 17s)
23:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Rearrange messages to always display license name (duration: 00m 18s)
23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throtle exception for eswiki (duration: 00m 18s)
23:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
23:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
21:10 ebernhardson: starting copy of elasticsearch eqiad indices to codfw
20:46 ebernhardson|lch: cancel copying elasticsearch eqiad to labsearch, looks to be writing to wrong disks and will fill up
20:16 AaronSchulz: Started running 8 threads of commonswiki refreshlinks jobs on terbium
20:14 ebernhardson|lch: starting copy of elasticsearch indices from eqiad cluster to labsearch cluster
18:42 ebernhardson: initializing elasticsearch index mapping for all wikis in the codfw and labsearch ES clusters
18:12 ejegg|mtg: updated crm from 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435 to f2fa7b942625b34ede520e11f20e7e0835ecb17d
16:53 hoo: Attached WeiaR@enwiki to the global account of the same name. T115699
15:50 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
15:49 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
15:38 akosiaris: depooled mw1083
15:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: touch and re-sync InitialiseSettings.php to bust cache (duration: 00m 17s)
15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:18 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 17s)
15:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 19s)
15:00 aude: rsync failed on mw1083: "failed to set times on "/srv/mediawiki/wmf-config": Read-only file system"
14:49 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on test.wikidata (duration: 00m 18s)
14:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: enabled geodata on beta wikidata (duration: 00m 18s)
14:08 jynus: applying schema change to testwikidatawiki (s3) and wikidatawiki(s5)
13:56 jynus: performing schema change on testwiki.geo_tags
13:35 jynus: stopping and fixing replication on labsdb1004 (not in production)
08:40 mobrovac: restbase deployment of 3006b77e Done
08:32 mobrovac: restbase deploying 3006b77e
07:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 21 07:36:04 UTC 2015 (duration 36m 3s)
03:13 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-21 03:13:42+00:00
03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 24s)
02:42 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-21 02:42:44+00:00
02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 05s)
01:12 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: revert (duration: 00m 18s)
01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
00:29 logmsgbot: legoktm wikidata'd everything
00:22 logmsgbot: hoo@tin Synchronized dblists/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 17s)
00:19 logmsgbot: hoo@tin Synchronized wmf-config/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 19s)

2015-10-20

23:54 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (duration: 00m 18s)
23:47 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (testwikidata) (duration: 00m 18s)
23:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEventsHooks.php: https://gerrit.wikimedia.org/r/#/c/247650/ (duration: 00m 17s)
23:26 logmsgbot: hoo@tin Synchronized wmf-config/: Revert "Temporarily disable "item-merge" right on Wikidata" (duration: 00m 18s)
23:25 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred: 2a1e1d7dd88a62aba9 (duration: 00m 17s)
23:22 logmsgbot: hoo@tin Synchronized wmf-config/: Bump the cache epoch for (test)wikidata (duration: 00m 18s)
23:20 logmsgbot: hoo@tin Finished scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies (duration: 48m 44s)
23:16 mutante: mw1232: restarted hhvm
22:31 logmsgbot: hoo@tin Started scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies
21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 17s)
21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 18s)
20:23 gwicke: reverted restbase deploy on restbase1001 to a4c55e40
20:15 mutante: starting restbase on restbase1001
20:09 mutante: errors argon' on argon
20:04 ottomata: uninstalling hadoop packages on analytics1017
19:46 ori: previous syncs was of I46200d4edb3: Revert "Revert "Revert "Enable config for all three search clusters, but only write to eqiad"""
19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/: (no message) (duration: 00m 18s)
18:25 mutante: re-enabling icinga notifications for icinga (neon) itself that were disabled for some reason though all OK
16:50 mutante: temp. disabled ircecho / neon puppet
16:50 jynus: enabling query profiling on a sample of queries on db1072
16:27 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Revert Temporarily increase redis logging to debug (duration: 00m 18s)
16:25 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily increase redis logging to debug (duration: 00m 17s)
16:21 aude: re-populated sites table on metawiki, mediawikiwiki and specieswiki with https protocol links
16:13 cwdent: updated fundraising crm from 738e8c3f8079765841ed4c5f79ecf066c541c7b9 to 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435
15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part V (duration: 00m 18s)
15:48 godog: bump netdev_max_backlog to 10000 on graphite1001, T101141
15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part IV (duration: 00m 17s)
15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part III (duration: 00m 17s)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part II (duration: 00m 17s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part I (duration: 00m 17s)
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation suggestion in all Wikipedia gerrit:247515 (duration: 00m 17s)
15:23 awight: update fundraising CRM from 6ad0ad090c23ee41003138a6676131abf70c72f4 to 738e8c3f8079765841ed4c5f79ecf066c541c7b9
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch graphoid to the local restbase proxy gerrit:247494 (duration: 00m 17s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part 2 (duration: 00m 17s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part I (duration: 00m 18s)
14:33 cmjohnson1: removing tele2(patchid 2953) from dmarc panel @eqiad
14:12 ottomata: deployed varnishreqstats diamond collector to remaining varnish caches
13:48 jynus: backing up and renaming user_daily_contribs table from all wikis as a previous step for its deletion
12:44 kart_: Update cxserver to 6452b68
12:01 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Temporarily disallow item-merge until T115892 is resolved (duration: 00m 19s)
11:16 mobrovac: mathoid deploying 8e1a3327
07:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 20 07:23:56 UTC 2015 (duration 23m 55s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-20 03:14:15+00:00
03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 16s)
02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Bring phase0 and phase1 inline with phase2 (duration: 00m 18s)
02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Elastica/: Bring phase0 and phase1 inline with phase2 (duration: 00m 21s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-20 02:43:53+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 20s)

2015-10-19

23:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: testwiki Graphoid to restbase (duration: 00m 17s)
23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/: resync after touching InitialiseSettings.php to bust caches (duration: 00m 18s)
23:12 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Revert multidc cirrussearch config, seeing unexplained errors on commonswiki (duration: 00m 18s)
23:11 mutante: restarted hhvm on mw1231
23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throttle exception for dewiki (duration: 00m 17s)
23:08 mutante: restarted apache on mw1231
23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Disable cirrus suggester AB test (duration: 00m 17s)
23:05 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Bump sampling rate of common terms test from 1:1000 to 1:200 (duration: 00m 17s)
23:01 logmsgbot: ebernhardson@tin Synchronized tests/: noop sync mediawiki-config test dir (duration: 00m 17s)
22:42 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable cirrusearch multi cluster configuration, only write to eqiad (duration: 00m 18s)
22:40 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Handle ElasticaWrite job failures internally (duration: 00m 18s)
22:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Deploy multi-dc cirrusearch code for CirrusSearch extension (duration: 00m 18s)
22:28 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Elastica/: Deploy multi-dc cirrusearch code for Elastica extension (duration: 00m 17s)
22:20 ebernhardson: sync-common on mw1017 to pre-test cirrussearch multi-dc deployment
22:17 paravoid: salt-run deluser --delete-home gmetric; delgroup systemusers
20.23 subbu: cherrypick deploy for parsoid completed: b317f33f and 60a82ae0 cherrypicked from parsoid master
15:42 logmsgbot: anomie@tin Started scap: SWAT: Add a change tag to cross-wiki uploads gerrit:246701
15:39 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247255 (duration: 00m 18s)
15:30 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247256 (duration: 00m 18s)
15:25 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247268 (duration: 00m 18s)
15:24 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247267 (duration: 00m 17s)
15:12 urandom: deploying a4c55e40 to RESTBase
15:11 joal: Scheduling icinga downtime for CQL checks on aqs while heavily loading data - joal (me) babysites the jobs - 1 day downtime, will reiterate tomorrow if needed
15:09 ottomata: enabling varnish reqstats diamond collector on all upload caches
15:09 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
15:07 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
14:52 akosiaris: disabled puppet on maps-test200{1,2,4}. Debugging cassandra multi-instance setup aftermath. Not to be enabled
14:34 urandom: canary deploy (a4c55e40) to restbase1001.eqiad
12:47 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl-eng_0.1.0~r20599-1
12:22 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl_0.1.0-1
06:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 19 06:19:09 UTC 2015 (duration 19m 8s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-19 03:04:49+00:00
02:59 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 40s)
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-19 02:36:53+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 15s)

2015-10-18

10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-mlt-ara_0.1.0~r57554-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-is-sv_0.1.0~r56030-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-ro_0.7.3~r57551-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-it_0.1.0~r51165-1
06:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 18 06:11:37 UTC 2015 (duration 11m 36s)
02:57 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-18 02:57:48+00:00
02:52 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-18 02:29:50+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 25s)

2015-10-17

20:02 godog: powercycle analytics1034, no console no ssh
14:05 godog: reboot krypton, unable to ssh and no console (VM) iowait through the roof
06:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 17 06:27:04 UTC 2015 (duration 27m 3s)
03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-17 03:02:58+00:00
02:58 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-17 02:35:14+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 09s)

2015-10-16

20:52 hashar: Restarting Jenkins to remove potential dead locks before the week-end
20:48 ejegg: updated payments from 33b3bd6bee11b3cc9de1570584a23354d0b6525f to 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c
19:29 ori: deleted /var/lib/carbon/whisper/MediaWiki/MediaWiki on graphite1001 & graphite2001 per tgr's request
16:14 awight: Update paymentswiki logging config per T107918
15:52 awight: Fundraising: remove FR fraud exception
13:18 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I14ecd0ae87: Turn off UserDailyContribs extension (duration: 00m 18s)
06:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 16 06:25:56 UTC 2015 (duration 25m 55s)
03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-16 03:11:14+00:00
03:06 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-16 02:43:42+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 38s)
02:27 jynus: repairing table and restarting replication on s7 from labsdb1002
02:23 jynus: repairing table and restarting replication on s5 from dbstore1002 (non-production host)
01:36 Jamesofur: reset email address for User:INeverCry after identify verification

2015-10-15

22:09 greg-g: don't worry, _joe_ was around and we approved Roan's last deploy as an exception :)
22:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Back out categories in sidebar feature (duration: 00m 20s)
20:30 _joe_: rebooting ms-be1011
19:41 awight: update crm from ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1 to a4e74f6f38e6cff16ecf79d28d6e4de9499a5017
19:08 cwdent: updated payments from bdec3220030a396e2a447763e40b940a332e2ab8 to 33b3bd6bee11b3cc9de1570584a23354d0b6525f
18:41 ejegg: updated refund queue name in IPN listener settings
18:13 ejegg: updated CiviCRM from 17dc351d92a8437c93b4a9fa2385840b3581dad6 to ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1
17:18 cwdent: updated payments from ba8f80ec7a858074cd3856a52f3758cf96571f67 to bdec3220030a396e2a447763e40b940a332e2ab8
09:54 _joe_: restarted gitblit, unresponsive
05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 15 05:41:25 UTC 2015 (duration 41m 24s)
02:51 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-15 02:51:27+00:00
02:49 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 05m 02s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-15 02:35:15+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 57s)

2015-10-14

23:46 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
22:26 akosiaris: restarted zotero on sca1002
21:54 akosiaris: restarted zotero on sca1001. complaining out of memory in logs
20:46 ejegg: updated civicrm from ba39f3181431c8409416e580dcf6e15ac5f96a21 to 17dc351d92a8437c93b4a9fa2385840b3581dad6
20:20 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
19:52 robh: reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
19:50 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
19:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
19:10 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
18:06 logmsgbot: demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
17:59 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
17:58 logmsgbot: demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target gerrit:246243 (duration: 00m 18s)
15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246225 (duration: 00m 17s)
15:14 ori: restarted navtiming and statsv services on hafnium
15:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246226 (duration: 00m 18s)
15:05 Krinkle: Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
14:24 ori: cleaned up tessera leftovers on graphite1001
14:15 mutante|1way: mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
13:30 ottomata: kafka-preferred-replica election after kafka1012's broker restarted last night
13:12 ottomata: restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
12:55 hashar: ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25 (from labs instances)
12:32 logmsgbot: krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
12:07 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
03:07 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
00:19 ejegg: updated civicrm from c7af7634e75eb8702f5e16081a86ab86ce69c7c2 to ba39f3181431c8409416e580dcf6e15ac5f96a21

2015-10-13

23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default (duration: 01m 14s)
23:27 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Turn on cirrus common terms test (duration: 01m 15s)
23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Add information for common terms ab test (duration: 01m 15s)
23:21 Krenair: restarted apache on silver
23:19 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Flow: Make flow board descriptions editable again (duration: 01m 16s)
23:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Flow: Bump flow submodule in 1.27.0-wmf.3 (duration: 01m 15s)
23:10 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Update WME for common terms AB test (duration: 01m 14s)
23:08 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Update cirrus for common terms AB tes (duration: 01m 15s)
23:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn on logging of the "warning" channel (duration: 01m 13s)
23:00 ejegg: updated SmashPig from b5ff2a7d5f17aaaa33a169ca101cbea639769c90 to c431c8d77521270236c72532a50806b2e852cf7b
21:05 ottomata: reenabling puppet on cp1052
20:51 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
20:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/db: I0e5f2d3b2: Revert Enforce lagged-slave read-only mode on the DB layer (duration: 01m 14s)
20:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/autoload.php: (no message) (duration: 01m 14s)
20:40 akosiaris: restart gitblit (once more)
20:27 _joe_: rebooting mw1157, stuck in a kernel soft lockup
20:22 ori: locally hacked testwiki and mediawikiwiki to point to php-1.27.0-wmf.3 on mw1017
20:19 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
20:18 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
20:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
19:59 ejegg: updated civicrm f25fbe856b92373104985185db77311ea3a4d841 to c7af7634e75eb8702f5e16081a86ab86ce69c7c2
19:38 JohnFLewis: delete wikidata-l mailing list (archives still accessible)
19:30 logmsgbot: twentyafterfour@tin Finished scap: Full scap sync for 1.27.0-wmf.3 (duration: 31m 26s)
18:58 logmsgbot: twentyafterfour@tin Started scap: Full scap sync for 1.27.0-wmf.3
18:45 ottomata: FYI i have puppet disabled on cp1052 while I try to figure out a diamond+VSL+multiprocessing bug
18:11 mutante|1way: started salt on mw2083
18:09 mutante|1way: restarted gitblit
17:58 ori: testing ldap integration in grafana 2
17:54 urandom: performing deploy of a4c55e4 to restbase staging
17:42 urandom: performing canary deploy of a4c55e4 to restbase staging (xenon)
16:14 ostriches: created education program tables for srwiki, T110619
16:14 mobrovac: restbase deploying a01c62a6
16:08 ottomata: restarting diamond on cp1052 in gdb in attempt to figure out why vanrishreqstats segfaults...
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions for de, fa, fi, he, nn, pa, pl and te wikipedias gerrit:245862 (duration: 01m 13s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default" (duration: 01m 13s)
15:15 ori: grafana-test: Imported Grafana dashboards from ElasticSearch
14:12 bblack: note to self: we should migrate all our service to Java
14:11 bblack: restarted gitblit on antimony
13:16 bblack: repooling cp2017 (codfw upload)
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 13 02:32:17 UTC 2015 (duration 32m 16s)
02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-13 02:32:17+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 01s)
01:53 logmsgbot: faidon@tin Synchronized wmf-config/CommonSettings.php: unbreak BounceHandler (duration: 01m 14s)
00:51 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Reapply: Flow-occupy talk namespaces on sewikimedia (duration: 01m 13s)
00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/modules/flow-initialize.js: https://gerrit.wikimedia.org/r/#/c/245596/ (duration: 01m 13s)
00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244724/ (duration: 01m 13s)
00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245608/ (duration: 01m 13s)
00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 13s)
00:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245588/ and https://gerrit.wikimedia.org/r/#/c/245589/ (duration: 01m 13s)

2015-10-12

23:53 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s)
23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s)
23:35 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s)
23:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s)
23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s)
23:11 logmsgbot: krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s)
23:06 logmsgbot: krenair@tin Synchronized database lists: (no message) (duration: 01m 13s)
20:21 bearND: MobileApps deployed sha1 95293e5
19:09 Tim: on ruthenium installed iotop for stall investigation
16:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed gerrit:244737 (duration: 01m 13s)
16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 gerrit:245472 (duration: 01m 13s)
15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki gerrit:244953 (duration: 01m 14s)
15:39 logmsgbot: thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist gerrit:244435 (duration: 01m 12s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes gerrit:241079 (duration: 01m 14s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces gerrit:242096 (duration: 01m 13s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis gerrit:244165 (duration: 01m 13s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata gerrit:244591 (duration: 01m 14s)
15:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary gerrit:244649 (duration: 01m 12s)
14:54 dcausse: closing unused cirrus indices in eqiad (T112863)
14:28 cmjohnson: rebooting mw1154
12:10 yurik: deployed kartotherian & tilerator to maps-test200{1-4}
08:54 hashar: zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243
08:44 hashar: Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243
02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s)
02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s)

2015-10-11

04:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s)

2015-10-10

06:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s)
02:41 ejegg|afk: enabled fundraising banner campaigns
02:36 ejegg|afk: updated payments-wiki from 24d5be6886d34b3600031290c7f55ee84f3dcee2 to ba8f80ec7a858074cd3856a52f3758cf96571f67
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s)

2015-10-09

22:20 ejegg: took down Fundraising campaigns
19:33 logmsgbot: ori@tin Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s)
18:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s)
16:04 godog: rolling restart cassandra test cluster T95253
15:35 urandom: performing Cassandra cleanup on restbase-test2003.codfw
15:15 urandom: bouncing Cassandra on restbase-test2001-a
15:07 urandom: starting nodetool cleanup on restbase-test2002
14:37 jynus: more configuration testing (with puppet disabled) and several mysql restarts on db1022
13:53 hashar: Restarted Zuul, had a deadlocked job
11:50 godog: bounce mathoid on sca100[12], stray instance found not running firejail
11:22 mobrovac: restbase deploying aaee7c31
11:16 godog: force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/
11:00 jynus: restarting db1022's mysql (depooled) for configuration testing
09:17 jynus: deployed visual glitch fix to dbtree
09:11 akosiaris: poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
09:11 akosiaris: poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
08:20 ori: Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/
05:13 mutante: @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s)
01:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s)
01:18 RoanKattouw: Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh
01:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s)

2015-10-08

23:56 logmsgbot: catrope@tin Finished scap: SWAT (duration: 29m 10s)
23:51 yurik: reverted Kartotherian to HEAD^^ - the service wouldn't start
23:27 logmsgbot: catrope@tin Started scap: SWAT
23:06 yurik: deployed kartotherian
18:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2
16:45 mobrovac: mathoid deploying 110abaf
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) gerrit:244332 (duration: 00m 18s)
15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki gerrit:244142 (duration: 00m 17s)
14:53 paravoid: replacing cr1-codfw<->asw-a-codfw QSFPs
14:48 logmsgbot: reedy@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
14:13 jynus: performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002)
13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s)
13:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s)
12:09 godog: update facts on puppet compiler from palladium
11:38 _joe_: rebooting mc2002
10:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s)
10:37 _joe_: rebooting mw1008, in oom spiral
10:15 paravoid: salt rm /home/*/.ssh/authorized_keys
09:21 jynus: downtime for db1055 for maintenance (kernel update, mysql update, config update)
08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s)
07:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s)
07:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s)
06:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 8 06:15:16 UTC 2015 (duration 15m 15s)
05:56 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00
03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s)
02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s)
01:17 twentyafterfour: phabricator is now up and running again
01:09 twentyafterfour: buggy update, stopping apache2 on iridium.
01:02 twentyafterfour: finished phabricator update
00:51 mutante: mw1160 - was oom-killer (convert, mw job 31497)
00:50 twentyafterfour: phabricator maintenance/upgrade. Expect 10 minutes downtime
00:44 mutante: powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt)
00:08 logmsgbot: krenair@tin Synchronized w/static/apple-touch/commons.png: https://gerrit.wikimedia.org/r/#/c/243685/ (duration: 00m 17s)

2015-10-07

23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242842/ (duration: 00m 17s)
23:34 logmsgbot: ori@tin Synchronized wmf-config: I924d8e19e17: Make the redis cache configuration multi-DC-ready (T111575) (duration: 00m 17s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244177/ (duration: 00m 17s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244176/ (duration: 00m 17s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243837/ (duration: 00m 17s)
21:25 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Revert "Don't use nutcracker on wikitech" (duration: 00m 16s)
20:56 mutante: applied firewalling on IRCd server, rc bot still working fine, all public IRC ports as before
20:25 mutante: argon: installing package upgrades
20:24 mutante: resetting drac on argon
20:14 logmsgbot: ori@tin Synchronized wmf-config/session.php: Ie25c368a: Switch mw1017 to use DC-specific redis cluster names (duration: 00m 17s)
19:50 urandom: RESTBase deploy complete
19:41 urandom: doing full deploy of c20e6336 to RESTBase
19:29 urandom: canary deploy to restbase1001.eqiad complete
19:14 urandom: deploying c20e6336 to canary node restbase1001.eqiad
19:09 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.2
18:56 logmsgbot: twentyafterfour@tin Synchronized wmf-config/CommonSettings.php: fix undefined variable warning that has been spamming logs (duration: 00m 17s)
17:56 awight_: update fundraising crm from 7003cc38797848631d0c4d5f6ff68ab1d6118ad8 to f25fbe856b92373104985185db77311ea3a4d841
16:39 mutante: powercycling analytics1035
16:14 mobrovac: citoid deploying ec149fd5
16:02 logmsgbot: marktraceur@tin Synchronized wmf-config/: Adding new config variable for uploads to Commons (duration: 00m 17s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/MobileFrontend: SWAT (duration: 00m 17s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 18s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1051 after maintenance (duration: 00m 17s)
15:03 logmsgbot: krenair@tin Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 17s)
15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 18s)
14:59 mobrovac: AQS restarting restbase on aqs100x
13:00 godog: decomission restbase-test2001 and reimage
12:27 paravoid: salt-rm'ing /var/lib/apt/lists/ubuntu.wikimedia.org_ubuntu_dists_trusty_main_i18n_Translation-en%5fUS
10:55 godog: reenable puppet on restbase / maps-test / aqs
10:10 _joe_: depooling cp1059 from pybal, varnish
08:44 godog: disable puppet on restbase, maps, aqs before merging https://gerrit.wikimedia.org/r/#/c/243127
08:05 moritzm: installed spice security updates on labvirt*
06:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 7 06:01:01 UTC 2015 (duration 1m 0s)
05:25 ebernhardson: generating elasticsearch indices in codfw, should run ~3 hours
04:33 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Touch InitiialiSettings.php to force config regeneration (duration: 00m 18s)
04:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/: enable second es cluster in testwiki one more time (duration: 00m 18s)
04:15 ebernhardson: sync-common on mw2187.codfw.wmnet to fix localisation cache errors in exception.log
03:37 logmsgbot: ebernhardson@tin Synchronized wmf-config: redisable second cluster only on testwiki (duration: 00m 16s)
03:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Reenable second ES cluster on testwiki only (duration: 00m 18s)
03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-07 03:10:02+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 10m 18s)
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-07 02:37:32+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: Fix JS error (duration: 00m 18s)
01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: Fix JS error (duration: 00m 17s)
00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 17s)
00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)

2015-10-06

23:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: Revert multicluster config for testwiki (duration: 00m 18s)
23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable multicluster ES on testwiki (duration: 00m 17s)
22:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
22:40 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.2: Deploy https://gerrit.wikimedia.org/r/#/c/244066/ (duration: 01m 40s)
21:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
21:15 twentyafterfour: restarted phd in response to a phabricator setup issue
20:58 bblack: disabling puppet on caches for VCL testing
20:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
20:53 logmsgbot: twentyafterfour@tin Finished scap: sync wmf/1.27.0-wmf.1 (duration: 32m 13s)
20:29 mutante: service "ishmael" has been removed (T109777) - removed docroot on neon. tarball exists in /root just in case. code is on https://github.com/asher/ishmael
20:20 logmsgbot: twentyafterfour@tin Started scap: sync wmf/1.27.0-wmf.1
18:48 yuvipanda: fixing https://phabricator.wikimedia.org/T109216 on labstore1002
18:10 godog: upgrade videoscalers to ffmpeg2theora 0.29.0~git+20150813-2
15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Suggestion in af, gl, gu, mk, oc, sh and simplewiki gerrit:243919 (duration: 00m 18s)
11:33 jynus: performing schema change on db1051 enwiki revision
11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1051 for more maintenance (duration: 00m 17s)
10:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1051 after maintenance (duration: 00m 17s)
10:41 jynus: potential extra load on mediawiki recent changes and watchlist on enwiki, please report any slowdown
10:22 jynus: dropping temp recovered tables from db1051 to prepare for repool
09:50 _joe_: uploaded a new pybal package for jessie
06:19 paravoid: eqord is back up
04:24 paravoid: all waves to eqord down, probably related to RT#9619
02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-06 02:34:31+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 53s)

2015-10-05

23:53 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/243833/ (duration: 00m 17s)
23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 00m 17s)
23:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/243729/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242942/ (duration: 00m 17s)
21:16 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Take2: Deploying ZeroBanner 242661+243808 (duration: 00m 17s)
21:00 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Rolling back ZeroBanner 242661 (duration: 00m 18s)
20:55 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Deploying ZeroBanner 242661 (duration: 00m 17s)
18:38 mutante: restarted gitblit
17:44 gwicke: rolling restart of eqiad restbase nodes done
17:27 gwicke: rolling restart of restbase cluster to rule out driver issues causing the increased p99 read latency
17:19 robh: analytics1035 pegged out, ssh unresponsive and raid failures, and then fixed itself 5 minutes later
16:32 legoktm: running backPopulateRenameQueueLogs.php (gerrit:237169) on metawiki
16:05 hoo: Updated Wikidata's property suggester with data from today's json dump
15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.1/extensions/Flow: SWAT: Fix exception on board and topic history pages gerrit:243337 (duration: 00m 20s)
14:41 hashar: puppet-lint Jenkins job is now strict and will -1 on errors as well as warnings https://gerrit.wikimedia.org/r/#/c/243185/
10:25 godog: roll-restart cassandra in restbase eqiad
09:58 _joe_: installing the new HHVM package to appservers
09:50 _joe_: rebooted mw1153, soft lockup due to bnx2 failure
09:50 godog: roll-restart cassandra in restbase codfw
09:18 godog: roll-restart cassandra on restbase test cluster
09:15 godog: stop puppet on restbase and maps in preparation for https://gerrit.wikimedia.org/r/#/c/242896/1
08:07 _joe_: upgrading HHVM on all API appservers
05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 5 05:33:18 UTC 2015 (duration 33m 17s)
02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-05 02:33:57+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 25s)

2015-10-04

23:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/extension.json: 8c80ec1273: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation (duration: 00m 17s)
05:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 4 05:13:32 UTC 2015 (duration 13m 31s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-04 02:31:56+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 06s)

2015-10-03

15:33 jynus: stopping temporarily labsdb1004 mariadb to complete clone process
09:38 _joe_: rolling restarting all parsoids in eqiad
09:14 _joe_: restarting parsoid on wtp1021
05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 3 05:28:54 UTC 2015 (duration 28m 53s)
03:12 ori: done; graphite-web back up; url shortening will now work.
03:11 ori: shutting down graphite-web for brief sqlite database schema update
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-03 02:36:33+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)

2015-10-02

22:46 mutante: holmium: apt-get clean for a little disk space - /var/log/designate/designate-mdns.log is more than half the size of / - needs logrotate
21:33 Krinkle: mwscript cleanupRemovedModules.php --wiki zhwiktionary
21:21 Krinkle: mwscript cleanupRemovedModules.php --wiki testwikidatawiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki dewiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki nlwiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki test2wiki
21:06 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
18:39 ottomata: kafka preferred-replica-election
17:53 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I680f3fda66c5: Configure ResourceLoader-specific ObjectCache instance (duration: 00m 17s)
17:53 ottomata: rolling restart of all hadoop-yarn-nodemanagers to pick up python3 + spark fix
17:12 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I8318fe892: Configure MultiWriteBagOStuff for ResourceLoader (duration: 00m 17s)
16:29 ori: deployed grafana 8e92884bae (with backport of upstream fb9f9548829f2d4cecf35cda933700e5c2fa1bd6)
15:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/resources/src/mediawiki/mediawiki.js: I8029208: Don't clobber existing styles when adding more in IE9 (duration: 00m 17s)
14:37 jynus: providing extra grants to wiki db users for heartbeat monitoring
14:32 Coren: Change of NFS mount options in labs pushed - puppet may report failures to refresh the mounts (once) on instances; expected and harmless.
11:03 moritzm: installed rpcbind security updates on all Ubuntu servers which runs it (jessie was already updated, since the DSA was released earlier)
10:53 jynus: restarting HHVM on mw1130
08:45 hashar: restarting Nodepool to take in account changes made to the logging configuration https://gerrit.wikimedia.org/r/#/c/240986/
06:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 2 06:07:05 UTC 2015 (duration 7m 4s)
02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.1/extensions/ParsoidBatchAPI: stats (duration: 00m 17s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-02 02:41:33+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 04s)
00:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I21bb3f08e7f and follow-ups (duration: 00m 18s)
00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/vendor: Add wikimedia/relpath 1.0.3 (duration: 00m 21s)

2015-10-01

22:04 RoanKattouw: Running FlowFixLinks.php on all wikis
20:33 subbu: deployed parsoid version 62971510b
19:33 twentyafterfour: re-deployed robots.txt patch and restarted apache on iridium (to expand the phabricator robots.txt)
18:22 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: Ic1d802ee2: ResourceLoader: cache minified user and site modules (duration: 00m 17s)
18:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.1
16:40 ottomata: brought analytics1049 back into hadoop after missing a disk since sept. 25th
16:34 andrewbogott: test log
16:26 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
16:26 valhallasw`cloud: testing, testing
15:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/240049 (duration: 00m 17s)
15:56 cmjohnson1: db1051 replacing failed disk slot 6
15:51 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/modules/dashboard/styles/ext.cx.dashboard.less: Fix: Clicking on down arrow in language selector should trigger ULS (duration: 00m 17s)
15:48 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki (duration: 00m 17s)
15:47 cmjohnson1: analytics1049 replacing failed disk /dev/sdi at slot 7
15:46 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 18s)
15:45 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 17s)
15:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 17s)
15:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 18s)
15:33 cmjohnson1: replacing failed disk ms-be1012 /dev/sdf slot 5
15:09 matt_flaschen: Did final run of convertAllLqtPages.php on sewikimedia immediately before freezing LQT
15:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on se.wikimedia (duration: 00m 18s)
14:58 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
14:19 andrewbogott: log test
14:17 andrewbogott: testing the log
14:07 _joe_: installing the new hhvm package to all the canaries
13:41 hashar: Am I logging?
13:32 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm7
13:03 bblack: repooling cp1046 (eqiad mobile) with caches wiped clean just before
12:24 akosiaris: disable SessionRemoteIPcheck on mendelevium's OTRS installation for checking
12:18 logmsgbot: aude@tin Finished scap: Put Wikidata extension back on wmf/1.27.0-wmf.1 (duration: 30m 46s)
12:09 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
11:47 logmsgbot: aude@tin Started scap: Put Wikidata extension back on wmf/1.27.0-wmf.1
10:35 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.1/extensions/: Use 1.27.0-wmf.1 for Wikidata again after fixing T114290 (duration: 01m 07s)
10:20 akosiaris: uploaded php5_5.3.10-1ubuntu3.20+wmf1 on apt.wikimedia.org precise-wikimedia
09:51 godog: reboot praseodymium to test cassandra systemd unit
09:47 godog: stop puppet on restbase* in preparation for https://gerrit.wikimedia.org/r/242548
08:59 akosiaris: disabling puppet on maps-test200* in expectance of https://gerrit.wikimedia.org/r/242548
06:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 1 06:56:51 UTC 2015 (duration 56m 50s)
03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-01 03:21:39+00:00
03:14 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 38s)
02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-10-01 02:47:31+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 54s)
01:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
01:29 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
00:44 mutante: applying puppet fix on fermium (illegal byte sequence in utf-8) as in T114289#1691038 but for other languages

2015-09-30

23:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
22:55 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s)
22:44 ottomata: perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif
22:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
22:12 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
21:36 cwdent: updated payments from bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11 to 24d5be6886d34b3600031290c7f55ee84f3dcee2
21:27 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s)
20:43 cscott: updated Parsoid to version 39c60c67
20:40 mutante: fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates
19:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s)
18:51 jynus: stopping replication on s2, s3 and s7 for dbstore1001
18:50 ottomata: restarted eventlogging with blacklist=^Analytics$
18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
17:39 ottomata: restarting eventlogging with 12 client side processors
16:28 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s)
16:20 logmsgbot: krenair@tin Finished scap: swat (duration: 53m 33s)
15:27 logmsgbot: krenair@tin Started scap: swat
13:53 moritzm: rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier)
12:34 moritzm: added debdeploy 0.0.8 to carbon
05:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s)
03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s)
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s)
01:02 logmsgbot: ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s)

2015-09-29

23:59 ottomata: restarting eventlogging so that processors use etcd to pick up shared token with which to consistently hash IPs
23:46 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo/: SWAT (duration: 00m 17s)
23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on mediawikiwiki (duration: 00m 16s)
23:33 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 18s)
23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 17s)
23:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 17s)
22:50 ejegg: updated payments-wiki from 3b0915a51a0fd567bdf22f3d4e17548a83e735d8 to bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11
22:47 ejegg: updated SmashPig from bf302444eae8236734fd43883b06c7b2512b1532 to 513ec01123e6dbb97b00888a3610a7c5ec24a63b
22:38 twentyafterfour: finished deployment of 1.27.0-wmf.1 to group0
22:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
22:34 logmsgbot: twentyafterfour@tin Finished scap: php-1.27.0-wmf.1/extensions/ (duration: 04m 29s)
22:30 twentyafterfour: 22:27:17 sync-dir failed: <ValueError> /srv/mediawiki-staging/php-1.27.0-wmf.1/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening <?php tag
22:30 chasemp: es-tool unban-node elastic1031
22:29 logmsgbot: twentyafterfour@tin Started scap: php-1.27.0-wmf.1/extensions/
22:23 chasemp: unbanning elastic1006 for shard population
22:05 ori: 22:03:51 Synchronized php-1.26wmf24/extensions/CentralNotice: 30bdfcb386: Updated mediawiki/core Project: mediawiki/extensions/CentralNotice 6bd658e155a02edb4cc506bc3494a3f4699d3e94 (duration: 00m 17s)
19:34 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.1 for validation on testwiki (duration: 31m 49s)
19:30 bd808: Updated iegreview.wikimedia.org to c3ac5e6 (Update to Twig 1.20.0) and applied latest schema changes
19:02 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.1 for validation on testwiki
18:11 cmjohnson1: shutting down elastic1031 to relocate rack/row
18:09 cmjohnson1: shutting down elastic1006 to relocate row/rack
17:46 mutante: size of conntrack table on iron might be increased due to test scans
17:32 ejegg: updated appeal template setting on payments-wiki
16:30 cmjohnson1: swapping failed disk db1050
16:14 subbu: reverted configuration hotfix from yesterday's Parsoid deploy (re-enabled use of Parsoid batching API)
15:58 bblack: ending VCL tests on cp1053
15:52 bblack: ending VCL tests on cp1065, starting on cp1053 instead
15:39 bblack: cp1065: live-testing some VCL patches, puppet disabled, etc...
15:19 chasemp: ban elastic1031 for T112559
15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Set TransitionDefault true for the English Wikipedia gerrit:242040 (duration: 00m 17s)
14:08 paravoid: repooling eqiad; 24h codfw test window is over
10:49 godog: bounce pybal on lvs2003 / lvs2006
09:13 paravoid: powercycling pybal-test2003
08:51 jynus: starting cloning of labsdb1005 (Tools DB), minimal disruption is expected
07:03 twentyafterfour: restarted apache2 and phd or iridium to get phabricator back into the correct state
06:58 twentyafterfour: checked out correct phabricator release tag on iridium. Something, somewhere, had reverted everything to an old deployment.
05:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 29 05:01:30 UTC 2015 (duration 1m 29s)
02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-29 02:34:55+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 57s)
02:25 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually
02:24 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20*
01:55 robh: upload-lb.codfw.wikimedia.org_ipv6 page flap
01:43 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s)
01:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s)
01:13 robh: didnt powercycle analytics1035 yet, it recovered on its own.
01:13 robh: powercycling analytics1035, seems oom, cannot login via ssh or serial console
00:56 logmsgbot: tstarling@tin Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s)

2015-09-28

23:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s)
23:43 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s)
20:51 bearND: MobileApps deployed sha1 9df72ec
20:42 subbu: deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors
20:26 yuvipanda: depooled cp2017 from pybal config too
20:22 yuvipanda: depooled cp2017 since it's down
19:02 chasemp: powercycle iridium via console as it's unresponsive
18:01 awight: update crm from 190f689ff7aec7fecefdf5af501293685c55e041 to 7003cc38797848631d0c4d5f6ff68ab1d6118ad8
15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s)
15:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s)
15:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s)
14:00 paravoid: running failover eqiad->codfw test for all frontend traffic
06:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s)
05:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s)
02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s)

2015-09-27

23:17 logmsgbot: ori@tin Synchronized php-1.26wmf24/./resources/src/mediawiki.toolbar/toolbar.less: I94ced06178: mediawiki.toolbar: temporary workaround for T113868 (duration: 00m 17s)
16:51 Krenair: ran sync-common on snapshot1001 to bring it up to date
16:43 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf21
16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf20
16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf19
15:50 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/241354/ - fix AbuseFilter block durations (duration: 00m 18s)
10:05 jynus: leaving innodb compression tests on es2005 running (could affect lag on that host)
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 05s)

2015-09-26

19:04 hashar: restarting Jenkins. Just in case :-D
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 48s)

2015-09-25

23:54 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s)
22:16 mutante: sodium - shutting down
21:53 ori: Armed new servicedeploy_rsa on mira
21:51 ori: Armed new servicedeploy_rsa on tin
20:45 urandom: starting a Cassandra repair on xeon (nodetool repair -pr)
19:30 mutante: restarted apache on gallium
19:07 bblack: configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11
18:04 jynus: performing schema change on m5-master "nova"
17:26 jynus: restarting and upgrading db1051 mysql (depooled)
17:20 godog: reboot ms-be1005, xfs
16:22 cwdent: updated payments from dc78ff5157b59a8f475dc86194a1059c2d6b2fad to 3b0915a51a0fd567bdf22f3d4e17548a83e735d8
16:17 akosiaris: duplicating database otrs to otrsupgradetest for testing the upgrade procedure
16:16 cmjohnson1: cp1046 stopped icinga checks for hardware troubleshooting
14:43 moritzm: installed rpcbind and apport security updates on various servers
14:36 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s)
14:22 chasemp: 'sudo -u hdfs hdfs haadmin -transitionToActive analytics1001-eqiad-wmnet' per otto on analytics1001
14:03 urandom: starting a Cassandra repair on restbase1006 (nodetool repair -pr -dc eqiad)
13:58 hashar: nodepool back in operations
13:44 moritzm: restarted saltmaster on palladium
13:12 mobrovac: restbase deploying e42bf0fc
13:12 moritzm: added debdeploy 0.0.7 to carbon
13:10 godog: enable puppet on restbase in production
12:54 andrewbogott: restarted rabbitmq-server on labcontrol1001
12:52 godog: stop puppet on restbase in production -- config deployment
12:46 hashar: stopping nodepool to clear out left over mysql connections
12:21 hashar: Nodepool is dead, or at least not adding new slaves anymore
12:16 hashar: Seems Zuul/Jenkins is in trouble somehow :-/
11:42 andrewbogott: upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59
04:58 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s)
02:57 Krinkle: sync-common failed on mw1010.eqiad.wmnet
02:53 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/DonationInterface: 381faf5 (duration: 00m 21s)
02:33 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 11m 45s)
01:44 Krinkle: mwscript deleteEqualMessages.php --wiki roa_tarawiki
01:42 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiquote
01:42 Krinkle: mwscript deleteEqualMessages.php --wiki alswiki
01:28 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/WikimediaEvents: I5608f8ffd1c - Fix trailing comma (duration: 00m 17s)
00:10 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/EventLogging/modules/ext.eventLogging.core.js: Increase maxUrlSize to 2000 (duration: 00m 17s)

2015-09-24

23:02 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240880/ (duration: 00m 17s)
21:18 ejegg: updated SmashPig from d1baa32267eaad7d69b47c657f4853eb306fad6b to bf302444eae8236734fd43883b06c7b2512b1532
21:15 ejegg: updated payments-wiki from 8428499feb8760d63faf681d53995697a2ba0fa7 to dc78ff5157b59a8f475dc86194a1059c2d6b2fad
19:57 logmsgbot: ori@tin Synchronized php-1.26wmf24/extensions/ContentTranslation: d079d5dd71: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation 8559ee614975f25b71a732ca0fb1bb6d489c9d33 (duration: 00m 18s)
19:35 bblack: depooled cp1046 from confd, committed pybal depool for LVS as well
19:34 chasemp: changing labs route on cr1 and cr2 from 10.68.16.0/22 to 10.68.16.0/21 which matches references, fw setting and manifests/network.pp
18:54 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow/: Debugging for FlowFixLinks.php (duration: 00m 20s)
18:21 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf24
18:20 legoktm: moved oauthadmin group from User:Yuvipanda@metawiki to User:YuviPanda@metawiki
18:19 godog: restart restbase on restbase1005
18:18 godog: restart restbase on restbase1004
18:17 ejegg: reenabling CRM jenkins jobs
18:16 godog: restart restbase on restbase1003
18:08 ejegg: updated civicrm from 9fa38d06a75363a8009bce7ced190e39c75b68bc to 190f689ff7aec7fecefdf5af501293685c55e041
18:06 paravoid: depooling cp1046, stability issues
18:05 ejegg|afk: disabled CRM jenkins jobs
18:00 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 17s)
17:59 ori: Merged Apache config change Ia095457fb. It will refresh the Apache service as it rolls out, causing elevated 503s for the next 20 minutes.
17:53 godog: rolling restart restbase in eqiad
17:35 chasemp: powercycling cp1046 at mgmt as I can't ssh in and it seems like it should be up
17:26 godog: bounce restbase on restbase1002, apply new datacenter config
17:10 _joe_: cleaning up /tmp on mw1152
17:09 cmjohnson1: powering down for the last time es1001 - es1010
16:17 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240727 (duration: 00m 26s)
16:01 robh: nothing on puppet swat window, easiest swat ever.
15:46 logmsgbot: thcipriani@tin Synchronized php-1.26wmf24/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240711 (duration: 00m 26s)
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suggestions in ca, en, es, fr, it, ja, tr, ru, zh gerrit:240638 (duration: 00m 17s)
14:37 paravoid: repooling codfw
12:54 bblack: restarting varnish daemons on second half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
12:50 bblack: restarting varnishd instances on text, mobile, upload clusters for package upgrade (slow salt, no parallelism, ~5m spacing - FE cache loss, BE cache stays, should take ~9h)
12:05 moritzm: installed rpcbind security updates on eeden, baham, radon, maerlant, rhenium
11:56 bblack: restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
11:36 bblack: reinstall lvs300[12] to jessie - T96375
11:21 akosiaris: killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space
11:14 bblack: stopping pybal on lvs300[12]; lvs300[34] taking over
11:07 bblack: upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk)
09:40 jynus: performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.)
08:39 jynus: restarted HHVM @ mw1056, 1104, 1122
05:33 yuvipanda: deleted logstash indexes for 08/27 and 28 too
05:31 yuvipanda: deleted indexes for 08/14, 15, 25, 26 on logstash
03:59 yuvipanda: restarting elasticsearch in logstash1001-3
03:53 yuvipanda: restarted es on logstash1004-6
03:02 yuvipanda: jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles
02:51 yuvipanda: restarted logstash on logstash1002
02:41 yuvipanda: gmond at 100% again, killing it and stopping puppet again
02:40 yuvipanda: re-enabling and running puppet on hafnium to see what it's bringing up
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
02:23 yuvipanda: kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU
02:16 Krinkle: Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z.
02:14 Krinkle: Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago)
01:53 mutante: started logstash on logstash1002 again
01:35 mutante: bast1001: unmounting /srv/home_pmtpa (backup on bacula)
01:34 mutante: removing subversion packages from bast1001
01:15 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s)
00:19 jynus: restarted replication on db1051
00:17 ori: restarting tcpircbot on neon
00:16 mutante: started logstash on logstash1002
00:08 bblack: varnish package on carbon for jessie updated to 3.0.6plus-wm8

2015-09-23

23:17 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s)
23:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s)
22:53 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s)
22:39 paravoid: cr1-codfw RE switchover(s)
22:35 logmsgbot: yurik@tin Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s)
21:13 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s)
20:46 subbu: deployed parsoid 6619409e
19:58 urandom: starting Cassandra on restbase2006
19:56 urandom: enabling puppet, and forcing a run on restbase2006
19:52 urandom: starting Cassandra on restbase2005
19:50 urandom: enabling puppet, and forcing a run on restbase2005
19:48 urandom: starting Cassandra on restbase2004
19:45 urandom: enabling puppet, and forcing a run on restbase2004
18:55 twentyafterfour: snapshot1001.eqiad.wmnet returned [12]: rsync: write failed on "/srv/mediawiki/wikiversions.cdb": No space left on device
18:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
18:01 mobrovac: restbase deploying f65313ed
17:49 urandom: starting Cassandra on restbase2003
17:45 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 25s)
17:35 urandom: enabling, and forcing puppet run on restbase2003
17:27 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
17:20 logmsgbot: ori@tin scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 44m 24s)
17:11 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/page/Article.php: (no message) (duration: 00m 13s)
17:11 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page/Article.php: (no message) (duration: 00m 17s)
16:35 logmsgbot: ori@tin Started scap: (no message)
16:35 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page: I952068d2d: Reduced the DOS potential of 404 page floods (duration: 00m 12s)
16:22 godog: start cassandra on restbase2002
16:05 godog: deploy restbase daacf4d on restbase2*
15:34 logmsgbot: demon@tin Synchronized php-1.26wmf24/extensions/Wikidata: (no message) (duration: 00m 20s)
15:34 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 21s)
15:34 chasemp: unbanning elastic1005 for T112559
15:32 logmsgbot: demon@tin scap aborted: (no message) (duration: 02m 07s)
15:30 logmsgbot: demon@tin Started scap: (no message)
15:15 logmsgbot: demon@tin Synchronized php-1.26wmf22/extensions/Wikidata: (no message) (duration: 00m 20s)
14:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 12s)
14:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
14:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
14:04 logmsgbot: demon@tin Synchronized langlist-labs: (no message) (duration: 00m 11s)
14:04 logmsgbot: demon@tin Synchronized docroot/noc/conf/: (no message) (duration: 00m 13s)
14:01 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 11s)
13:56 godog: force puppet run on restbase2001 to deploy new cassandra config
13:28 godog: reboot ms-be1012, xfs hosed
13:22 paravoid: upgrading cr1-codfw with newer junos
07:43 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: semver the special:version hook (duration: 00m 12s)
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 12s)
02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-23 02:38:44+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 21s)
00:56 logmsgbot: legoktm@tin Synchronized php-1.26wmf24/extensions/Echo/Hooks.php: Remove duplicate 'MediaWiki' prefix from echo.unseen stats (duration: 00m 12s)
00:03 RoanKattouw: Running FlowFixLinks.php on testwiki

2015-09-22

23:40 mutante: renaming search mailing lists to discovery mailing lists
23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf24/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/240283/ and https://gerrit.wikimedia.org/r/#/c/240281/ (duration: 00m 13s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/240278/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/240259/ (duration: 00m 12s)
23:15 logmsgbot: krenair@tin Synchronized w/static/images/sul/wikimania.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 11s)
23:14 logmsgbot: krenair@tin Synchronized w/static/images/sul/commons.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 12s)
22:44 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgFlowMigrateReferenceWiki to false in production (duration: 00m 12s)
22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on testwiki for testing (duration: 00m 12s)
21:56 cwdent: updated payments from 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3 to 8428499feb8760d63faf681d53995697a2ba0fa7
21:49 chasemp: unban elastic1030 from T112559
21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf24
21:13 logmsgbot: twentyafterfour@tin Finished scap: Test 1.26wmf24 (duration: 50m 34s)
20:44 mutante: cancel backup job of bast1001 on helium because running low on disk
20:22 logmsgbot: twentyafterfour@tin Started scap: Test 1.26wmf24
16:50 robh: all mw servers returned to puppet enabled, puppet swat window over
16:40 awight: updated paymentswiki 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
16:23 robh: re-enabled puppet on mw hosts, as both redirection changes are good
16:14 robh: re-enabling puppet on mw hosts, as the new patchset 239278 deployed and tested fine on a single host, deploying to rest
16:04 robh: disabling puppet across mw hosts for new configuration deployment
15:46 godog: running puppet on restbase2001
15:41 godog: stop puppet on restbase2* pending codfw expansion
15:31 cwdent: updated payments from 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
14:42 cmjohnson1: shutting down elastic1005 and elastic1030 to move around within the data center
14:19 bblack: starting slow restart of varnish + varnish-frontend daemon processes on global text, upload, and mobile clusters for shm_reclen (all randomly blended, no parallelism, ~5 minute spacing, will take ~9 hours - FEs will lose cache data, BEs will not)
14:14 chasemp: depool elastic nodes for T112559
11:18 logmsgbot: aude@tin Synchronized php-1.26wmf23/extensions/Wikidata: Fix autocomment and change handling bugs (duration: 00m 21s)
10:42 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access for Wikibooks (duration: 00m 12s)
10:42 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks - try again for snapshot hosts (duration: 00m 12s)
10:35 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks (duration: 01m 12s)
10:17 moritzm: enabled ferm on mw1152 (videoscaler)
10:03 godog: finished stressdisk on restbase200[123] no errors reported
10:03 moritzm: enabled ferm on mw1259 (videoscaler)
04:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 22 04:35:46 UTC 2015 (duration 35m 45s)
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-22 02:22:56+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 00s)
01:53 mutante: sodium - deleted salt key, revoked puppet cert, rm from icinga ..
00:32 ori: Disabled Puppet for 24h on hafnium and stopped ganglia-monitor. gmond was saturating CPU.

2015-09-21

23:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/extensions/NavigationTiming/modules/ext.navigationTiming.js: T112593 (duration: 00m 14s)
23:21 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/239828/ (duration: 00m 21s)
23:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238357/ (duration: 00m 13s)
22:26 mutante: restarting gerrit for ssh config change
21:15 awight: legacy PayPal listener updated from 1c9ac2e66d11bbf768ea873d6e1a2522ca9841c1 to 55aeef63f6508381e3a8b7fcabddf9a3c3b73b8e
20:43 cwdent: updated worldpay config on payments
20:41 mdholloway: MobileApps deployed sha1 013044e
20:39 chasemp: banning elastic1005 for T112559
20:29 subbu: deployed parsoid version 9984d221
19:41 urandom: temporarily stopping codfw restbase cassandra nodes to test quorum auth
19:15 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ieccb23f: Enable async secondary writes for mysql-multiwrite cache (on testwiki) (duration: 00m 13s)
18:36 ejegg: re-enabled paypal audit parser
18:16 cmjohnson1: disabling puppet on mw1031
17:58 chasemp: banning 1030 from eqiad elastic cluster for T112559#1660068
17:57 ejegg: disabled paypal audit parser
16:08 ejegg: updated payments-wiki from 4d9d165c40070e036176dba8987243f6dbc7415e to 153418195a45cab820bc2aacf9a4f7dbc9dde768
15:22 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/ContentTranslation/modules/entrypoint/ext.cx.interlanguagelink.js: SWAT: Revert "Do not call cxserver to display gray interwiki link" gerrit:239819 (duration: 00m 11s)
15:10 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Remove redundant entries from robots.txt gerrit:239403 (duration: 00m 12s)
14:33 ottomata: restart eventlogging with mysql consumer replace=True (AKA INSERT IGNORE)
14:09 godog: rolling restart restbase in production after cassandra credentials change
12:53 godog: rolling restart cassandra after enabling dc encryption, no nodes in codfw yet
12:01 moritzm: repooled mw1160 (for T104969)
11:54 moritzm: depooled mw1160 (for T104969)
11:51 moritzm: repooled mw1158, mw1159 (for T104969)
11:39 moritzm: depooled mw1158, mw1159 (for T104969)
11:37 moritzm: depooled and repooled mw1156, mw1157 (for T104969)
11:26 moritzm: repooled mw1154, mw1155 (for T104969)
11:21 moritzm: depooled mw1154, mw1155 (for T104969)
10:39 moritzm: repooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
10:24 moritzm: depooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
10:17 moritzm: repooled mw1100-mw1109 (for T104968)
10:17 godog: create restbase user on cassandra cluster
10:06 moritzm: depooled mw1100-mw1109 (for T104968)
09:56 moritzm: repooled mw1140 and mw1142-mw1148 (for T104968)
09:41 moritzm: depooled mw1140 and mw1142-mw1148 (for T104968)
09:36 moritzm: repooled mw1130-mw1139 (for T104968)
09:22 moritzm: depooled mw1130-mw1139 (for T104968)
09:14 moritzm: repooled mw1120-mw1129 (for T104968)
09:02 moritzm: depooled mw1120-mw1129 (for T104968)
08:48 moritzm: repooled mw1189 and mw1200-mw1208 (for T104968)
08:33 moritzm: depooled mw1189 and mw1200-mw1208 (for T104968)
08:29 godog: switch to 'restbase' cassandra user on restbase test cluster
08:29 moritzm: repooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
08:21 _joe_: restarted the logstash agent on logstash1003, OOM'd
08:18 moritzm: depooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
08:07 _joe_: installing the new HHVM package on the api canaries
08:04 moritzm: repooled mw1221-mw1229 (for T104968)
07:53 moritzm: depooled mw1221-mw1229 (for T104968)
07:49 moritzm: repooled mw1230-mw1235 (for T104968)
07:43 _joe_: installing the new hhvm package on the canary appservers
07:08 moritzm: depooled mw1230-mw1235 (for T104968)
04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s)
02:06 MaxSem: Maps: created indexes on admin. <3 Postgres :(
01:56 bblack: downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time
01:46 bblack: downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 )

2015-09-20

22:34 yuvipanda: reloda pybal on lvs1012
17:01 bblack: repooling cp1046 varnish-be + varnish-be-rand in confctl, fresh storage, purge queue caught up - T113184
16:44 bblack: depooling cp1046 varnish-be + varnish-be-rand in confctl, wiping storage, re-pooling - T113184
07:24 paravoid: temporarily disabling puppet on fermium and applying antispam countermeasures
04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 20 04:29:16 UTC 2015 (duration 29m 15s)
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-20 02:22:53+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 12s)

2015-09-19

23:12 urandom: begining Cassandra repair on restbase1005 (nodetool repair -pr)
23:08 urandom: begining Cassandra repair on restbase1004 (nodetool repair -pr)
19:56 jynus: restarting once more giblit, last chance
19:04 paravoid: salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it
19:00 ejegg|away: updated SmashPig from d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85 to d1baa32267eaad7d69b47c657f4853eb306fad6b
18:45 _joe_: restarted gitblit. I will now substitute myself with a clever perl one-liner.
18:38 paravoid: pooling back cp1046 to pybal eqiad/mobile, has stayed stable
18:34 paravoid: reactivating ΒGP with GTT @ eqiad
08:42 _joe_: cp1046 dead on console again, powercycling to inspect it
05:49 logmsgbot: aaron@tin Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s)
05:22 paravoid: pybal-depooling cp1046 from eqiad/mobile until further investigation
05:21 paravoid: powercycling cp1046, dead on console
05:01 awight: deploy SmashPig config to limit weekend spam
04:40 awight: update crm from 15ea14f61338ca9f34e9ccb9f56eae14a161380a to 9fa38d06a75363a8009bce7ced190e39c75b68bc
04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s)

2015-09-18

23:22 awight: update fundraising-tools from 3e0e3ae799a507b378d0ece3e71631b10b361329 to e1b60fa2c258fd4ff55905b03a4d8886132278c1
20:52 ebernhardson: restart es on elastic1025 to disable dynamic scripting
20:34 gwicke: dropped by_ns indexes on restbase title_revisions tables
19:54 gwicke: finished deploy of restbase daacf4daa
19:45 gwicke: re-enabled puppet on restbase100*
19:35 gwicke: canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend.
18:38 cwdent: updated payments from 1bdd287b083032ff418434ad6bb6920735af918a to 4d9d165c40070e036176dba8987243f6dbc7415e
17:54 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s)
16:41 mutante: mailman now on 2.1.18 and jessie
16:14 dcausse: elastic in eqiad plugin updates: restarting elastic1021
16:07 paravoid: deactivating ΒGP with GTT @ eqiad
15:20 godog: create restbase user on cassandra test cluster
14:55 dcausse: elastic in eqiad plugin updates: restarting elastic1020
14:22 bblack: committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 )
14:14 bblack: committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3...
14:11 mutante: sodium - stopped exim - rsyncing lists to fermium
14:10 dcausse: elastic in eqiad plugin updates: restarting elastic1019
14:07 mutante: stopped mailman on sodium
14:01 bblack: rollback on asw-b-eqiad changes above
13:56 bblack: committing eqiad lvs1007-1012 port/vlan changes for asw-b-eqiad
13:20 bblack: committing eqiad lvs1007-12 port/vlan changes for asw-c-eqiad
13:16 bblack: commiting eqiad lvs1007-12 port/vlan changes for asw2-a5-eqiad
13:12 dcausse: elastic in eqiad plugin updates: restarting elastic1018
12:21 godog: restart logstash on logstash1001, OOM in logs
11:55 dcausse: elastic in eqiad plugin updates: restarting elastic1017
11:06 dcausse: elastic in eqiad plugin updates: restarting elastic1016
10:28 moritzm: restarted salt-master on palladium
09:46 moritzm: installed openldap security updates on plutonium
09:37 moritzm: installed openldap security updates on pollux
09:33 dcausse: elastic in eqiad plugin updates: restarting elastic1015
08:22 dcausse: elastic in eqiad plugin updates: restarting elastic1014
07:21 dcausse: elastic in eqiad plugin updates: restarting elastic1013
06:15 dcausse: elastic in eqiad plugin updates: restarting elastic1012
04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 18 04:37:42 UTC 2015 (duration 37m 41s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-18 02:31:49+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 08s)
02:21 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 12s)
02:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 11s)
02:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237149/ (duration: 00m 12s)
02:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234544/ (duration: 00m 12s)
01:58 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 12s)
01:58 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 11s)
01:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://phabricator.wikimedia.org/T106264 (duration: 00m 12s)
01:36 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237331/ (duration: 00m 12s)
00:14 ori: restarted logstash on logstash1001

2015-09-17

23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238978/ (duration: 00m 12s)
23:05 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
23:04 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
22:53 gwicke: puppet on restbase cluster disabled since about 21:30 UTC for gradual deploy; ran into minor issue in staging, which is now being addressed, after which deploy will continue
21:22 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 13s)
21:22 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 12s)
20:44 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
20:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/239206/ (duration: 00m 12s)
19:46 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
19:41 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/239181/ (duration: 00m 14s)
19:12 mutante: powercycling unresponse mw1005
18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf23
17:38 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 13s)
17:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 11s)
16:43 chasemp: restart elasticsearch on 1005
16:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235900/ (duration: 00m 12s)
15:15 dcausse: elastic in eqiad plugin updates: restarting elastic1004 (take 2)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable Suggestions in ptwiki gerrit:238097 (duration: 00m 13s)
14:22 mutante: analytics1029 - Failed to start Hadoop datanode
14:20 mutante: starting hadoop datanode on analytics1029
14:14 _joe_: reimaging tmh1001 to mw1259
14:11 jynus: stopping replication and applying schema change to db1051
14:05 dcausse: elastic in eqiad plugin updates: can't restart elastic1004 (2 timeouts when disabling replication, too much load?), waiting for more shards to rebalance...
13:58 dcausse: elastic in eqiad plugin updates: restarting elastic1004
13:50 moritzm: repooled mw1236-mw1239 (T104968)
13:34 moritzm: depooled mw1236-mw1239 (T104968)
13:26 moritzm: repooled mw1090-mw1099 (T104968)
13:16 moritzm: depooled mw1090-mw1099 (T104968)
13:13 moritzm: repooled mw1080-mw1089 (T104968)
13:05 moritzm: depooled mw1080-mw1089 (T104968)
13:01 moritzm: repooled mw1070-mw1079 (T104968)
12:49 moritzm: depooled mw1070-mw1079 (T104968)
12:35 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
12:24 moritzm: depooled mw1060 and mw1062-mw1069 (T104968) (not repooled)
12:24 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
12:16 moritzm: repooled mw1050-mw1059
12:04 moritzm: depooled mw1050-mw1059
11:39 moritzm: repooled mw1040 and mw1042-mw1049 (T104968)
11:36 dcausse: elastic in eqiad plugin updates: restarting elastic1003
11:26 moritzm: typoed earlier entry: "mw1032-mw1039" instead of "mw1032-mw1239"
11:26 moritzm: depooled mw1040 and mw1042-mw1049 (T104968)
11:18 moritzm: repooled mw1030 and mw1032-mw1239 (T104968)
11:03 moritzm: depooled mw1030 and mw1032-mw1239 (T104968)
10:35 moritzm: repooled mw1250-mw1258 (T104968)
10:27 moritzm: depooled mw1250-mw1258 (T104968)
10:25 _joe_: killing temporarily subra
10:24 moritzm: repooled mw1240-mw1249 (T104968)
10:19 _joe_: experimenting with poolcounter issues on subra
10:18 logmsgbot: oblivian@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: Use codfw poolcounters in codfw (duration: 00m 12s)
10:12 moritzm: depooled mw1240-mw1249 (T104968)
10:12 dcausse: elastic in eqiad plugin updates: restarting elastic1002
10:05 logmsgbot: hoo@tin Synchronized wmf-config/: Set 'repoConceptBaseUri' for all Wikibase clients (duration: 00m 13s)
10:00 dcausse: elastic in eqiad plugin updates: unfreezing indices
09:48 dcausse: elastic in eqiad plugin updates: no more groovy in warmers, waiting for few more shards to move in elastic1001 and will unfreeze indices to test warmers
09:39 dcausse: elastic in eqiad plugin updates: deleting warmers manually for old unused indices (eswikisource_content_1415240352, ruwiki_content_1415302164, thwiki_content_1415318677). We will have to remove these indices.
09:39 paravoid: repooling ulsfo US-West traffic back to ulsfo for the first time since May :)
09:01 dcausse: elastic in eqiad plugin updates: updating warmers on all wikis
08:58 paravoid: penalizing ulsfo-eqiad direct MPLS links to higher OSPF weights
08:57 paravoid: adjusting OSPF weights to be latency-based across the US network
08:53 _joe_: removed iptables rules for dropping traffic to helium on mw1017
08:52 dcausse: elastic in eqiad plugin updates: index warmer queries are outdated with inline groovy script, updating warmers on warwiki first to test
08:05 paravoid: eqiad-codfw -> eqiad-eqord-codfw migration
07:49 moritzm: repooled mw1180-mw1188 (T104968)
07:42 dcausse: elastic in eqiad plugin updates: restarting elastic1001
07:42 moritzm: depooled mw1180-mw1188 (T104968)
07:37 moritzm: repooled mw1170-mw1179 (T104968)
07:36 dcausse: elastic in eqiad plugin updates: freezing indices
07:27 moritzm: depooled mw1170-mw1179 (T104968)
07:14 _joe_: uploading new HHVM package
07:07 moritzm: repooled mw1161-1168 (T104968)
06:57 moritzm: depooled mw1161-1168 (T104968)
06:45 moritzm: repooled mw1209-mw1220 with ferm enabled
06:33 moritzm: depooling mw1209-mw1220 (in two steps)
05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 17 05:47:47 UTC 2015 (duration 47m 46s)
03:06 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-17 03:06:33+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-17 02:45:48+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 11s)
00:35 cwdent: updated payments from 155cdeb737c01baf62551292764fd2f5a93a9a63 to 1bdd287b083032ff418434ad6bb6920735af918a

2015-09-16

23:27 bblack: updating eqiad switch configs for lvs1007-1012 vlan/trunk settings
23:19 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/MobileFrontend/resources/mobile.overlays/Overlay.less: https://gerrit.wikimedia.org/r/#/c/238865/ (duration: 00m 11s)
23:13 gwicke: started `nodetool rebuild -- eqiad` on restbase-test200{1,2
23:03 cwdent: updated payments from 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1 to 155cdeb737c01baf62551292764fd2f5a93a9a63
22:26 ejegg: updated SmashPig from fdb053efa617162ac9f695e493c390987a069140 to d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85
22:08 urandom: disabling puppet in RESTBase eqiad staging cluster to test new code and config
22:08 ottomata: powercycling analytcis1029, it is down?
20:47 cscott: updated OCG to version 4032a596ce6eb442b02cc6ee9b79263b1eb23275
19:42 ejegg: updated crm from abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b to 15ea14f61338ca9f34e9ccb9f56eae14a161380a
19:38 ori: Deployed statsv 0bfd9f06f / change I050a12d3b
18:47 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf23
18:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf23: syncing wmf23 ahead of deployment to group1 (duration: 01m 35s)
17:34 paravoid: asw-d-eqiad: toggling RE mastership again
17:26 godog: stop puppet on restbase* to apply https://gerrit.wikimedia.org/r/#/c/238738/ / merge / reenable puppet
16:54 _joe_: turned on the hhvm tmh, stopping the zend ones for testing
16:44 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: use ffmpeg whereever possible (duration: 00m 12s)
16:16 bblack: upgrading pybal on lvs400[12]
16:12 bblack: upgrading pybal on lvs400[34], lvs300[34]
16:08 bblack: upgrading pybal on lvs200[123]
16:05 bblack: upgrading pybal on lvs200[456]
15:44 _joe_: uploading pybal 1.10 to reprepro, installing to the test cluster
15:24 moritzm: uploaded debdeploy 0.0.6 to apt.wikimedia.org
15:10 hashar: Started using Nodepool spawned instances. Moved integration-jjb-config-diff Jenkins job to Nodepool with https://gerrit.wikimedia.org/r/#/c/238752/ . See also: https://phabricator.wikimedia.org/T112750
15:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Add m.wikidata.org to wgCrossSiteAJAXdomains (duration: 00m 12s)
14:51 _joe_: experimenting on testwiki for poolcounter failure scenarios
14:45 moritzm: enabled ferm on mw1010 (jobrunner) in eqiad
14:27 paravoid: asw-d-eqiad: toggling RE mastership
14:18 paravoid: disabling/ignoring asw-d-eqiad @ librenms
14:09 jynus: upgrading and restarting db1051
13:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1051 for maintenance (duration: 00m 12s)
13:40 urandom: initiating Cassandra repair on restbase1007 (nodetool repair -pr)
13:40 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 01m 37s)
13:35 moritzm: repooled mw1149-mw1151 (with ferm enabled)
13:24 moritzm: depooled mw1149-mw1151 (for enabling ferm)
13:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting depool of es1055 (duration: 00m 12s)
13:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1055 for maintenance (duration: 00m 12s)
13:03 paravoid: disabling asw-d-eqiad xe-8/0/23, xe-8/0/24, xe-8/0/25, xe-8/0/26, xe-8/0/27, xe-8/0/28; servers reboot-looping -> asw-d's SNMP unhappy -> librenms unhappy -> faidon's mailbox unhappy
12:48 moritzm: repooled mw1115-mw1117, mw1119 (with ferm enabled)
12:42 moritzm: depooling mw1115-mw1117, mw1119 (mw1118 was already depooled) to enable ferm
11:32 moritzm: repooled mw1019-mw1025 with ferm enabled
11:24 jynus: making db1069 a sibling of db1055 (s1)
11:13 godog: create restbase user on cassandra test cluster
11:07 moritzm: depooled mw1019-mw1025 (to enable ferm)
10:52 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 02m 04s)
10:49 logmsgbot: catrope@tin Synchronized php-1.26wmf22: (no message) (duration: 02m 12s)
10:48 jynus: reenabling semisync on db1072 and db1073
10:47 logmsgbot: catrope@tin scap aborted: (no message) (duration: 00m 21s)
10:47 logmsgbot: catrope@tin Started scap: (no message)
10:24 logmsgbot: catrope@tin Synchronized php-1.26wmf23/includes/changes/EnhancedChangesList.php: T112738 (duration: 00m 12s)
10:09 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: (no message) (duration: 00m 11s)
09:37 awight: ruthlessly disabled PayPal IPN listener
08:12 moritzm: repooled mw1153 with ferm enabled
07:57 jynus: truncated some tables from ContentTranslation extension on x1
07:57 moritzm: depooled mw1153 (it's an image scaler, of course) to enable ferm
07:56 moritzm: depooled mw1153 (videoscaler) to enable ferm
06:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 16 06:31:58 UTC 2015 (duration 31m 57s)
03:28 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Don't waste milliseconds counting microseconds (duration: 00m 12s)
03:27 logmsgbot: ori@tin Synchronized php-1.26wmf23/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Dont waste milliseconds counting microseconds ; sync-file php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php Iccfda47689: monolog: Dont waste milliseconds counting microseconds (duration: 00m 12s)
03:12 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-16 03:12:08+00:00
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 10m 30s)
02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-16 02:38:48+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 07m 02s)
01:03 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 12s)
00:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 11s)
00:15 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
00:14 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
00:11 bblack: reinstalling lvs400[12] to jessie (traffic on 400[34], already jessie)
00:08 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/VisualEditor/modules/ve-mw/ui/styles/dialogs: https://gerrit.wikimedia.org/r/#/c/238646/ (duration: 00m 12s)

2015-09-15

23:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238617/ (duration: 00m 12s)
23:48 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238618/ (duration: 00m 12s)
23:42 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 14s)
23:42 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 12s)
23:40 logmsgbot: krenair@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238544/ (duration: 00m 11s)
23:38 logmsgbot: krenair@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238545/ (duration: 00m 11s)
23:24 yurik: deployed kartotherian & tilerator
23:22 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238512/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238513/ (duration: 00m 12s)
21:15 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: touch files edited in I0cb6fe37e and re-sync to cluster (duration: 00m 13s)
21:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf23
21:10 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded (duration: 03m 52s)
21:07 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded
21:05 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, again (duration: 47m 49s)
20:47 mutante: mw1010 - extremely slow,finally got on and attempted to restart hhvm. load going down
20:17 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, again
20:17 logmsgbot: twentyafterfour@tin scap aborted: sync 1.26wmf23 to testwiki (duration: 82m 58s)
20:05 ottomata: restarted mysql (and oozie) on analytics1027 to start mysql binlogging
18:54 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki
16:55 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1003, es1004, es1007 and es1010 for decommision (duration: 00m 12s)
16:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Revert depool db1055 for maintenance (duration: 00m 11s)
16:39 ottomata: reinstalling analytics1015
16:32 RoanKattouw: Putting wmf22 versions of Echo and MobileFrontend on mw1017 for testing
16:30 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/WikimediaEvents.php: touch file that is serving old version in prod (duration: 00m 12s)
16:29 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSuggest.js: Touch file that is serving old version in prod (duration: 00m 12s)
16:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for maintenance (duration: 00m 11s)
16:11 bblack: traffic DNS depooled out of codfw for now T112639
15:38 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: CX: Enable suggestion for testwiki (part 2) gerrit:237327 (duration: 00m 13s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable suggestion for testwiki (part 1) gerrit:237327 (duration: 00m 12s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js: SWAT: Do not fail horribly when invalid categories are passed gerrit:238421 (duration: 00m 12s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-eqiad.php: SWAT: poolcounter: enable connect_timeout for testwiki gerrit:238109 (duration: 00m 19s)
15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: SWAT: poolcounter: add connect_timeout in codfw gerrit:238108 (duration: 00m 12s)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Exclude Flow topic boards and Draft NS from Special:UnconnectedPages gerrit:229197 (duration: 00m 11s)
14:51 godog: bounce cassandra on test cluster to deploy https://gerrit.wikimedia.org/r/236391
14:22 cmjohnson1: swapped disk on db1043
13:12 moritzm: repool mw1114 (with ferm enabled)
13:11 bblack: failing over LVS service in ulsfo to secondariess (400[12] pybal stopped, traffic on jessie-based 400[34])
12:53 moritzm: depooled mw1114 (for enabling ferm)
11:42 moritzm: repool mw1018 (with ferm enabled)
11:23 moritzm: depooled mw1018 (for enabling ferm)
08:53 _joe_: created a 100 G partition on a LV on copper, for /tmp
08:24 godog: bounce ms-be2006, xfs
08:22 moritzm: bumped default size of iptables connection tracking table to 256k
06:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 15 06:10:52 UTC 2015 (duration 10m 51s)
02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-15 02:46:50+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 53s)
02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/MobileFrontend: Revert Echo to 1.26wmf21 state (duration: 00m 11s)
02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo: Revert Echo to 1.26wmf21 state (duration: 00m 12s)
01:30 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src: T112287 (duration: 00m 11s)
00:49 bblack: reinstalling lvs300[34] to jessie
00:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: noop sync of labs config change (duration: 00m 11s)
00:03 logmsgbot: tstarling@tin Synchronized php-1.26wmf22/extensions/ParsoidBatchAPI: for I56d28e9a for RT testing, not live yet (duration: 00m 13s)

2015-09-14

23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: Change bucket selection methods in CompletionSuggestions AB test (duration: 00m 12s)
23:23 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/UploadWizard/: Swat out badtoken fix to UploadWizard in 1.26wmf22 (duration: 00m 12s)
22:37 yurik: deployed tilerator
21:15 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: Ie44fcb500: Avoid checking blacklists in isBlacklisted() for existing titles (duration: 00m 12s)
21:15 mutante: labnodepool1001 - re-enable puppet and nodepool
20:59 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Hack around OOUI's icon pack being too large by creating our own (duration: 00m 12s)
20:53 cscott: updated OCG to version 5811056e28f2bc6408b6da96095352ab381bb11f
20:21 andrewbogott: graceful’d apache2 on labcontrol1001
20:15 subbu: deployed parsoid sha 3d5f4359
19:25 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Only load nojs Special:Notifications styles on the special page (duration: 00m 12s)
18:05 urandom: rebuilding restbase-test2001.codfw (nodetool rebuild -- eqiad)
16:12 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: For real this time (duration: 00m 11s)
16:06 ottomata: stopping hdfs journalnode on analytics1011 to copy journal edits to new journalnodes on analytics1035 and analytics1052
15:46 godog: switch to openjdk-8 and bounce cassandra on restbase-test200*
15:39 bblack: reinstalling lvs4003, lvs4004 (jessie upgrade: T96375) (typo earlier)
15:39 bblack: reinstalling lvs4003, lvs4003 (jessie upgrade: T96375)
15:34 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 13s)
15:05 logmsgbot: krenair@tin Synchronized .gitignore: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 13s)
15:05 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 12s)
15:04 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 11s)
15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234980/ (duration: 00m 12s)
13:38 godog: stop puppet on restbase-test2001 and turn up cassandra
12:56 bblack: rebooting lvs2006 to test eth hw params stuff...
12:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238125/ (duration: 00m 13s)
12:50 urandom: starting Cassandra repair on restbase1003 (nodetool repair -pr)
12:32 godog: enable dc encryption on cassandra test cluster and rolling restart
11:33 mobrovac: citoid deploying d569951
10:35 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002, es1005, es1008 (duration: 00m 12s)
10:04 jynus: db1029 (x1-master) temporarily saturated by connections- flow was unresponsive for 10 minutes; migration partially aborted
09:08 jynus: applying schema change to flowdb
08:52 godog: rename cassandra test cluster and restart
08:44 godog: silence mendelevium for today, status unclear T111532
08:30 jynus: endinf profiling and executing pt-query-digest on db1043 [ETA:4h]
07:52 godog: reboot ms-be1010 to pick up disk ordering change
04:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 14 04:47:58 UTC 2015 (duration 47m 57s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-14 02:29:48+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 59s)
01:31 Krinkle: mwscript deleteEqualMessages.php --wiki sqwiki

2015-09-13

06:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 13 06:02:52 UTC 2015 (duration 2m 51s)
02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-13 02:40:43+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 13s)

2015-09-12

20:15 ori: Rolling back Echo to 1.26wmf21 branch on mw1017 (testwiki) to measure increase in render-blocking CSS size
19:21 urandom: performing Cassandra repair on restbase1002 (nodetool repair -pr)
14:50 jynus: phab.wmfusercontent.org has been temporarily switched to phab.wikivoyage.org due to cert issues
04:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 12 04:52:01 UTC 2015 (duration 52m 0s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-12 02:35:36+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 54s)

2015-09-11

21:21 hashar: shutdown nodepool on labnodepool1001.eqiad.wmnet until monday
18:01 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression fixes #2 (duration: 00m 12s)
16:43 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: T112232 (duration: 00m 12s)
16:37 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression backports (duration: 00m 12s)
16:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (again) (duration: 00m 13s)
16:33 legoktm: ssh: connect to host mw1156.eqiad.wmnet port 22: Connection timed out
16:32 paravoid: powercycling mw1156, multiple kernel backtraces in console output
16:32 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (duration: 01m 07s)
16:15 cmjohnson1: mw1031 rebooting for f/w update
16:07 bblack: enabled LRO+GRO on lvs200[123], starting pybal there again ([456] testing looks good so far)
15:45 bblack: enabled LRO+GRO on lvs200[456] (backups). Stopping pybal on lvs200[123] to test...
15:11 cmjohnson1: swapping pem2 cr2-eqiad
10:03 jynus: starting nodepool in labnodepool1001
09:21 jynus: starting profiling of phabricator db (db1043). Very low overhead.
06:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 11 06:03:00 UTC 2015 (duration 2m 59s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-11 02:41:24+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 18s)
01:16 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: 9bf13dbe0b, 3203b045f7 (duration: 00m 12s)

2015-09-10

23:52 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237064/ (duration: 00m 11s)
23:47 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237056/ (duration: 00m 11s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/221825 (duration: 00m 13s)
23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/224771 (duration: 00m 12s)
21:13 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/modules: Align popup footer buttons to take 50% width each (duration: 00m 15s)
20:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1001; increase weight of es1015 and es1019 (duration: 00m 19s)
20:47 ottomata: restarting eventlogging with 12 client side processors on eventlog1001
20:31 ottomata: turning off varnishncsa eventlogging eventlistener instances on frontend caches, it is now superseded by varnishkafka
20:28 mutante: killed/restarted ganglia aggregator process for mobile-cache, upload cache, misc esams ...
20:22 jynus: last SCAP failed on 266/466 hosts
20:21 mutante: killed/restarted ganglia aggregator process for text-caches esams on hooft
20:17 yurik: deployed kartotherian
20:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001; increase weight of es1015 and es1019 (duration: 00m 11s)
19:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf22
19:09 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/237458/ (duration: 00m 12s)
18:57 twentyafterfour: restarted phd on iridium
18:51 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/Wikidata: Deploy wikidata patch: https://gerrit.wikimedia.org/r/#/c/237449/ (duration: 00m 19s)
18:23 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22: deploy https://gerrit.wikimedia.org/r/#/c/237440/ (duration: 01m 42s)
18:09 cmjohnson1: reseating pem2 cr2-eqiad
16:52 akosiaris: puppetswat done
16:50 mobrovac: restbase rolling restart of rb100x
16:49 mobrovac: restbase enabled puppet on rb100x
16:13 akosiaris: started puppetSWAT
16:10 logmsgbot: marktraceur@tin Finished scap: Make sure codfw got the last few patches sync'd to it (duration: 07m 36s)
16:03 logmsgbot: marktraceur@tin Started scap: Make sure codfw got the last few patches sync'd to it
16:02 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/: [SWAT] [wmf22] Revert opera redirect loop fix that caused redirect loops in Firefox (duration: 02m 30s)
15:55 mobrovac: restbase disabled puppet on rb100x
15:45 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: [SWAT] [wmf22] Always set 'offset' with chunked uploads, even for first chunk (offset == 0) (duration: 02m 21s)
15:26 ottomata: started hadoop decomission of analytics1016
15:21 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] Attempting another sync to mw2187 hoping it's up now (duration: 02m 22s)
15:05 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] [config] Beta: Enable Content Translation suggestions (duration: 02m 22s)
13:35 moritzm: enabled ferm on mediawiki app servers in codfw
13:30 jynus: performing schema change and maintenance on officewiki and public all wikis with flow enabled
12:51 moritzm: enabled ferm on mediawiki API servers in codfw
12:36 moritzm: enabled ferm on mediawiki video scalers, image scalers and job runners in codfw
09:20 mobrovac: restbase deploying 0182962
06:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 10 06:13:14 UTC 2015 (duration 13m 13s)
03:02 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-10 03:02:45+00:00
02:59 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 10s)
02:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237304 (duration: 00m 11s)
02:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237303 (duration: 00m 10s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-10 02:43:20+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 10m 45s)
02:24 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.js: Ic0b1fb64ee7 backport (duration: 00m 12s)
01:04 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 13s)
01:03 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 12s)
00:54 mutante: powercycling unresponsive mw1154

2015-09-09

23:34 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:31 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:29 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:23 MaxSem: deployed Kartotherian config updates
23:23 logmsgbot: catrope@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 11s)
23:22 RoanKattouw: Running updateinterwikicache
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance: SWAT (duration: 00m 13s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Flow: SWAT (duration: 00m 32s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance: SWAT (duration: 00m 14s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/Flow: SWAT (duration: 00m 29s)
20:17 subbu: deployed parsoid version ffd0b444
18:15 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf22
16:47 andrewbogott: systemctl stop nodepool on labnodepool1001
16:06 logmsgbot: aude@tin Synchronized database lists: Remove unused usagetracking.dblist (duration: 00m 12s)
16:01 logmsgbot: krenair@tin Synchronized robots.txt: https://gerrit.wikimedia.org/r/#/c/236200/ (duration: 00m 12s)
15:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236701/ - noop (duration: 00m 12s)
15:56 ejegg: updated payments from from 4c5e30288370db926cbbf7a7528edb9c41c65716 to 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1
15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237104/ (duration: 00m 12s)
15:46 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
15:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.searchSuggest.js: Enable completion suggester AB experiment (duration: 00m 12s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/: Enable suggester AB experiement (duration: 00m 11s)
15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/237091/ (duration: 00m 21s)
15:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234425/ (duration: 00m 12s)
15:21 logmsgbot: krenair@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/236994/ (duration: 00m 12s)
15:15 bd808: Running sync-common manually on mw2187.codfw.wmnet. Host is missing l10n cache files
15:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236025/ (duration: 00m 11s)
15:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236042/ (duration: 00m 13s)
14:03 mutante: beginning mailman migration - expect lists to be down
13:14 moritzm: enabled ferm on test.wikipedia.org (mw1017)
13:05 urandom: issuing Cassandra repair on restbase1001 (nodetool repair -pr)
13:02 moritzm: enabled ferm on various initial mediawiki hosts in codfw: videoscaler (mw2007), appserver (mw200[89]), jobrunner (mw2081), api (mw2050), imagescaler (mw2086)
10:33 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Remove unused usagetracking tag (duration: 00m 11s)
10:30 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 12s)
10:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: rv usage tracking (duration: 00m 12s)
10:23 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on commons and test2wiki (duration: 00m 11s)
10:21 logmsgbot: aude@tin Synchronized wikidataclient.dblist: Sorted dblist (duration: 00m 12s)
09:41 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikinews (duration: 00m 12s)
08:35 moritzm: installed spice security updates on labvirt*, ganeti* and labnodepool1001
05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 9 05:11:28 UTC 2015 (duration 11m 27s)
02:55 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-09 02:55:24+00:00
02:52 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 05m 34s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-09 02:31:50+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 44s)
00:00 logmsgbot: catrope@tin Finished scap: Need to update i18n for a new Echo message (duration: 23m 08s)

2015-09-08

23:36 logmsgbot: catrope@tin Started scap: Need to update i18n for a new Echo message
23:36 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT (duration: 00m 10s)
23:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT (duration: 00m 13s)
23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 12s)
23:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 12s)
23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: SWAT (duration: 00m 11s)
23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 14s)
23:14 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 11s)
22:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: re-apply patch 1/2 (jscs) (duration: 00m 12s)
21:36 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: temporarily revert T109756 (duration: 00m 11s)
21:02 csteipp: deployed patches for T108616 T91850 T91205 to wmf21 & 22
20:45 bblack: upgrading nginx to 1.9.4 on cp*
20:38 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 12s)
20:38 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 15s)
20:37 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 14s)
20:07 logmsgbot: aude@tin Finished scap: Update group0 to new Wikidata branch (duration: 24m 27s)
19:42 logmsgbot: aude@tin Started scap: Update group0 to new Wikidata branch
19:14 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf21/: sync php-1.26wmf21 as well (duration: 02m 31s)
19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf22
18:55 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
18:50 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf22 (duration: 29m 29s)
18:20 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf22
18:01 ejegg: rolled back payments to 6ac552f280fb839069d117386c4ecbe9e52f90a8
17:59 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
17:43 moritzm: enabled ferm on remaining hadoop workers (analytics1040-analytics1057)
17:09 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: T109756 (duration: 00m 11s)
16:56 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/CentralAuth: T108253 sul2 token store (duration: 00m 12s)
16:16 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: I5af46eb3: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 14s)
15:43 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 12s)
15:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236785/ (duration: 00m 12s)
15:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234910/ (duration: 00m 12s)
14:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1015 and es1019 (duration: 00m 11s)
14:30 moritzm: enabled ferm on hadoop workers up to analytics1039
12:41 godog: change whisper aggregation for 'sum.wsp' files T111170
10:48 moritzm: restarted salt master on palladium
10:32 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikibooks (duration: 00m 11s)
09:55 moritzm: uploaded debdeploy 0.0.5 to carbon
04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 8 04:37:06 UTC 2015 (duration 37m 5s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-08 02:23:51+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 30s)
00:46 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki

2015-09-07

21:45 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/236682/ (duration: 00m 12s)
21:44 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/#/c/236196/1 (duration: 00m 12s)
21:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikiEditor: https://gerrit.wikimedia.org/r/#/c/236197/1 and https://gerrit.wikimedia.org/r/#/c/236679/ (duration: 00m 12s)
18:15 andrewbogott: graceful’d apache, restarted keystone on labcontrol1001
15:41 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: https://gerrit.wikimedia.org/r/#/c/236558/ (duration: 00m 12s)
15:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1004, pool es1018 (duration: 00m 10s)
10:04 godog: powercycle ms-be1003, loadavg skyrocketed
08:13 hashar: Jenkins upgraded to latest LTS ( https://phabricator.wikimedia.org/T111326 )
08:05 hashar: Upgrading Jenkins
04:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 7 04:33:11 UTC 2015 (duration 33m 10s)
02:29 Krinkle: mwscript deleteEqualMessages.php --wiki pmswiki
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-07 02:23:27+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 22s)

2015-09-06

04:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 6 04:27:57 UTC 2015 (duration 27m 56s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-06 02:23:08+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 14s)

2015-09-05

23:37 Krinkle: mwscript deleteEqualMessages.php --wiki fywiktionary
04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 5 04:31:34 UTC 2015 (duration 31m 33s)
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-05 02:30:06+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 53s)

2015-09-04

23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 12s)
23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 11s)
22:49 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/Citoid: https://gerrit.wikimedia.org/r/#/c/236218/ and https://gerrit.wikimedia.org/r/#/c/236222/ (duration: 00m 12s)
21:55 urandom: bouncing Cassandra on restbase1001 to restore default GC settings
18:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/236063/ (duration: 00m 11s)
18:06 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/modules/ext.wikimediaEvents.statsd.js: Ib98988f67ef (duration: 00m 11s)
17:35 MaxSem: Maps: dropped duplicate index on water_polygons
16:27 jynus: cloning es1 mysql data from es1004 to es1018 [ETA:16h]
16:11 paravoid: updating firewall border ACLs and BGP border filters across all cr
15:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1002, es1016; Depool es1004 (duration: 00m 11s)
15:35 godog: python varnishlog collector + gdb running on cp1052 for debugging T83580
12:55 moritzm: restarted salt-master on palladium
12:47 moritzm: uploaded debdeploy 0.0.4 to carbon
10:18 logmsgbot: kartik@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: php-1.26wmf21/extensions/ContentTranslation/extension.json T111490:Use the VirtualRESTService to configure CX (duration: 00m 12s)
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-fr-ca_1.0.3~r61329-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-fr_0.9.0~r28336-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-es_0.9.1~r60655-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-ca_0.9.1~r60655-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-ca-it_0.1.1~r57554-1
07:50 jynus: cloning es3 mysql data from es1008 to es1019
04:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 4 04:19:20 UTC 2015 (duration 19m 19s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-04 02:26:04+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 21s)
01:56 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T111439 (duration: 00m 12s)
00:11 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/includes/resourceloader/ResourceLoader.php: I24f68e34a9fa4918 (duration: 00m 12s)
00:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235940/ (duration: 00m 11s)

2015-09-03

23:53 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235853/ (duration: 00m 12s)
23:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
23:50 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
23:40 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
23:37 mutante: mw1224 - killed and restarted defunct hhvm, version is different from the one on mw1225
23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/235728 (duration: 00m 13s)
23:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikisource.png: https://gerrit.wikimedia.org/r/#/c/235728/ (duration: 00m 12s)
23:32 Krenair: mw1224 has been sending segfault warnings and "Lost parent, LightProcess exiting" to hhvm.log since about 21:17:34
23:29 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/235905/ (duration: 00m 13s)
23:28 logmsgbot: krenair@tin Synchronized php-1.26wmf21/package.json: bd2eb6cc1919c7dab056d5f8fe5b4a164236d78f (duration: 00m 13s)
23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235908/ (duration: 00m 13s)
21:21 ori: rebuilt HHVM with updated diff from facebook/hhvm PR #6071 (T109540), uploaded to apt as 3.6.5+dfsg1-1+wm5
21:18 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
19:54 bearND: MobileApps deployed sha1 553c399
19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf21
18:13 ottomata: rolling restart of hadoop yarn nodemanagers to pick up Yarn AppMaster port range limitation to apply ferm rules.
18:04 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
18:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
17:39 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235769/ (duration: 00m 12s)
17:34 mutante: bromine - deleting policy docroot
17:06 jynus: cloning es1006 mysql data into es1015 [ETA:8h]
16:30 bblack: updating nginx->1.9.4 on cp1071, cp3033 for prod validation before broader rollout
16:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es3 master switchover from es1009 to es1014 (eqiad) (duration: 00m 13s)
16:28 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es3 master switchover from es1009 to es1014 (codfw) (duration: 00m 13s)
16:26 mutante: imported jenkins 1.609.3 into APT repo
16:23 legoktm: fixed content model of Template:Languages@metawiki
16:21 robh: re-enabling puppet on all mw systems
16:14 robh: disabling puppet on all mw systems for apache config update
16:01 jynus: performing es3 master switchover from es1009 to es1014
15:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1006 (duration: 00m 12s)
15:17 hashar: stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet
15:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es2 master switchover from es1006 to es1011 (eqiad) (duration: 00m 13s)
15:14 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es2 master switchover from es1006 to es1011 (codfw) (duration: 00m 12s)
15:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
15:04 logmsgbot: demon@tin Synchronized php-1.26wmf21/extensions/Translate/: (no message) (duration: 00m 15s)
14:51 jynus: performing es2 master switchover from es1006 to es1011
14:33 paravoid: rebooting msw1-eqiad
14:28 twentyafterfour: restarted phd (phabricator daemon) to pick up new configuration
14:25 paravoid: changing IPv6 RA interval/lifetime/virtual-router-only @ eqiad
14:21 paravoid: rebooting msw1-codfw
13:17 paravoid: upgrading mr1-esams and mr1-eqiad to newer junos
13:13 godog: bounce carbon daemons on graphite1001
12:42 chasemp: unban elastic1001 and put back in service
12:24 chasemp: move all shards off of elastic1001
12:24 chasemp: disable elastic1001 in lvs as we are gonig to try fw apply round #2
11:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1028; increase the load of es1010, es1013 and es1017 (duration: 00m 12s)
10:45 jynus: applying schema change for ContentTranslation on x1-master "wikishared"
10:02 godog: reenable puppet on ms-be1*
09:16 jynus: started profiling mysql queries at phabricator. Only a 1% overhead is expected.
09:12 moritzm: updated rsyncd firewall rules (see https://gerrit.wikimedia.org/r/235425 for details)
09:12 godog: stop puppet on ms-be1* after ferm rsync change
08:23 godog: fixup current graphite retention T96662
07:26 moritzm: enabled ferm on dbstore* servers in codfw
06:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 3 06:29:35 UTC 2015 (duration 29m 34s)
03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-03 03:09:20+00:00
03:06 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 32s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-03 02:45:36+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 41s)
01:32 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
00:36 logmsgbot: ori@tin Synchronized php-1.26wmf21/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 11s)
00:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 13s)
00:27 RoanKattouw: Deployed patch for T111029

2015-09-02

23:58 logmsgbot: andyrussg@tin Synchronized php-1.26wmf20/extensions/CentralNotice/: CentralNotice update (duration: 00m 13s)
23:33 logmsgbot: andyrussg@tin Synchronized php-1.26wmf21/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
23:02 logmsgbot: andyrussg@tin Finished scap: Update CentralNotice to 2.6.0 for wmf21 (duration: 48m 18s)
22:13 logmsgbot: andyrussg@tin Started scap: Update CentralNotice to 2.6.0 for wmf21
20:27 arlolra: updated Parsoid to version 5f2fae6c
20:08 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf21
20:02 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/startup.js: Ie65427caee (duration: 00m 12s)
19:09 mutante: restarted gitblit, stopped counting
19:07 paravoid: upgrading mr1-codfw, mr1-ulsfo to newer junos
19:01 urandom: bouncing Cassandra on restbase1001 to address bogus icinga process failure alert
18:52 legoktm: deployed patch for T110553
18:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf21
18:32 cmjohnson1: replacing disk 10 on db1028
18:13 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
17:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor/modules/ve-mw/ui/inspectors: https://gerrit.wikimedia.org/r/#/c/235511/ (duration: 00m 12s)
17:07 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/UniversalLanguageSelector: 78a5908fd9: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 16s)
17:07 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/UniversalLanguageSelector: 2154acc529: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 13s)
16:25 mutante: restarting NTP on lvs2004
16:12 jynus: setting BBU auto-learn mode to warn only (disabled if not possible) on all database hosts
16:03 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235484/ (duration: 00m 12s)
16:01 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235486/ (duration: 00m 12s)
15:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235483/ (duration: 00m 13s)
15:56 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235485/ (duration: 00m 12s)
15:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T110837 (duration: 00m 13s)
15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235482/ (duration: 00m 12s)
15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235479/ (duration: 00m 13s)
15:19 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235442/ (duration: 00m 12s)
15:14 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235441/ (duration: 00m 12s)
15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234942/ and https://gerrit.wikimedia.org/r/#/c/234944/ (duration: 00m 13s)
14:40 Nikerabbit: TTMServer reindex complete
11:59 mark: removed tools LV snapshots on labstore1002
11:47 mark: kill STOP'ed rsync on labstore1002
11:00 jynus: cloning mysql data from es1002 into es1016 [ETA:16h]
10:30 moritzm: installed qemu security updates on labvirt*
09:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002 (duration: 00m 12s)
09:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1010, pool es1017 (duration: 00m 13s)
09:19 hashar: Merged in "delete 1.26wmf12" https://gerrit.wikimedia.org/r/235347 which was left unmerged in Gerrit but was present on tin /srv/mediawiki-staging confusing people.
08:03 bblack: restarting ntp on lvs2004
08:01 moritzm: enable ferm on db1069/sanitarium
07:50 moritzm: enable ferm on remaining phabricator db hosts
04:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 2 04:54:37 UTC 2015 (duration 54m 36s)
02:52 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-02 02:52:51+00:00
02:50 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 09s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-02 02:29:56+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 31s)
00:33 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235366/ (duration: 00m 13s)

2015-09-01

23:59 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/221731/ (duration: 00m 13s)
23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235285/ (duration: 00m 14s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235362/ (duration: 00m 14s)
23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/235361/ (duration: 00m 13s)
22:50 awight: update CRM from 0fc8474338e7a31fdde79287bd667b98cd96a252 to abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b
22:18 MaxSem: Maps: creating and populating admin table
21:20 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/235177/ (duration: 00m 12s)
20:54 ori: restarted nutcracker on mw1142
20:33 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf21 (duration: 30m 37s)
20:03 dcausse: unfreezing elasticsearch indices
20:03 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf21
19:52 YuviPanda: removed tools20150901132642 from labstore vg on labstore1002
19:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/skins/SkinTemplate.php: cc643a0934: Deprecate unconditional loading of mediawiki.ui.button on all pages (duration: 00m 13s)
17:31 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
17:28 dcausse: freezing elasticsearch indices before applying ferm fules on master
17:23 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata: Fix for change dispatcher (duration: 00m 20s)
16:45 jynus: performing schema change on testwiki and metawiki
16:12 robh: policy.wikimedia.org dns change happening now
16:00 chasemp: ferm for elastic1003/2/1(master)
15:57 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235168/ (duration: 00m 13s)
15:51 YuviPanda: stopped replicate-tools on labstore1002, and cleaned out lockdir
15:47 logmsgbot: reedy@tin Synchronized php-1.26wmf20/extensions/SecurePoll/: Stop cronspam (duration: 00m 13s)
15:47 mark: labstore1002: echo 10000 > /sys/block/md123/md/sync_speed_min
15:44 mark: labstore1002: update-initramfs -k all -u
15:38 mark: labstore1002: mdadm /dev/md/slice51 --add /dev/sd{bh,bg,bf,be,bd,bc}
15:36 moritzm: disabled ferm in analytic1028, needs some more work on possibly dynamic mapreduce ports
15:16 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sd{bb,ba,az}
15:14 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sdaw
15:07 mark: labstore1002: mdadm --zero-superblock /dev/sd{aw,bh,bg,bf,be,bd,bc,bb,ba,az}1
15:04 moritzm: enabled ferm in analytic1028 (initial hadoop worker)
15:04 mark: labstore1002: mdadm --zero-superblock /dev/sdax1 && mdadm /dev/md/slice15 --re-add /dev/sdax
15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231465/ - VE for all new enwiki accounts (duration: 00m 13s)
14:58 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sday
14:58 mark: labstore1002: mdadm --zero-superblock /dev/sday1
14:53 mark: labstore1002: mdadm --stop /dev/md3
14:37 ebernhardson: reset elasticsearch cluster.routing.allocation.disk.high back to 90%
13:38 logmsgbot: krinkle@tin Synchronized w/: Remove rl-test.php (duration: 00m 13s)
13:17 moritzm: enabled ferm on db1048
13:09 moritzm: enabled ferm on labsdb100[467]
12:01 YuviPanda: disable puppet on labsdb1006
08:58 moritzm: enabled ferm on labsdb1001
08:58 godog: fixup current graphite retention for metrics under "servers" hierarchy T96662
08:51 moritzm: enabled ferm on labsdb1002
08:31 moritzm: enabled ferm on labsdb1003
08:29 godog: repool mw1125 mw1142 after nutcracker failures
07:45 jynus: cloning mysql data from es1010 to es1017 [ETA: 6h]
07:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1010 (duration: 00m 12s)
07:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1007, pool es1013 (duration: 00m 13s)
06:36 mutante: uploaded survey2012 to dumps/dataset1001; ownership as it is for survey2011; - T110746 in time for midnight PST
05:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 1 05:18:09 UTC 2015 (duration 18m 8s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-01 02:28:30+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 00s)

2015-08-31

23:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233665/ (duration: 00m 11s)
23:49 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: reenable config changes for cirrus experimental completion api (duration: 00m 12s)
23:40 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/EducationProgram: 97ab82eab2: Updated mediawiki/core Project: mediawiki/extensions/EducationProgram 85a7d3932c1a4ad28f1a8dd05704f4e524152349 (duration: 00m 14s)
23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf20/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: revert update for cirrussearch experimental suggestions api (duration: 00m 12s)
23:21 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: update config of cirrussearch experimental suggestions api (duration: 00m 12s)
22:45 chasemp: disabled puppet on elastic hosts temporarily to safely roll out fw change. elastic seems to have not taken it well and I'm holding for green cluster state.
21:20 mutante: installing package upgrades on argon
20:58 ori: imported pybal_1.08_amd64.changes to jessie-wikimedia
20:44 chasemp: ferm for elastic100[4-7] and adjust ferm to include wikitech source
20:21 subbu: deployed parsoid version c3e4df5e
16:22 godog: depool mw1125 + mw1142 from api, nutcracker client connections exceeded
16:06 logmsgbot: thcipriani@tin Finished scap: SWAT: Ask the user to log in if the session is lost gerrit:234228 (duration: 27m 07s)
15:59 jynus: restarting hhvm on mw2187
15:39 logmsgbot: thcipriani@tin Started scap: SWAT: Ask the user to log in if the session is lost gerrit:234228
15:33 mutante: terbium - Could not find dependent Service[nscd] for File[/etc/ldap/ldap.conf]
15:28 logmsgbot: thcipriani@tin Synchronized closed-labs.dblist: SWAT: Creating closed-labs.dblist and closing es.wikipedia.beta.wmflabs.org gerrit:234594 (duration: 00m 13s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Remove files from Commons from search results on wikimediafoundation.org gerrit:234040 (duration: 00m 11s)
15:25 ottomata: starting varnishkafka instances on frontend caches to produce eventlogging client side events to kafka
15:21 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - Fix formatting of client edit summaries gerrit:234991 (duration: 00m 21s)
15:16 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/controller/uw.controller.Step.js: SWAT: Keep the uploads sorted in the order they were created in initially gerrit:234553 (duration: 00m 12s)
14:43 ebernhardson: elasticsearch cluster.routing.allocation.disk.watermark.high set to 75% to force elastic1022 to reduce its disk usage
14:41 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
14:06 akosiaris: rebooted krypton. was reporting 100% cpu steal time
13:40 paravoid: running puppet on newly-installed mc2001
13:40 paravoid: restarting hhvm on mw1065
11:10 moritzm: restart salt-master on palladium
10:45 paravoid: reenabling asw2-a5-eqiad:xe-0/0/36 (T107635)
10:36 godog: repool ms-fe1004
10:32 godog: repool ms-fe1003 and depool ms-fe1004 for firewall changes
10:19 godog: update graphite retention policy on files with previous retention and older than 30d T96662
10:18 godog: repool ms-fe1002 and depool ms-fe1003 for firewall changes
10:05 godog: depool ms-fe1002 to apply firewall changes
09:55 jynus: cloning es1007 mysql data into es1013 (ETA: 5h30m)
09:51 godog: repool ms-fe1001
09:35 godog: depool ms-fe1001 in preparation for ferm changes
09:27 godog: update graphite retention policy on files with previous retention and older than 60d T96662
09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1007 for maintenance (duration: 00m 13s)
08:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 12s)
04:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 31 04:34:14 UTC 2015 (duration 34m 13s)
04:05 bblack: disabled ipv6 autoconf on neon, flushed old dynamic addr
02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-31 02:32:25+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 42s)

2015-08-30

12:58 godog: lvchange -ay labstore/others on labstore1002
12:52 godog: start-nfs on labstore1002
12:31 godog: lvchange -ay labstore/tools on labstore1002
12:30 godog: also disabled puppet on labstore1002 while investigating
12:15 godog: trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose
11:19 YuviPanda: powered labstore1002 back up
11:17 YuviPanda: shut down labstore1002, going to powercycle from mgmt
10:34 YuviPanda: disabled backups on labstore1002 to prevent overwriting of good backups on 2001
10:08 YuviPanda: rebooted labstore1002
04:16 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s)

2015-08-29

15:26 jynus: killing idle mysql connections from phabricator and setting wait and interactive timeout to 60
09:30 jynus: SCAP failed, cannot depool db1028
09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
09:05 jynus: about to depool db1028 due to disk issue
04:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s)
02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s)

2015-08-28

23:45 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s)
22:51 logmsgbot: bd808@tin Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s)
22:05 ori: disabling puppet on tin for a few minutes to test an ssh-agent-proxy change
20:04 logmsgbot: catrope@tin Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s)
18:09 robh: updating ldap-codfw cert
17:10 logmsgbot: catrope@tin Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s)
17:08 urandom: bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings
16:03 chasemp: ferm for elasticsearch10(0[8-9|1[0-13])
15:31 awight: updated crm from fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8 to 0fc8474338e7a31fdde79287bd667b98cd96a252
15:23 chasemp: ferm for elasticsearch10[14-17]
11:09 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s)
10:29 godog: reenable puppet on ms-fe1, ferm changes will go out on monday
09:48 jynus: Cloning es1001 database into es1012
09:45 moritzm: enabled ferm for swift on esams
09:28 moritzm: enabled ferm on strontium puppetmaster backend
09:00 moritzm: enabled ferm on rhodium puppetmaster backend
08:29 moritzm: uploaded debdeploy 0.0.3 to carbon
08:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s)
05:59 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s)
04:58 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s)
01:59 Tim: on ruthenium: started parsoid_vd which was previously killed by oom-killer
01:58 Tim: on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things
01:37 Tim: on ruthenium restarted parsoid-rt-client and parsoid-vd-client
00:24 mutante: powercycled mw2027
00:19 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s)
00:06 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s)
00:05 Krenair: Another codfw host broke: mw2027
00:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s)

2015-08-27

23:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s)
23:57 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s)
23:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s)
23:30 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s)
23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
23:23 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s)
23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s)
20:11 chasemp: ferm setup on elasticsearch10(1[8-9|2[0-3])
20:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20
19:57 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s)
18:16 chasemp: setting up ferm on elastic1027-31
17:47 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 13s)
17:43 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/2 (duration: 00m 13s)
17:37 urandom: ack'd Cassandra process alert on restbase1001; temporary command args have pushed the class name beyond the limit
17:34 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
17:24 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 12s)
17:08 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:51 moritzm: ferm rules on logstash100[1-3] have been amended to allow grafana from reading dashboard configs
16:39 bd808: new ferm rules on logstash100[1-3] are blocking grafana from reading dashboard configs.
16:22 moritzm: ferm enabled on logstash1003
16:18 moritzm: ferm enabled on logstash1002
16:16 bd808: ferm enabled on logstash1001
16:06 bd808: logstash1001 back up after system reboot; we applied a default drop rule without applying the other iptables changes; will try again
15:58 chasemp: rebooting logstash1001.mgmt.eqiad.wmnet for moritz as it is having issues
15:47 bblack: killed hung ubuntu mirror rsync commands on carbon, from Jul 10
15:45 bd808: logstash1001 not responding over ssh following ferm rules application; moritzm investigating
15:30 bd808: Disabled puppet on logstash100[1-3] prior to trying to enable ferm
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable newarticle campaign in itwiki gerrit:234223 (duration: 01m 52s)
14:52 bblack: re-imaging lvs200[123]
14:47 godog: reenable puppet on ms-be1*
14:22 godog: disable puppet on ms-fe1 / ms-be1 in prepration for puppet work
14:15 godog: reenable puppet on ms-fe2*
13:47 bblack: re-imaging lvs2004 + lvs2005
13:29 ottomata: doing rolling restart of kafka brokers to apply auto_create_topics change
13:21 godog: enable puppet on ms-be2*
13:21 ottomata: stopping kafka on analytics1021, it is no longer a kafka broker.
13:09 godog: disable puppet on ms-be2* in preparation for firewall changes
13:09 jynus: cloning es1008 into es1014
13:04 ottomata: running leader election now that all topics and partitions are rebalanced across new kafka nodes
12:46 bblack: re-imaging lvs2006
12:45 andrewbogott: re-imaging labnet1001 (I hope)
11:33 _joe_: restarted hhvm on mw1143, locked in __lll_lock_wait for stat_cache deadlock
11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1011 for the first time, depool es1008 (duration: 00m 12s)
09:27 jynus: installing and configuring servers es1012-es1019
06:39 ostriches: tin: dropped useless "gerrit" remote from /srv/mediawiki-staging (uses ssh, lol), pointed {origin,readonly} at the actual repo instead of a redirect.
06:00 _joe_: powercycling mw2140, not responding to ping, blank console
03:17 awight: deploy config cleanup for paymentswiki
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 44s)
02:16 awight: push config change to the payments orphan slayer: explitly give stomp port to work around strict notice, clean up unused globals. T109911
01:32 ejegg: updated payments from 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b to 6ac552f280fb839069d117386c4ecbe9e52f90a8
00:31 twentyafterfour: finished phabricator upgrade, everything appears to be working
00:24 logmsgbot: aaron@tin Synchronized php-1.26wmf19/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
00:22 logmsgbot: aaron@tin Synchronized php-1.26wmf20/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
00:20 twentyafterfour: taking phabricator offline for scheduled upgrade

2015-08-26

23:59 Krinkle: mwscript deleteEqualMessages.php --wiki rowiki
23:57 yurik: git deployed tilerator - had the 4/5 issue - https://phabricator.wikimedia.org/T110434
23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234072/ (duration: 01m 12s)
23:37 logmsgbot: krenair@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234038/ (duration: 01m 12s)
23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234037/1 (duration: 01m 12s)
23:27 yurik: deployed kartotherian
23:21 jynus: cloning es1005 into es1011, ETA 9 hours
22:41 ori: armed keyholder on tin
22:40 ori: Disabled Puppet on mw1017 for 2hrs and applied I059b0c96c9 for testing.
21:55 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
21:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1005 (duration: 01m 12s)
21:40 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
21:32 ori: Disabling Puppet on tin again to test an ssh-agent-proxy change
20:30 logmsgbot: ori@tin Synchronized README: testing ssh-agent-proxy changes (duration: 00m 13s)
20:25 ori: Disabling puppet on tin and hacking some debug logging into ssh-agent-proxy
20:24 ori: armed ssh-agent key on mira
20:21 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 00m 03s)
20:11 subbu: deployed parsoid version 44d657de
19:52 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Echo/includes/mapper/EventMapper.php: https://gerrit.wikimedia.org/r/#/c/234082/ (duration: 00m 12s)
19:47 mutante: sodium - deleting shunted messages older than 7 days
19:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234042/ (duration: 00m 12s)
19:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/234024/ (duration: 00m 12s)
19:20 logmsgbot: krenair@tin Synchronized multiversion/MWWikiversions.php: https://gerrit.wikimedia.org/r/#/c/232672/ (duration: 00m 12s)
18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 11s)
18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 13s)
18:38 twentyafterfour: ^ stupid typo. That sync was group1 to 1.26wmf20
18:37 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: tig
18:31 logmsgbot: ori@tin Synchronized w/404.php: Ided1facc0: Remove auto-redirection from 404 page. (duration: 00m 13s)
17:51 ejegg: updated SmashPig from 258f2c917b1ae50b01231927bcd6f58ecaa8940b to fdb053efa617162ac9f695e493c390987a069140
17:30 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
17:12 andrewbogott: ok, /now/ I’m running a dist-upgrade on labcontrol1001, to sort out weird oslo dependencies
17:09 chasemp: adding firewall to elasticsearch2[4-6] (3 was just done as a pilot)
17:03 andrewbogott: upgraded labnet1002 nova services to Juno
16:34 andrewbogott: stopping keystone, updating db, restarting
16:18 andrewbogott: switching labcontrol1001 hiera to Juno which will add the cloud-archive repo for Juno.
16:11 andrewbogott: backing up labs openstack databases into /home/andrew/openstackdbbackups on db1009
16:11 andrewbogott: starting labs openstack update to Juno
15:53 moritzm: ferm enabled on elastic1023
15:45 godog: repool restbase1009 in pybal
15:28 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - wrap usage tracking batch updates in transaction gerrit:233970 (duration: 00m 23s)
13:47 andrewbogott: rebooting/reimaging labnet1001
13:11 mobrovac: restbase deploying 1dfba85
12:54 yurik: git synced kartotherian
11:02 jynus: dropping optin_survey_old table on all wikis
10:33 godog: reenable puppet on ms-fe/ms-be, base::firewall still not enabled
09:58 godog: test-reboot ms-be2001
08:17 godog: disable puppet on ms-be/ms-fe in preparation for merging firewall changes
07:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 26 07:53:31 UTC 2015 (duration 53m 30s)
07:01 jynus: restarting mw1239 HHVM, which is unresponsive
04:47 logmsgbot: ori@tin Synchronized wmf-config: I73721936: Enable ParsoidBatchAPI everywhere (duration: 00m 13s)
03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-26 03:11:29+00:00
03:06 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings-labs.php: Push labs config to keep in sync with master (duration: 00m 13s)
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 45s)
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-26 02:37:51+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 29s)
02:00 ottomata: kafka topic webrequest_upload has finished rebalancing across new brokers. starting move of last topic webrequest_text
01:50 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow for reply fix (duration: 00m 15s)
00:28 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
00:26 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
00:26 Danny_B: 2586dd1c7c obviously broke many pages
00:19 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 14s)
00:14 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I79ffa78fa: Collection/OCG: Turn on plain text output format in Book Creator (duration: 00m 12s)
00:12 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: 2586dd1c7c: Updated mediawiki/core Project: mediawiki/extensions/Scribunto (duration: 00m 13s)

2015-08-25

23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233860/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233872/ (duration: 00m 13s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
23:12 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
23:10 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
23:10 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233781/ (duration: 00m 12s)
22:20 cscott: updated Parsoid to version c3b037b0
22:10 ejegg: disabled paypal audit downloader and parser due to them warning of incorrect data
21:16 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 13s)
21:13 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Cite/modules/ext.cite.styles.css: 7344e02216: Updated mediawiki/core Project: mediawiki/extensions/Cite (duration: 00m 12s)
21:09 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 14s)
20:54 tgr: finished OAuth migration
20:34 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: make OAuth DB writable again T108648 (duration: 00m 12s)
20:32 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: change wgMWOAuthCentralWiki mediawikiwiki -> metawiki T108648 (duration: 00m 12s)
20:24 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: set OAuth to readonly for DB migration T108648 (duration: 00m 13s)
20:13 subbu: deployed parsoid version 759916fc
19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf20
19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf20 (duration: 50m 12s)
18:31 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf20
17:11 YuviPanda: run authdns-update on radon (ns0.wikimedia.org)
17:10 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:58 Krinkle: mwscript deleteEqualMessages.php --wiki kawiki
16:56 andrewbogott: restarting pdns on labcontrol1001 and labcontrol2001 to handle a nembus reboot
16:53 Krinkle: mwscript deleteEqualMessages.php --wiki huwiki
16:31 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki
16:17 Krinkle: mwscript deleteEqualMessages.php --wiki frpwiki
15:50 godog: powercycle ms-be1004, likely xfs
15:44 andrewbogott: dist-upgrade and rebooting nembus in an attempt to resolve this acpi_pad issue
15:36 Krinkle: mwscript deleteEqualMessages.php --wiki euwiki (T45917)
15:29 Krinkle: mwscript deleteEqualMessages.php --wiki eowiki (T45917)
15:07 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/233718/ (duration: 00m 16s)
13:56 jynus: dropping old tables on s7 - T5493
13:48 jynus: dropping old tables on s6 - T54932
12:53 Jeff_Green: authdns-update to change bismuth's IP
11:16 jynus: dropping old tables on s3 - T54932
10:46 jynus: dropping old tables on s2 - T54932
10:05 YuviPanda: restart puppetmaster on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/233184/
07:35 _joe_: stopping redis, wiping aof, restarting redis on rdb100{1,2} - snapshot saved on rdb1002:/root
07:12 _joe_: stopping redis on rdb1003,4, wiping AOF, restarting
06:38 jynus: performing schema change on officewiki, mediawikiwiki and metawiki
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 26s)
01:48 ottomata: starting move of kafka partitions for topic webrequest_upload to new brokers. this will take a while!
01:44 ottomata: restarting kafka on new brokers kafka1013,1014,1020 to apply increase in num.replica.fetchers

2015-08-24

23:46 logmsgbot: mattflaschen@tin Synchronized wmf-config: Remove wgFlowOccupyPages (duration: 00m 12s)
23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233636/ (duration: 00m 12s)
22:16 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: change OAuth DB on beta +enable writes (duration: 00m 12s)
21:55 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
21:54 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
21:42 akosiaris: enabled puppet on maps-test200{1,2,3,4}.codfw.wmnet
20:21 arlolra: updated Parsoid to version 0b2fbae7
18:58 bblack: reloading primary LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ ) + ulimit fixup ( https://gerrit.wikimedia.org/r/#/c/233484/ )
18:31 bblack: reloading backup LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ )
17:19 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf18
16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf17
16:05 andrewbogott: rebooting labnet1001
15:53 _joe_: restarted nutcracker on mw1010, holding a 150 GB deleted logfile
15:47 Krenair: running sync-common on mw1010 to bring it up to date after clearing some space
15:44 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf16
15:41 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf15
15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/233411/1 (duration: 00m 49s)
15:37 hashar: stopped and restarted Zuul
15:31 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232919/ and https://gerrit.wikimedia.org/r/#/c/232915/ (duration: 01m 34s)
15:29 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikiquote.png: https://gerrit.wikimedia.org/r/#/c/232919/ (duration: 02m 04s)
15:19 Krenair: No space left on mw1010, cannot ping or ssh to mw2180
15:16 logmsgbot: krenair@tin Synchronized docroot/noc/db.php: https://gerrit.wikimedia.org/r/#/c/232920/ (duration: 01m 34s)
15:14 hashar: apt-get upgrade on gallium
14:48 andrewbogott: forcing wikitech logouts in order to flush everyone’s service catalog
14:18 ottomata: starting to move kafka topic-partitions to new brokers (and off of analytics1021)
14:12 yurik: git deploy synced kartotherian
13:55 akosiaris: disable puppet on fermium preparing for reinstallation
13:55 akosiaris: disable puppet on fermium
12:54 akosiaris: stop etcd on etcd1002.eqiad.wmnet. Already removed from the cluster
11:58 _joe_: stopping etcd on etcd1001
11:50 _joe_: restarting etcd on etcd1001
09:00 YuviPanda: starting up replicate for tools on labstore1002
09:00 YuviPanda: cleaning up lockdir on labstore for maps and tools
09:00 YuviPanda: others replication on labstore1002 completed successfuly
08:31 YuviPanda: cleaned up others lockdir for replication on labstore1002 and started it manually
06:43 jynus: reloading dbproxy1003 service
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 36s)

2015-08-23

16:54 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 23s)

2015-08-22

23:08 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php: (no message) (duration: 01m 05s)
19:41 YuviPanda: manually remove old snapshots from labstore1002
17:28 chasemp: tweaking apache on iridum T109941
16:45 chasemp: scratch that as we have mpm_prefork enabled :)
16:33 chasemp: raising values in mpm_worker.conf for iridium to to debug and hopefully head off further crashing
14:44 twentyafterfour: restarted apache2 on iridium. Segfault again. This time I at least got one clue in the log: "zend_mm_heap corrupted"
09:18 twentyafterfour: phabricator seems stable now, restarting apache2 on iridium did the trick, unfortunately we didn't learn why
08:36 twentyafterfour: restarted phd on iridium
08:36 twentyafterfour: restarted apache2 on iridium
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 09s)
00:26 mutante: deleting blog.sh and blog_pageviews crontab from stat1003

2015-08-21

23:34 urandom: restarting Cassandra on restbase1001 to restore baseline settings
23:11 yurik: synced kartotherian
22:35 mutante: deleting held messages on mailman that are older than 1 year
21:56 awight: increasing paymentswiki orphan gc-cc-limbo expiry time to 30 days
21:45 mutante: had to reset list creator password for mailman - ask me if you think you should have it and don't (this is not the master pass)
20:37 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes: I1eb8dfc: Revert Count API and hook calls, with 1:1000 sampling (duration: 01m 09s)
19:43 awight: update paymentswiki from 2b08853c977eee0fd17bf00a673a3bbf2a146554 to 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b
18:58 awight: disabling Amazon gateway
18:52 awight: updated paymentswiki from 049ad15323564fd5cd7f5efcadddb532a3590cef to 2b08853c977eee0fd17bf00a673a3bbf2a146554
16:06 jynus: checksumming dewiki database, higher write rate/dbstore lag expected temporarily
15:10 ottomata: rebooting kafka broker analytics1021 to hopefully reload /dev/sdg with new disk, also will turn on hyperthreading
14:13 ottomata: rebooting analytics1056 after upgrading kernel to linux-image-3.13.0-61-generic
13:58 urandom: restarting restbase1001 to apply temporary GC setting
13:34 ottomata: stopping kafka broker on analytics1021 due to bad disk.
13:30 bblack: wiped ganglia apache access log on uranium, to free up half of the (full) rootfs
10:07 godog: enable puppet on ms-fe1/ms-be1
09:49 godog: disable puppet on ms-fe1/ms-be1 before merging https://gerrit.wikimedia.org/r/#/c/231240/
07:06 _joe_: restarting gitblit, because it will be decommissioned "soon"...
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 11m 19s)

2015-08-20

23:40 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf19/extensions/CirrusSearch/: Fix some cirrussearch logspam (duration: 00m 13s)
23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232854/ (duration: 00m 13s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232671/ (duration: 00m 12s)
23:15 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/LiquidThreads/classes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/232783/ (duration: 00m 12s)
23:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/resourceloader/ResourceLoaderFileModule.php: T102578 (duration: 00m 13s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232782/ (duration: 00m 12s)
22:48 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/libs/CSSMin.php: Icc1c23a2: CSSMin: remove dot segments in relative local URLs (duration: 00m 12s)
21:36 cscott: updated Parsoid to version db6e6404f67a9f971b4fbefe9de239735426c738
21:25 matt_flaschen: Ran FlowUpdateRevContentModelFromOccupyPages.php on all wikis
20:41 twentyafterfour: scap failed to sync to mw2180.codwf.wmnet
20:41 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf19
20:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: Silence the undefined index error in CirrusSearch (duration: 06m 24s)
19:40 chasemp: moving enwiki_content_1432182861 elastic shard from 1022 to 1004 due to space (1022 is at 91%)
20:57 mutante: no log bot
18:56 mutante: labvirt1007 "only" 29G space left - but since we have 2.2T there that means 99% full
17:39 ottomata: stopping kafka on analytics1018 and bringing it down for reinstall as kafka1018 with Jessie
16:38 YuviPanda: puppet swat done
15:44 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-1
15:43 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232729/ (duration: 00m 12s)
15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232730/ (duration: 00m 13s)
15:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
15:38 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
15:32 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232687/ (duration: 00m 13s)
15:31 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232688/ (duration: 00m 11s)
15:27 greg-g: on mw2187: rsync: failed to set times on "/srv/mediawiki/wmf-config": Read-only file system (30)
15:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231464/ (duration: 00m 13s)
15:08 urandom: restarting restbase1001 to apply temporary heap size of 12G
15:02 jynus: performing online schema change on wikidata
15:00 andrewbogott: rebooting labvirt1008
12:48 jynus: restarted nutcracker on mw1142
12:08 godog: reenable puppet on ms-fe1/ms-be1
12:04 godog: repool ms-fe1001
11:53 godog: depool ms-fe1001 to test a reboot
11:45 godog: disable puppet on ms-fe/be1 in preparation to apply https://gerrit.wikimedia.org/r/#/c/231237
12:08 kart_: Updated cxserver to e221462
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 27s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-20 02:45:14+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 10m 41s)

2015-08-19

23:27 logmsgbot: rmoen@tin Synchronized wmf-config/InitialiseSettings.php: Remove reference to lost wikitech apple touch icon file (duration: 00m 13s)
23:21 logmsgbot: rmoen@tin Synchronized php-1.26wmf19/extensions/TimedMediaHandler/: Re-disable 2-pass Theora encoding temporarily
23:16 logmsgbot: rmoen@tin Synchronized php-1.26wmf18/extensions/Flow: Add debugging code to detect and workaround type hint failure (duration: 00m 14s)
21:20 robh: livehack reverted sodium back to normal, testing done
21:08 robh: disabled puppet on sodium for livehacking tests for T109609
21:04 andrewbogott: disabling puppeton labnet1001 and labnet1002
20:46 urandom: restarting Cassandra on restbase1001 to enable -XX:+PrintAdaptiveSizePolicy
20:43 urandom: disabling puppet on restbase1001 to temporarily enable additional GC logging
20:17 subbu: deployed parsoid version 8d617c99
20:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232580/ (duration: 00m 12s)
19:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232565/ (duration: 00m 12s)
19:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232543/ (duration: 00m 13s)
18:59 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: deploy hotfix for Wikidata: https://gerrit.wikimedia.org/r/#/c/232556/ (duration: 02m 39s)
18:30 ottomata: starting reinsatll of analytics1022 -> kafka1022 as jessie
18:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf19
16:26 ottomata: added analytics105[012456] into hadoop cluster as worker nodes
16:02 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temp disable notifications for cx gerrit:232504 (duration: 00m 13s)
15:57 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temporarily disable notifications for cx gerrit:232505 (duration: 00m 12s)
15:51 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232484 (duration: 00m 12s)
15:35 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232485 (duration: 00m 13s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust mediawiki.org RSS whitelist to allow technology blog feeds gerrit:118956 (duration: 00m 13s)
15:00 andrewbogott: rebooting labvirt1007
13:58 godog: stop puppet on ms-fe1* while merging swift refactoring
13:58 godog: stop puppet on ms-be1* while merging swift refactoring
09:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 for normal traffic only (duration: 00m 13s)
09:49 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Add *.webarchive.org.uk to wgCopyUploadsDomains whitelist (duration: 00m 12s)
08:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 19 08:28:19 UTC 2015 (duration 28m 18s)
07:39 jynus: About to perform a schema change on flowdb
03:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
03:42 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-19 03:14:06+00:00
03:07 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 48s)
02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-19 02:39:54+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 07m 51s)
01:10 yurik: updated kartotherian
00:40 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow 1.26wmf19 for RC insert failure. (duration: 00m 15s)
00:39 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for RC insert failure. (duration: 00m 14s)

2015-08-18

23:59 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: T59040 (duration: 00m 12s)
23:37 mutante: added papaul (pt1979) to WMF LDAP group
23:08 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for watchlist fix. (duration: 00m 14s)
23:02 logmsgbot: hoo@tin Synchronized wmf-config/: Set $wgPropertySuggesterClassifyingPropertyIds for testwikidata (duration: 00m 14s)
22:28 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: https://gerrit.wikimedia.org/r/#/c/232385/ (duration: 00m 12s)
22:17 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/OutputPage.php: 1a4f1df2fe (duration: 00m 12s)
21:22 awight: updated paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to fca36026b1e90298abd93562803d3ea7d6893d96
19:26 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf19
19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf19 (duration: 51m 01s)
18:30 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf19
18:11 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: 6ee94ca47c: Load all CSS in the top queue (duration: 00m 13s)
18:07 robh: sodium returned to normal, mailman window over.
17:38 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes: 91ae6a39df, 4cc9622214: Added wfTransactionalTimeLimit() method and applied it; Try to make POSTs as transactional as possible (duration: 00m 16s)
17:21 robh: T108099 complete, mailman restarted for a few minutes while i prepare next task.
17:17 robh: puppet disabled on sodium, no touch.
17:03 robh: mailman maint window starts now, list delivery will remain sporadic until I finish. (It'll work off and on, no messages should be lost)
15:42 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232241/ (duration: 00m 13s)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove extra transcode enablings gerrit:232228 (duration: 00m 13s)
15:04 andrewbogott: rebooting labvirt1006
08:18 _joe_: reimaging mw1152
08:14 godog: restart cassandra on restbase100[569] to pick up latest openjdk
08:04 _joe_: depooling mw1152 from the imagescalers pool
08:03 godog: restart cassandra on restbase100[348] to pick up latest openjdk
07:23 legoktm: live hacking on mw1017 for T109236
05:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 18 05:45:47 UTC 2015 (duration 45m 46s)
02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-18 02:25:28+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 50s)
00:02 ottomata: analytics1041 down, attempting power cycle

2015-08-17

22:19 matt_flaschen: LQT->Flow done on MediaWiki.org.
21:57 logmsgbot: mattflaschen@tin Synchronized wmf-config: LQT->Flow: Make frozen wikis no longer able to create LQT pages (duration: 00m 13s)
21:31 chasemp: remove php5-xdebug from terbium per mattflaschen
21:10 MaxSem: renamed Gadget:Invention, Travel, & Adventure --> Gadget Invention, Travel, & Adventure on enwiki using moveBatch.php to work around a permissions screwup
20:54 bd808: T109369: Restarted logstash on logstash1003; parsoid gelf events not being recorded since 2015-08-15
20:16 subbu: deployed parsoid version 4b656b72
19:19 ottomata: stopping kafka on analytics1012, preparing to reinstall with Jessie and rename to kafka1012
15:44 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232051/ - remove WikiGrok from extension-list, extension is no longer deployed (duration: 00m 11s)
15:40 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/231982/2 - clear up --wiki usage to mwscript (duration: 00m 12s)
15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/232048/ (duration: 00m 11s)
15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231984/ (duration: 00m 13s)
15:05 andrewbogott: rebooting labvirt1004
15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232021/ (duration: 00m 12s)
14:30 mobrovac: restbase updated production cluster to ed17952
14:12 mobrovac: restbase deployed ed17952 on restbase1001
13:58 mobrovac: restbase deploying ed17952 on staging
11:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting change to settings before schema change (no more lag) (duration: 00m 12s)
11:29 jynus: reloading dbproxy1003 haproxy config- it was a temporal max_connections issue; db1043 should be the canonical server again
10:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Solving lag issues while schema change is ongoing (duration: 00m 12s)
09:43 jynus: about to perform schema change on centralauth
09:36 godog: upgrade openjdk on restbase100[127] and restart cassandra
05:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 17 05:54:24 UTC 2015 (duration 54m 23s)
05:11 twentyafterfour: restarted phd to pick up new configuration, (and to silence the phabricator 'setup issue' warning
04:47 twentyafterfour: changed phabricator policy for the multimeter application from 'public' to 'all users'
04:44 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/231983/ to iridium and restarted apache
04:17 mutante: free some disk space on iridium. apt-get clean; gzip /var/log/account/pacct.0; some apache logs .;.
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-17 02:29:19+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 08s)

2015-08-16

18:15 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 21s)
18:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 25s)
05:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 16 05:39:28 UTC 2015 (duration 39m 27s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-16 02:29:19+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 52s)

2015-08-15

22:56 legoktm: removed 13 bounce_records for User:odder from bouncehandler database
16:07 _joe_: removing manually core dumps from last night's outage on all appservers in eqiad, they occpy on average 30 GB/server
16:05 ottomata: starting rolling restart of kafka brokers to apply auto leader rebalance enable = false
14:49 ottomata: stopping kafka broker on analytics1012 to again try to figure out why camus can't consume from it
12:46 bblack: restarted gitblit on antimony, because Java is Awesome
05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 15 05:41:57 UTC 2015 (duration 41m 56s)
04:38 andrewbogott: killing some rsync processes on labstore1002 because iowaits are through the roof
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-15 02:29:05+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 37s)
01:15 ottomata: starting broker on analytics1012, camus wasn't happy about that either. hrm.
00:58 ottomata: stopping kafka broker on analytics1012, it is causing consumption problems with camus, will look into why later.

2015-08-14

23:47 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/libs/ReplacementArray.php: (no message) (duration: 00m 27s)
21:54 mutante: restarted nutcracker on mw1010
21:26 ori: deployed job runner 808d1ae08d40
21:15 ejegg: updated crm from 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1 to fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8
19:29 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: f72009a543: ResourceLoader: apply minify-js filter to config scripts (duration: 00m 13s)
18:27 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/MultimediaViewer: 9ee0437bc6: Updated mediawiki/core Project: mediawiki/extensions/MultimediaViewer 645b6c9e93fae13e09e5b493547aecc5a2e933ae (duration: 00m 12s)
18:24 ori: Repooling mw1041 now that T108601 is resolved.
17:59 yurik: deployed latest kartotherian
14:59 andrewbogott: rebooting labvirt1003
13:53 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1042 (vslow, dump) (duration: 00m 12s)
13:14 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: Fix ElasticaQuery logspam (duration: 00m 13s)
13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/GeoData: Fix ElasticaQuery logspam (duration: 00m 13s)
13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Flow: Fix ElasticaQuery logspam (duration: 00m 13s)
12:30 jynus: Restarting db1042 after data import
12:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Load-balance db1036 roles (duration: 00m 11s)
11:57 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Translate: Stop calling deprecated Elastica function (duration: 00m 13s)
08:26 akosiaris: upgraded and restarted apertium on sca100{1,2}
08:11 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.1+svn~61425-1
07:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 14 07:10:37 UTC 2015 (duration 10m 36s)
06:34 Jamesofur: reset email/password for User:Auréola after multi factor user confirmation.
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-14 02:35:19+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 35s)
01:49 matt_flaschen: Resumed LQT->Flow conversion of mw:Project:Support_desk on mw1041
01:46 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: I5e6c79c70: Optimize the order of styles and scripts in <head> (duration: 00m 12s)

2015-08-13

23:45 awight: rollback paymentswiki from 2e7b449224317779d53ff84527166c0d378a0a40 to 823393264d6795bbaec490ff86f17580f722e598
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/228477/ and https://gerrit.wikimedia.org/r/#/c/231074/ (duration: 00m 12s)
23:15 awight: update paymentswiki-staging 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 65b05fc11896325ae9749318b296c4396a64f649
23:15 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/mrwikibooks.png: https://gerrit.wikimedia.org/r/#/c/228477/ (duration: 00m 12s)
23:11 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/231450/1 (duration: 00m 14s)
23:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227330/ (duration: 00m 12s)
22:35 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
22:29 awight: update paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to 2e7b449224317779d53ff84527166c0d378a0a40
22:23 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 2e7b449224317779d53ff84527166c0d378a0a40
22:14 Krenair: Running refreshLinks --dfn-only in a screen on terbium for T44180
21:45 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
21:45 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 96a369651c1130b0a8e53a6395f83c0b9329b9f8
21:43 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Roll back: Test impact of I5e6c79c (duration: 00m 12s)
21:37 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Test impact of I5e6c79c (duration: 00m 12s)
21:16 mutante: torrus broken - doing https://wikitech.wikimedia.org/wiki/Torrus#Deadlock_problem
21:14 mutante: service gitblit restart on antimony (maybe that should be paging :)
21:06 bd808: `sudo /etc/init.d/ganglia-monitor restart` on logstash100[1-6] fixed ganglia data loss
20:46 mutante: killed ganglia aggregator for logstash on carbon
20:40 bd808: ganglia not getting elasticsearch jvm data for logstash cluster since 2015-08-13T12:00 -- https://ganglia.wikimedia.org/latest/?c=Logstash+cluster+eqiad&&m=es_heap_used
19:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf18
19:38 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: (no message) (duration: 00m 11s)
19:35 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 10s)
19:31 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 11s)
18:32 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/SyntaxHighlight_GeSHi/extension.json: If0851400: Fix-up for I2de8a400d: explicitly declare module position (duration: 00m 12s)
17:47 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
17:46 logmsgbot: kaldari@tin Finished scap: (no message) (duration: 50m 05s)
16:55 logmsgbot: kaldari@tin Started scap: (no message)
16:54 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner off (duration: 00m 11s)
16:51 logmsgbot: kaldari@tin Synchronized wmf-config/CommonSettings.php: syncing CommonSettings for WikidataPageBanner (duration: 00m 12s)
16:50 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
16:31 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/.gitmodules: (no message) (duration: 00m 11s)
16:30 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/.gitmodules: (no message) (duration: 00m 13s)
16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
16:24 logmsgbot: demon@tin Synchronized wmf-config/: undeploy wikigrok (duration: 00m 12s)
16:12 yurik: sync deployed tilerator
15:20 logmsgbot: thcipriani@tin Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231229 (duration: 00m 11s)
15:10 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231230 (duration: 00m 12s)
15:04 andrewbogott: rebooting labvirt1002
13:36 jynus: kill custom query hiting s6 master from terbium. Use of a slave is required.
13:11 andrewbogott: graceful’d apache2 on labcontrol1001
12:42 andrewbogott: restarted keystone on labcontrol1001
08:16 _joe_: removing all stale aggregator configs from netmon1001
08:15 godog: upgrade cassandra on restbase1009
08:11 godog: upgrade cassandra on restbase1006
08:07 godog: upgrade cassandra on restbase1005
08:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 13 08:00:40 UTC 2015 (duration 0m 39s)
07:36 _joe_: killing all gmond instances on netmon1001, trying to fix ganglia-monitor-aggregator
06:48 matt_flaschen: Stopped Support desk LQT->Flow conversion for tonight
05:03 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
05:02 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-13 03:04:52+00:00
03:01 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 14s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-13 02:44:49+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 47s)
01:26 matt_flaschen: Restarted conversion of support desk from LQT->Flow using convertLqtPageOnLocalWiki.php, using hhvm on mw1041
01:16 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I482b120289: Ensure all Xenon records begin with the script base name (duration: 00m 12s)
01:07 ori: Depooled mw1041 so it can be set aside for LQT->Flow conversion script (T108601)

2015-08-12

23:55 ori: fluorine is struggling due to I941660b5; I'm fixing.
23:54 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231194/ (duration: 00m 12s)
23:47 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231193/ (duration: 00m 12s)
23:44 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231169/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 11s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 13s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/230729 (duration: 00m 13s)
23:02 logmsgbot: legoktm@tin Synchronized wmf-config: Isolate wikidata.org cookies and CORS policies (duration: 00m 12s)
22:21 matt_flaschen: Killed support desk conversion again to review XDebug information.
22:17 matt_flaschen: Resumed the support desk conversion.
21:53 awight: updated paymentswiki from 325640bd70680a08ae77fd117433565634a98d88 to 99e3ce08117d18b15bc8138b447c4c21bd452d28
20:42 subbu: deployed parsoid version a271c205
20:23 milimetric: deployed the latest EventLogging master to eventlog1001
14:34 godog: upgrade cassandra on restbase1008
14:30 godog: upgrade cassandra on restbase1004
14:22 godog: upgrade cassandra on restbase1003
13:00 akosiaris: disabled puppet on maps-test200X
12:36 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access on dewiki, frwiki, jawiki and s3 wikis (duration: 00m 12s)
12:21 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-labs.php: Add Wikisource badge config (duration: 00m 13s)
12:20 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Add Wikisource badge config (duration: 00m 11s)
10:27 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: depool db2040 (duration: 00m 11s)
07:47 _joe_: restarted apertium-apy on sca1001 and sca1002, too many open files, probably leaking
06:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 12 06:12:31 UTC 2015 (duration 12m 30s)
05:54 matt_flaschen: Killed support desk conversion. Will resume with profiling tomorrow.
04:45 bblack: starting slow "apt-get -y upgrade" on cp* (mostly, nginx -> +wmf2), will execute over ~18-24h
03:34 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/229608 (duration: 00m 12s)
03:06 ori: Installing xdebug on terbium so matt_flaschen can debug memory leak
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-12 03:04:47+00:00
02:58 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 13s)
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-12 02:30:14+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 52s)
01:22 mutante: zirconium - shut down, i'm sure, mollyguard
00:22 yurik: updated kartotherian
00:09 mutante: restarted gitblit
00:06 logmsgbot: ori@tin Synchronized php-1.26wmf17/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 32e5bcf90c702 (duration: 00m 13s)
00:06 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 3ab0b7e0f4948 (duration: 00m 12s)
00:02 matt_flaschen: Resumed convertLqtPageOnLocalWiki.php run on MediaWiki.org's Project:Support_desk.

2015-08-11

23:19 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for memory leaks (duration: 00m 14s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230805/ (duration: 00m 12s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230804/ (duration: 00m 12s)
21:03 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf18
21:02 twentyafterfour: deployed scap fixes for my dumb mistakes
20:10 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/objectcache/MultiWriteBagOStuff.php: 0acfe6a5bb: Fix argument handling in MultiWriteBagOStuff::get() (duration: 00m 12s)
19:28 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 13s)
19:28 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 11s)
19:26 logmsgbot: catrope@tin Synchronized php-1.26wmf18/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: Fix missing editor switcher (duration: 00m 12s)
18:58 logmsgbot: twentyafterfour@tin Finished scap: again: sync new branch 1.26wmf18 and update testwiki (duration: 04m 58s)
18:53 logmsgbot: twentyafterfour@tin Started scap: again: sync new branch 1.26wmf18 and update testwiki
18:44 logmsgbot: twentyafterfour@tin scap failed: OSError [Errno 1] Operation not permitted: '/srv/mediawiki-staging/wikiversions.php' (duration: 29m 27s)
18:37 mutante: grafana switched to node krypton (jessie/VM)
18:21 bd808: logstash log event volume back to normal levels following elasticsearch upgrade
18:15 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.26wmf18 and update testwiki
18:06 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1006
18:03 bd808: upgraded elasticsearch to 1.7.1 on logstash1006; logstash-2015.08.11 shard recovering
18:02 bd808: upgrading elasticsearch on logstash1006
18:01 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1005
17:43 bd808: log event volume in logstash dropped dramatically again; seems to correlate with final recovery of logstash-2015.08.11 shard
17:29 bd808: upgraded elasticsearch to 1.7.1 on logstash1005; logstash-2015.08.11 shard recovering
17:28 bd808: upgrading elasticsearch on logstash1005
17:27 bd808: logstash event volume recovered after restarting all 3 logstash services
17:14 bd808: log event volume in logstash dropped dramatically at 16:49; investigating
17:13 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1004
16:42 bd808: upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering
16:42 mutante: restarted Apache on Etherpad
16:38 bd808: upgraded elasticsearch to 1.7.1 on logstash1003
16:37 bd808: upgraded elaasticsearch to 1.7.1 on logstash1002
16:36 bd808: upgraded elaasticsearch to 1.7.1 on logstash1001
16:23 bd808: logstash upgrade on logstash1003 complete
16:20 bd808: logstash upgrade on logstash1002 complete
16:16 bd808: logstash upgrade on logstash1001 complete
15:50 jynus: nuking db1002-db1007 on icinga
15:49 bd808: upgrading logstash on logstash1001
15:47 bd808: Trebuchet deploy of logstash/plugins: Add logstash-filter-prune 0.1.5 (36144b2)
15:36 bd808: Disabled puppet on logstash100[1-3] in preparation for upgrade to 1.5.3
15:31 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (actually rebased this time) (duration: 00m 11s)
15:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (actually rebased this time) (duration: 00m 11s)
15:17 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touched wmf-config/InitialiseSettings.php (duration: 00m 13s)
15:12 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (duration: 00m 12s)
15:11 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (duration: 00m 12s)
10:45 jynus: general maintenance on db1042 (restart, upgrade, db reconstruction)
10:38 godog: upgrade cassandra on restbase1007
10:31 godog: upgrade cassandra on restbase1002
10:25 godog: upgrade cassandra on restbase1001
09:56 paravoid: switched routing-system autonomous-system to eqiad's subAS on cr1-eqiad/cr2--eqiad
09:09 godog: reboot ms-be2009, cpu soft lockup
05:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 11 05:27:33 UTC 2015 (duration 27m 32s)
02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-11 02:26:58+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 48s)
01:20 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: Revert resourceloader: Add must-revalidate to Cache-Control (duration: 00m 12s)
00:10 mutante: apache restart on krypton

2015-08-10

23:53 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: Limit the number of states in a cirrussearch query (duration: 00m 11s)
23:44 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/TimedMediaHandler/TimedMediaIframeOutput.php: https://gerrit.wikimedia.org/r/#/c/230656/ (duration: 00m 12s)
23:40 logmsgbot: ori@tin Synchronized multiversion/MWMultiVersion.php: I511999: Convert multiversion scripts to use wikiversions.php (duration: 00m 12s)
23:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:07 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/228040/ (duration: 00m 14s)
23:04 ori: deployed scap a404a39b32... Build wikiversions.php in addition to wikiversions.cdb
22:22 mutante: restart gitblit
21:53 mutante: krypton unloaded mod proxy_balancer
21:30 awight: updated paymentswiki from af16d371f9c46d4f0b78986080f2a2be3226ace8 to 325640bd70680a08ae77fd117433565634a98d88
20:57 logmsgbot: ori Synchronized php-1.26wmf17/includes/cache/MessageCache.php: I2089b21fc: MessageCache: use APC for local caching, rather than files (duration: 00m 12s)
20:12 subbu: deployed parsoid version 7b554ce2f
19:57 yurik: synced new kartotherian
19:22 logmsgbot: ori Synchronized php-1.26wmf17/includes: I9a1aa76de: Moved ObjectCacheSessionHandler renewal logic to wfSetupSession() (duration: 00m 16s)
17:12 akosiaris: stopped postgres on maps-test200{2,3,4}
16:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (2/2) (duration: 00m 11s)
16:58 logmsgbot: ori Synchronized php-1.26wmf17/resources/Resources.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (1/2) (duration: 00m 11s)
16:50 logmsgbot: ori Synchronized php-1.26wmf17/extensions/wikihiero: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/wikihiero (duration: 00m 12s)
15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Enable scrubWikitext=1 in HTML to wikitext conversion using parsoid gerrit:230381 (duration: 00m 13s)
14:47 akosiaris: running an rsync from nas1001-a to local disks on helium
14:42 ottomata: restarted all varnishkafka instances to pick up proper confs (puppet should have done this!)
14:29 ottomata: starting upgrade of existing kafka cluster to 0.8.2.1 jessie - https://etherpad.wikimedia.org/p/kafka_0.8.2.1_migration2
12:38 bblack: deployed nginx-1.9.3-1+wmf2 to cp1065, cp1070, cp1071 (1x each text, upload, misc) for validation
11:16 logmsgbot: hoo Synchronized wmf-config/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 12s)
11:16 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 20s)
09:35 paravoid: manually firewalled backup4001 TCP on neon to temporarily stop the nsca alert storm
09:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1042 for maintenance (duration: 00m 12s)
08:43 _joe_: manually running logrotate on iridium
07:44 godog: reboot ms-be2006, xfs hosed
07:27 akosiaris: rebooting backup4001
07:20 jynus: schema change on testwikidatawiki
06:52 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1035 weight (duration: 00m 13s)
05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 10 05:31:29 UTC 2015 (duration 31m 28s)
03:13 bblack: restarted apache2 on iridium JIC
03:13 bblack: rm /var/log/apache2/phabricator_access.log.1 on iridium (disk full, fixed for now)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-10 02:23:47+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 43s)

2015-08-09

17:57 urandom: issuing nodetool cleanup on restbase1006
12:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Set dispatchBatchChunkFactor to 10 for now (duration: 00m 20s)
06:20 twentyafterfour: restarted phabricator phd (just in case - the full partition may have caused the daemons to be in a broken state)
06:17 twentyafterfour: moved some log files on iridium into /srv/logs to free space on /
05:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 9 05:12:23 UTC 2015 (duration 12m 22s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-09 02:23:22+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 32s)

2015-08-08

14:58 urandom: issuing nodetool cleanup on restbase1005
14:57 urandom: issuing nodetool cleanup on restbase1007
05:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 8 05:30:22 UTC 2015 (duration 30m 21s)
04:20 urandom: issuing nodetool cleanup on restbase1008
03:00 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Hack: Don't write change rows where LENGTH(change_info) > 65500 (duration: 00m 21s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-08 02:26:20+00:00
02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 20s)
01:53 hoo: Deleted change 237365841 as well
01:37 hoo: Deleted changes 237357747 and 237363245 from wikidata's wb_changes
01:32 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: I664ba9b0af: Override document.writeln to prevent it from blanking pages (duration: 00m 13s)

2015-08-07

21:00 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/230227/ - should be a noop for prod (duration: 00m 12s)
19:40 logmsgbot: hoo Synchronized wmf-config/: Bump wgCacheEpoch for Wikidata (duration: 00m 13s)
19:29 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix WB spinner, UnresolvedRedirectException handling on client (duration: 00m 21s)
18:08 gwicke: switched restbase1001 to CMS temporarily, to gather metrics; will switch back to G1GC tonight
17:09 yurik: synced kartotherian to maps-test* servers again, restarted the service
16:00 jynus: repool db1035 database with low traffic
15:54 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: FIX: Use .attr() to set the resource attribute of image, while adapting gerrit:230101 (duration: 00m 11s)
15:10 yurik: synced kartotherian to maps-test* servers
15:10 gwicke: switched cassandra staging cluster back to G1GC / default puppet
13:40 moritzm: install pcre security updates on elastic*, analytics*, wtp*, db* and es*
13:40 urandom: starting nodetool cleanup on restbase1004 (see: T108083)
13:37 urandom: starting nodetool cleanup on restbase1002 (see: T108083)
12:07 moritzm: restarted HHVM on jobrunners/imagescalers in eqiad/codfw for libtidy/PCRE security updates
09:24 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
09:02 akosiaris: disabled helium as a poolcounter temporarily while applying base::firewall again
09:01 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
08:52 moritzm: restarted HHVM on API apaches in codfw for libtidy/PCRE security updates
08:26 godog: restart cassandra on test cluster
08:11 godog: upgrade cassandra test cluster to openjdk 8u66-b01-1~bpo8+1
07:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 7 07:34:34 UTC 2015 (duration 34m 33s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-07 02:43:35+00:00
02:40 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 15s)
02:22 bblack: pooled cp105[67],cp1069,cp1070 into eqiad misc caches
02:19 logmsgbot: krinkle Finished scap: Rebuild l10n for Gadgets after I2089b21fc (duration: 24m 28s)
01:54 logmsgbot: krinkle Started scap: Rebuild l10n for Gadgets after I2089b21fc
01:08 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Gadgets: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Gadgets (duration: 00m 13s)
00:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: c0ca5700c6: resourceloader: Restore anticipated loader states for hardcoded module requests (duration: 00m 12s)

2015-08-06

23:24 logmsgbot: ori Synchronized php-1.26wmf17: c5c52ec1d8: resourceloader: Async all the way (duration: 01m 41s)
23:20 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I053a6e9: Enable authmetrics logging on group0 wikis (duration: 00m 12s)
23:15 logmsgbot: ebernhardson Synchronized wmf-config/: Redeploy cirrussearch ab test start (duration: 00m 14s)
23:09 logmsgbot: ebernhardson Synchronized wmf-config/: Start cirrussearch suggester confidence AB test (duration: 00m 13s)
23:07 mutante: puppet/salt-master: signing certs and adding keys for fermium
23:05 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch: Bump cirrusearch in 1.26wmf17 for SWAT (duration: 00m 11s)
22:44 logmsgbot: ori Synchronized php-1.26wmf17/tests/phpunit/includes/OutputPageTest.php: (no message) (duration: 00m 13s)
22:32 mutante: starting new instance fermium on ganeti
22:31 ori: Previous two syncs were of I2089b21fc and I3f46fee7c
22:31 logmsgbot: ori Synchronized php-1.26wmf17/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: (no message) (duration: 00m 12s)
22:23 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/startup.js: (no message) (duration: 00m 12s)
22:22 logmsgbot: ori Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: (no message) (duration: 00m 11s)
22:21 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: 94703bc291: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
22:02 mutante: if up for watching the (auto)-upgrade and restarting: @carbon:/srv/wikimedia/incoming# reprepro -C main include adminbot_1.7.12_amd64.changes
22:00 mutante: built adminbot 1.7.12 and copied to carbon to incoming - but not imported
21:56 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Graph: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Graph (duration: 00m 12s)
21:55 hoo: Running "updateSpecialPages.php --wiki wikidatawiki --only DoubleRedirects" on terbium
21:46 ejegg: updated payments from 5bc32b7d0969878e441394c828620d5a44683c18 to af16d371f9c46d4f0b78986080f2a2be3226ace8
21:25 logmsgbot: krinkle Synchronized php-1.26wmf17/extensions/EducationProgram/EducationProgram.hooks.php: T107980 (duration: 00m 12s)
21:12 gwicke: switched cassandra staging cluster (xenon, cerium, praseodymium) to CMS & started a load test on that
21:07 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/includes/Hooks.php: Repush file spewing notices into hhvm.log (duration: 00m 12s)
20:26 chasemp: es-tool restart-fast on elastic1031 to test alerting issues
20:06 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
19:42 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: actually deploy the hotfix this time (duration: 01m 33s)
19:38 urandom: issuing nodetool cleanup on restbase1003
19:36 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
19:35 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: sync hotfixes before deploying 1.26wmf17 to group2 (duration: 02m 18s)
18:57 ejegg: updated payments from bbec5799db42f6f5302920a1a69123de7e4986df to 5bc32b7d0969878e441394c828620d5a44683c18
18:55 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: revert 1.26wmf17
18:47 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
18:11 ejegg|mtg: updated payments from a8c0ecbedef6179c78ed833da9f2049cb0f2641b to bbec5799db42f6f5302920a1a69123de7e4986df
16:59 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/startup.js: touch (duration: 00m 11s)
16:56 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/Resources.php: T108191 unbreak mobile js (duration: 00m 11s)
16:24 chasemp: upgrading elastic1031 to 1.7.1
15:08 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/: bump cirrussearch in 1.26wmf17 for swat (duration: 00m 14s)
14:50 dcausse: es1.7.1: restart elastic1030
14:19 urandom: beginning nodetool cleanup on restbase1001
14:14 moritzm: restarted HHVM on appservers in codfw for libtidy/PCRE security updates
14:06 dcausse: es1.7.1: restart elastic1029
13:27 dcausse: es1.7.1: restart elastic1028
12:59 dcausse: es1.7.1: restart elastic1027
12:55 godog: stop syslog-ng on lithium before switching to rsyslog
11:55 dcausse: es1.7.1: restart elastic1026
10:33 dcausse: es1.7.1: restart elastic1025
09:50 moritzm: uploaded openjdk-8_8u66-b01-1~bpo8+1 to jessie-wikimedia and jessie-backports/debian.org
09:39 jynus: Applying schema change to Commons db master
09:39 moritzm: restarted HHVM on API apaches in eqiad for libtiny/PCRE security updates
09:30 dcausse: es1.7.1: restart elastic1024
09:18 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1068 (duration: 00m 13s)
08:20 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 6 08:20:24 UTC 2015 (duration 20m 23s)
07:34 dcausse: es1.7.1: restart elastic1023
07:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1056, Depool db1068 (duration: 00m 12s)
06:52 moritzm: restart HHVM on canary API servers (mw1114-mw1119) for libtiny/PCRE security updates
06:16 dcausse: es1.7.1: restart elastic1022
05:57 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 12s)
05:55 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 13s)
05:39 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 12s)
05:32 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 13s)
05:02 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/OutputPage.php: I885c36398 (duration: 00m 12s)
04:48 ebernhardson: es1.7.1 upgrade on elastic1021
04:42 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: I885c36398 (duration: 00m 12s)
04:04 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: T108139 (duration: 00m 12s)
03:58 logmsgbot: krinkle Synchronized php-1.26wmf17/includes: T108124 (duration: 00m 17s)
03:57 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: T108124 (duration: 00m 12s)
03:25 ebernhardson: es1.7.1 upgrade on elastic1020
03:18 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-06 03:18:27+00:00
03:12 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 32s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf16) at 2015-08-06 02:44:39+00:00
02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: l10nupdate for 1.26wmf16 (duration: 10m 42s)
02:16 ebernhardson: es1.7.1 upgrade on elastic1019
01:34 ebernhardson: es1.7.1 upgrade on elastic1018
00:49 Jamesofur: reset password for User:Tonval after identify verification
00:42 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
00:34 twentyafterfour: phabricator upgrade complete
00:33 ebernhardson: es1.7.1 upgrade on elastic1017
00:31 RoanKattouw: <twentyafterfour> ok I'm gonna take phabricator down for upgrade
00:04 gwicke: restarted restbase old-render clean-up scripts on wikipedia html and data-parsoid

2015-08-05

23:56 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Unset $wgDiff (duration: 00m 12s)
23:37 logmsgbot: ori Synchronized php-1.26wmf17/extensions/FlaggedRevs: I2089b21fc (duration: 00m 13s)
23:32 logmsgbot: bd808 Synchronized php-1.26wmf17/extensions/VisualEditor/extension.json: VisualEditor b/c anon IP module name fix (Ia92ecc0) (duration: 00m 12s)
23:09 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: beta: Configure and (I7d20abb) (duration: 00m 13s)
23:01 logmsgbot: ori Synchronized php-1.26wmf17/extensions/EducationProgram: I2089b21fc (duration: 00m 13s)
23:00 ebernhardson: es1.7.1 upgrade on elastic1016
22:47 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 12s)
22:47 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 13s)
22:29 hoo: Started dumpwikidatajson.sh on snapshot1003 again to create a Wikidata json dump after earlier attempts this week and today failed.
22:27 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 21s)
22:27 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 20s)
22:27 ebernhardson: es1.7.1 upgrade on elastic1015
21:44 subbu: deployed cherry-picked ba49b80bdc3a156604eb3996830af0d5bc45c503 hotfix to the parsoid cluster to deal with crashers from deploy earlier today
21:17 gwicke: finished deploy of restbase 9e177f3 (deploy 7006f9f) on restbase cluster
21:12 hoo: Started dumpwikidatajson.sh on snapshoot1003 to create a Wikidata json dump after earlier attempts this week failed.
21:05 ebernhardson: es1.7.1 upgrade for es1014
20:59 gwicke: restbase 9e177f3 (deploy 7006f9f) canary deploy on restbase1001
20:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
20:55 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
20:25 subbu: deployed parsoid version d5a5722c
20:22 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
20:21 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 11s)
20:13 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
20:12 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 13s)
20:07 logmsgbot: ori Synchronized php-1.26wmf17/extensions/PageTriage: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/PageTriage 22eddf4ad5bf6b3fe7c49af5812ce5fcfa5e1911 (duration: 00m 14s)
19:55 gwicke: re-enabled puppet on restbase staging cluster in preparation for deploy
19:52 gwicke: disabled puppet on restbase hosts in preparation for the deploy
19:36 dcausse: es1.7.1: resume writes to indices
19:31 dcausse: es1.7.1: restart elastic1013
19:19 bblack: all caches depooled for thermal stuff repooled
18:54 bblack: depooled cp1060, cp1064 ( thermal batch 3: https://phabricator.wikimedia.org/T103226 )
18:37 dcausse: es1.7.1: restart elastic1012
18:34 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf17
18:07 bblack: depooled cp1059, cp1062, cp1067 ( thermal batch 2: https://phabricator.wikimedia.org/T103226 )
18:02 moritzm: restarted HHVM on appservers (mw1136-mw1158) for tidy/pcre security updates
17:56 dcausse: es1.7.1: restart elastic1011
17:48 dcausse: es1.7.1: freeze indices (take 2)
17:36 logmsgbot: bblack Synchronized wmf-config/squid-labs.php: (no message) (duration: 00m 12s)
17:15 moritzm: restarted HHVM on appservers (mw1149-mw1151, mw1161-1188, mw1209-1220) for tidy/pcre security updates
17:09 logmsgbot: hoo Finished scap: Rebuild l10n cache for wmf17, got forgotten during the train (duration: 26m 02s)
17:07 bblack: really depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
17:02 bblack: depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
16:43 logmsgbot: hoo Started scap: Rebuild l10n cache for wmf17, got forgotten during the train
16:28 bblack: cache puppets disabled for a little while, to make sure do_esi doesn't melt things
15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: FIX: Not able to set cursor in previous sections gerrit:229328 (duration: 00m 12s)
15:02 andrewbogott: rebooting labvirt1009
14:51 gwicke: stopped restbase on restbase1009
14:44 moritzm: restarted HHVM on appservers (mw1026-mw1113) for tidy/pcre security updates
14:42 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 (duration: 00m 12s)
14:29 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1059 (duration: 00m 13s)
13:16 hoo: Removed Wikidata JSON dumps from Monday and Tuesday as they were incomplete/ had the wrong serialization format
12:41 moritzm: restarted HHVM on canary appservers for tidy/pcre security updates, remaining app servers following soon
12:32 paravoid: upgrading asw-c-codfw and asw-d-codfw to newer junos
11:17 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1056, depool db1059 (duration: 00m 12s)
11:01 godog: depool restbase1009, investigating healthcheck returning 500s
10:52 godog: pool restbase100[789] in pybal
10:43 paravoid: upgrading asw-b-codfw to newer junos
10:36 jynus: applying schema change for s4 on codfw, some lag expected
09:08 dcausse: es1.7.1: upgrade elastic1010
07:46 dcausse: es1.7.1: upgrade elastic1009
07:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 for maintenance, db1064 set to 100% (duration: 00m 12s)
06:29 springle: finish OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
06:27 logmsgbot: @tin ResourceLoader cache refresh completed at Wed Aug 5 06:27:08 UTC 2015 (duration 27m 7s)
06:26 dcausse: es1.7.1: upgrade elastic1008
04:56 ebernhardson: restarted elasticsearch on elastic1007 for 1.7.1 upgrade
03:34 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable two more wikis due to namespace conflicts - https://gerrit.wikimedia.org/r/229292 (duration: 00m 12s)
03:09 ebernhardson: restarted elasticsearch on elastic1006 for 1.7.1 upgrade
03:04 logmsgbot: @tin LocalisationUpdate completed (1.26wmf17) at 2015-08-05 03:04:08+00:00
02:57 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: (no message) (duration: 10m 30s)
02:31 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-05 02:31:44+00:00
02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 56s)
01:44 ebernhardson: restarting elasticsearch of es1005

2015-08-04

23:59 logmsgbot: maxsem Synchronized php-1.26wmf16/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
23:57 logmsgbot: maxsem Synchronized php-1.26wmf17/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
23:08 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Disable Flow on betawikiversity (duration: 00m 13s)
22:07 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: forgot submodule update (duration: 01m 39s)
20:46 logmsgbot: twentyafterfour Finished scap: fixup wikidata submodule version (duration: 23m 26s)
20:22 logmsgbot: twentyafterfour Started scap: fixup wikidata submodule version
19:46 dcausse: es1.7.1: upgrade elastic1003
19:12 ori: Applied Icba6d7a87 on mw1017 for a couple of webpagetest runs
19:08 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf17
18:51 logmsgbot: twentyafterfour Finished scap: rebuild localization cache, sync 1.26wmf17 (duration: 28m 39s)
18:42 dcausse: es1.7.1: upgrade elastic1002
18:22 logmsgbot: twentyafterfour Started scap: rebuild localization cache, sync 1.26wmf17
18:00 andrewbogott: re-imaging labnodepool1001
17:35 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1064 traffic (duration: 00m 13s)
17:18 dcausse: es1.7.1: upgrade elastic1001
17:17 hoo: Started dumpwikidatajson.sh on snapshot1003 to create a correct Wikidata json dump
17:14 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Fix maintenance/dumpJson.php fatal (duration: 00m 21s)
17:11 chasemp: freezing elasticsearch indexes for 1.7.1
16:23 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1064 with low traffic after maintenance (duration: 00m 12s)
15:34 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Flow on ptwikibooks gerrit:229133 (duration: 03m 40s)
15:28 jynus: restarting db1064 for regular maintenance and upgrade given that it was depooled in the first place for a schema change
15:24 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Add configuration for authmetrics logging (part II) gerrit:227630 (duration: 02m 41s)
15:21 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Add configuration for authmetrics logging (part I) gerrit:227630 (duration: 03m 11s)
15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on enwiki gerrit:227329 (duration: 03m 13s)
14:36 paravoid: cr2-codfw upgrading SCBs
14:23 paravoid: upgrading junos on asw-a-codfw again
13:45 _joe_: repooling mw1159,mw1160
13:21 paravoid: rebooting asw-a-codfw, member 2
13:04 Coren: labstore1001 rebooting (possibly a couple of times) during tests and reinstallation
12:55 hoo: Syncing to mw1160 failed (Host key verification failed.)
12:50 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fixes for JSON dump creation (duration: 00m 39s)
12:06 moritzm: updated canary appservers mw1017/mw1018 to updated pcre3 + hhvm restart
12:03 moritzm: added pcre3_8.31-2ubuntu2.1+wm1 to trusty-wikimedi (reroll of security update with our JIT enablement patch)
11:48 _joe_: killed ircecho to prevent furter icinga spam
11:44 jynus: schema update on Commons failed, expect some minor inestabilities until everything is fixed
11:41 _joe_: reimaging mw1159 to HAT
11:01 paravoid: upgrading junos on asw-a-codfw
10:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 00m 13s)
10:27 godog: bootstrap cassandra on restbase1009
10:21 akosiaris: enabling puppet on tin
09:30 jynus: rolling schema change on image table to all wikis
08:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increasing load for db1027 and db1015 (duration: 00m 12s)
07:38 logmsgbot: @tin ResourceLoader cache refresh completed at Tue Aug 4 07:38:01 UTC 2015 (duration 38m 0s)
06:14 _joe_: depooled mw1061
06:14 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on Japanese Wikiversity (duration: 00m 13s)
06:09 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
06:07 legoktm: sync to mw1061 failed
06:07 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
02:32 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-04 02:32:18+00:00
02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 09m 16s)
02:18 logmsgbot: twentyafterfour Finished scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1 (duration: 25m 41s)
01:52 logmsgbot: twentyafterfour Started scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1
00:02 awight: updated paymentswiki to a8c0ecbedef6179c78ed833da9f2049cb0f2641b

2015-08-03

23:56 awight: updating paymentswiki to b20559f75e0fc0d863efe027d76b78462555767c
23:45 ottomata: rebuilding kafka cluster
23:21 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/VisualEditor/: Bump visualeditor for swat in 1.26wmf16 (duration: 00m 13s)
23:18 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/WikimediaEvents/: Bump WikimediaEvents in SWAT for 1.26wmf16 (duration: 00m 12s)
23:17 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/Flow: Bump flow submodule in swat for 1.26wmf16 (duration: 00m 14s)
23:05 logmsgbot: ebernhardson Synchronized wmf-config/: (no message) (duration: 00m 13s)
22:46 awight: reverting paymentswiki, to 6dbbb4c784349ace5a0ac616c61ec0c3fffa0eff
22:33 ejegg: updated crm from db417a28a247a3fdf3e3023a700d6266e04f3e9d to 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1
22:27 awight: deployed debug hack to payments1004
21:43 awight: deploy paymentswiki-staging configuration: add explicit queue name for payments4 connecting to payments1-3
21:32 awight: deploy paymentswiki-staging configuration
21:25 awight: updating payments1004 to 1daf9d0fe773c022a2ab8de5542fc15ddc261e75
21:04 logmsgbot: bd808 Synchronized wmf-config/logging.php: Remove code duplication from monolog config (Ia960203) (duration: 00m 11s)
20:51 awight: updating paymentswiki from d4bdce1cae168448b116d75e3dcd3303b0f13dd2 to d56dad49ef0da0a8b9c7da410bcac12e48724ae5
20:26 arlolra: updated Parsoid to version 38d0cdb13734a40bc2908e779e1a0cde158048f2
19:49 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: Fix T104609 and fix/debug T107711 (duration: 00m 19s)
19:21 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on enwiki (duration: 00m 12s)
19:20 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Add debug log group for T107711 (duration: 00m 12s)
19:07 ottomata: stopped a couple of kafka brokers. acknowldeging..
19:02 bblack: https://gerrit.wikimedia.org/r/228882 reversion salted + nginx reloaded
18:28 gwicke: switched restbase1002 and restbase1003 to iojs as well
17:36 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on zhwiki (duration: 00m 12s)
17:21 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/Revision.php: https://gerrit.wikimedia.org/r/228853 (duration: 00m 12s)
17:21 ottomata: starting kafka partition reassignment to balance all partiions over to 3 new kafka brokers and off of analytics1021
17:21 gwicke: switching from node 0.10 to iojs 2.5 on restbase1001 after load testing on xenon went well
17:02 logmsgbot: legoktm Synchronized wmf-config/logging.php: logging: Enable stacktrace printing (duration: 00m 12s)
17:00 hoo: Started dumpwikidatajson.sh on snapshot1003 to re-create today's dump
16:55 logmsgbot: legoktm Synchronized php-1.26wmf16/autoload.php: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 12s)
16:54 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/debug/logger/: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 11s)
16:49 hoo: Removed today's Wikidata json dump (wikidata-20150803-all.json.gz) because it was incomplete due to the dataset problems earlier
16:27 paravoid: upgrading junos on cr2-codfw
15:34 bblack: wiping cp3034 disk cache (upload esams) for ipsec reload testing
15:23 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv (touch and re-sync) gerrit:228218 (duration: 00m 12s)
15:22 ottomata: reinstalling analytics1013,1014 and 1020 with Jessie
15:10 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv gerrit:228218 (duration: 00m 12s)
14:59 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on trwiki (duration: 00m 12s)
14:54 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/SemanticResultFormats: https://gerrit.wikimedia.org/r/#/c/228793/ (duration: 00m 13s)
14:42 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on thwiki (duration: 00m 12s)
14:33 mutante: temp. stop puppet on dataset1001
14:27 paravoid: upgrading junos on cr1-codfw
14:23 moritzm: updated iojs on apt.wikimedia.org to 2.5.0 for jessie-wikimedia
14:21 ottomata: upgrading kernel on analytics1042-1049 from 3.13.0.24.28 to 3.13.0.61.68 because T107698
14:18 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on svwiki (duration: 00m 12s)
13:50 bblack: re-enabling puppet + ircecho on neon (vast majority of recovery spam is over with)
13:17 bblack: re-enable agent, restarted apache2 on palladium, strontium, rhodium (fact_values truncated in mysql)
13:10 bblack: rhodium too (puppetmaster stop)
13:05 bblack: stopped puppet-agent + apache2 on strontium + palladium (no masters alive, for mysql maintenance)
12:59 bblack: stopped ircecho + puppet-agent on neon (spam from epic puppetmaster fail)
12:52 bblack: stop->wait->restart of apache2 service on palladium (seemed dead to puppet reqs)
12:21 _joe_: bumped ganglia-monitor-aggregator on bast4001, the upstart script needs immediate fixing
11:01 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: avoid db1044 SPOF by repooling db1027 and db1015 (duration: 00m 12s)
10:56 paravoid: switching GeoDNS to GeoIP2
10:45 paravoid: upgrading all AuthDNS servers to gdnsd 2.2.0
09:31 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1035 for maintenance (duration: 00m 12s)
05:22 logmsgbot: @tin ResourceLoader cache refresh completed at Mon Aug 3 05:22:15 UTC 2015 (duration 22m 14s)
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-03 02:23:21+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 21s)
01:47 springle: starting OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
00:03 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/228198/ (duration: 00m 12s)

2015-08-02

17:52 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: If7fcb6e6: Default wikipedias to enwiki.png (duration: 00m 12s)
13:26 jynus: powercycling analytics1044: same kernel fatal issues as 1043
13:10 jynus: powercycling analytics1043: kernel issues
12:05 bblack: started pybal on lvs3001
04:56 logmsgbot: @tin ResourceLoader cache refresh completed at Sun Aug 2 04:56:29 UTC 2015 (duration 56m 28s)
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-02 02:23:09+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)

2015-08-01

06:04 _joe_: removing some old apache access logs from mw1114
05:06 logmsgbot: @tin ResourceLoader cache refresh completed at Sat Aug 1 05:06:46 UTC 2015 (duration 6m 45s)
03:53 andrewbogott: cleared out nova-conductor.log on labcontrol1001, restarted nova-conductor, graceful’d apache
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-01 02:23:15+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)
00:12 logmsgbot: ori Synchronized extract2.php: Ie919881a4: Add an API listing template to the allowed templates in extract2.php
00:01 logmsgbot: ori Synchronized php-1.26wmf16/includes: Revert I4afaecd8: "Avoiding writing sessions for no reason", and undo several uncommitted live-hacks for debugging T102199 (duration: 00m 16s)

<inputbox> type=fulltext prefix=Server Admin Log/ searchbuttonlabel=Search archives break=no </inputbox>

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s