You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server admin log/Archive 22: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Krinkle
No edit summary
imported>Nintendofan885
(Nintendofan885 moved page Server admin log/Archive 22 to Server Admin Log/Archive 22: Match top page)
 
Line 1: Line 1:
== July 31 ==
#REDIRECT [[Server Admin Log/Archive 22]]
* 23:40 logmsgbot: rmoen synchronized wmf-config/InitialiseSettings.php  'Remove config variables for VE gender survey'
* 23:39 logmsgbot: rmoen synchronized wmf-config/CommonSettings.php  'Remove config variables for VE gender survey'
* 23:32 logmsgbot: rmoen Finished syncing Wikimedia installation... : update visualeditor to master
* 23:25 logmsgbot: rmoen Started syncing Wikimedia installation... : update visualeditor to master
* 23:18 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo  'Update Echo to master'
* 22:14 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth  'eventlogging patch'
* 18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:33 cmjohnson1: amaranth currently unreachable according to nosy....going to try rebooting
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf12
* 15:42 logmsgbot: hashar synchronized wmf-config  '{{gerrit|{{Gerrit|71774}}}} set some paths to use  'bits' and 'test''
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 31 02:46:48 UTC {{Gerrit|2013}}
* 02:31 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 31 02:29:48 UTC {{Gerrit|2013}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Jul 31 02:15:56 UTC {{Gerrit|2013}}
* 01:09 springle: restarting db45 slave threads
* 01:08 springle: restarting db43 slave threads
 
== July 30 ==
* 23:36 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor  'Update VE to master'
* 23:35 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor  'Update VE to master'
* 23:15 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js  'Refresh js file'
* 23:14 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js  'Refresh js file'
* 23:11 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js  'Refresh js file'
* 23:10 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js  'Refresh js file'
* 23:07 awight: listeners updates from {{Gerrit|246f50c3a0f39bfd7e26afd5b4ee18712ab44c9c}} to {{Gerrit|243aeba728ad72ee59f8e746880ee17a71ce68a9}}
* 22:20 awight: payments updated from {{Gerrit|74b8b03ac816c8a0d678ca0358af5387097b541a}} to {{Gerrit|fd276e06cfa0d9f05228e2d38265ded9ade2e3b2}}
* 21:35 awight_: payments updated from {{Gerrit|855aa0f8c0bbc5eb3eed78dfccdf61797e284c2d}} to {{Gerrit|74b8b03ac816c8a0d678ca0358af5387097b541a}}
* 21:29 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:28 awight_: listeners updated from {{Gerrit|90ddcc35ccab7366a0fc6ba12f457a513d4515e3}} to {{Gerrit|246f50c3a0f39bfd7e26afd5b4ee18712ab44c9c}}
* 21:18 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:35 mutante: gracefull'ing apaches, RT-{{Gerrit|5539}} & bug {{Gerrit|51593}}
* 20:19 paravoid: switching wikidata back to squid; reports of "bad token" messages
* 19:24 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 19:19 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Revert Enable Secure Login everywhere'
* 19:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php  'Revert Enable Secure Login everywhere'
* 18:41 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 18:20 mutante: deploying fix for Bug {{Gerrit|51593}} - Correct the mime type for WOFF to application/font-woff
* 18:05 awight: fr-civicrm updated from {{Gerrit|6e4e641c973f2e63b18608a7df6726e5236b234d}} to {{Gerrit|661a1af4dc941675a1edb183830f8dc9529fbd6a}}
* 18:04 awight: fr-civicrm updated from {{Gerrit|820c54a659006a358e27e7451ed0274a3198f3cb}} to {{Gerrit|6e4e641c973f2e63b18608a7df6726e5236b234d}}
* 17:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable Translate on collabwiki'
* 17:37 Reedy: Created Translate tables on collabwiki
* 17:15 logmsgbot: reedy synchronized wmf-config/  'Disable UploadBlacklist'
* 17:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:09 logmsgbot: reedy synchronized wmf-config/
* 15:22 Reedy: Purging old SecurePoll private data from all wikis
* 12:21 mark: Put all non-European wikidata traffic on cp1052 and cp1065
* 11:21 mark: Pooled cp1052 as text varnish box in PyBal group 'text-varnish' with weight 1, to serve a tiny fraction of wikidata traffic
* 08:56 Nikerabbit: Translation memory migrated from vanadium to zinc and repopulated
* 08:12 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php  'Translation memory to zinc from vanadium'
* 02:58 sbernardin: racked and plugged back pappas
* 02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 30 02:52:51 UTC {{Gerrit|2013}}
* 02:34 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 30 02:33:27 UTC {{Gerrit|2013}}
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Jul 30 02:17:54 UTC {{Gerrit|2013}}
 
== July 29 ==
* 23:23 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php  'Enabling wgNoticeUseLanguageConversion for automatic zh conversion in CentralNotice'
* 23:21 logmsgbot: mwalker synchronized php-1.22wmf12/extensions/CentralNotice  'Updating CentralNotice to master on wmf12'
* 23:18 logmsgbot: mwalker synchronized php-1.22wmf11/extensions/CentralNotice  'Updating CentralNotice to master on wmf11'
* 23:03 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php  'enable captcha on ptwiki'
* 22:35 andrewbogott: moved private puppet manifests into modules
* 22:31 mutante: creating operations/debs/etherpad-lite gerrit project
* 22:21 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
* 22:19 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
* 21:18 mutante: removing unused mysql-server from hooper, the only service uses db9
* 21:16 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Hack for Cirrus in labs'
* 21:13 mutante: upgrading packages on hooper (etherpad/racktables)
* 20:34 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php  'More Cirrus goodies'
* 20:17 logmsgbot: reedy synchronized php-1.22wmf11/extensions/DataTypes
* 20:13 logmsgbot: reedy synchronized wmf-config/
* 20:10 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Syncing Cirrus config'
* 20:09 logmsgbot: demon synchronized wmf-config/CirrusSearch-labs.php  'Syncing Cirrus config'
* 20:08 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php  'Syncing Cirrus config'
* 20:03 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CirrusSearch
* 19:49 logmsgbot: reedy synchronized php-1.22wmf12/extensions/DataTypes
* 19:41 springle: delaying slave db45 during OSC bug {{Gerrit|49199}}
* 19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.22wmf12
* 19:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Actually disable VE for anons on dewiki, hopefully'
* 19:01 Jeff_Green: thulium kernel update, reboot, tweak CPU power settings
* 19:01 springle: delaying slave db43 during OSC bug {{Gerrit|49199}}
* 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf12
* 19:00 andrewbogott: set root password to ! on all labs instances.
* 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity
* 18:55 ^demon: restarted gitblit service
* 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, wikimedia and special to 1.22wmf12
* 18:42 sbernardin: server pappas being taken offline to ship to EQIAD
* 18:41 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 18:18 mutante: DNS update - added voyagewiki.org and .com
* 18:12 springle: restarting db56 slave threads
* 16:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable VE for anons on es/fr/he/it/pl/ru/svwiki; set dewiki back to opt-in mode'
* 16:35 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor  'Update VE to master'
* 16:34 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor  'Update VE to master'
* 14:50 logmsgbot: maxsem synchronized wmf-config/squid.php  'Amsterdam varnish IPv6'
* 14:25 andrewbogott: restarted apache on virt0, fixing (at least temporarily) bug {{Gerrit|52217}}
* 13:37 Jeff_Green: removed spammy deprecated cron from gadolinium
* 13:24 ottomata: stopping nginx on maerlant.esams.wikimedia.org (this machine should probably be fully decommissioned)
* 04:31 springle: delaying slave db56 during OSC bug {{Gerrit|49199}}
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 29 02:33:00 UTC {{Gerrit|2013}}
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Jul 29 02:19:08 UTC {{Gerrit|2013}}
* 02:19 springle: enabling slave db45
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 29 02:10:33 UTC {{Gerrit|2013}}
* 01:23 springle: delaying slave db45
 
== July 28 ==
* 22:37 MaxSem: mw1157 was running out of /tmp space, nuked all *.v older than 1 day. VIPS bug?
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 28 02:07:37 UTC {{Gerrit|2013}}
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Jul 28 02:02:52 UTC {{Gerrit|2013}}
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sun Jul 28 02:02:01 UTC {{Gerrit|2013}}
 
== July 27 ==
* 23:43 Krinkle: Graceful reload of Zuul (in attempt to purge the ghost entries caused by bug {{Gerrit|52161}})
* 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 27 02:49:20 UTC {{Gerrit|2013}}
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Jul 27 02:31:17 UTC {{Gerrit|2013}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 27 02:16:34 UTC {{Gerrit|2013}}
 
== July 26 ==
* 23:12 mutante: activated softwarewikipedia.com/.org
* 23:08 mutante: activated http://wikimediacommons.pt/
* 23:02 mutante: DNS update - removing zones that link to other links. use wikipedia.org not wikipedia.com
* 20:05 mutante: synced apache, gracefull'ed, activate visualwikipedia.com and .net
* 18:22 mutante: installing package upgrades on iron
* 18:15 binasher: db1042 unresponsive on serial console, power cycling
* 18:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1042 from s4'
* 17:07 mark: Rebooting cp3011
* 15:34 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 15:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 02:36 ori-l: ran the git-clone update component of l10nupdate (l10nupdate-1, lines 17-45) as 'l10nupdate' user to try and reproduce "git pull of extensions failed" error, but it completed successfully for me.
* 02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 01:11 logmsgbot: reedy synchronized php-1.22wmf11/languages/messages/MessagesTl.php
* 00:48 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/special/ext.echo.special.js  'update cache for resourceloader'
* 00:47 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/overlay/ext.echo.overlay.js  'update cache for resourceloader'
* 00:17 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|76043}}/'
* 00:16 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|76043}}/'
* 00:11 logmsgbot: bsitu Finished syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
* 00:00 logmsgbot: bsitu Started syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
 
== July 25 ==
* 20:50 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted  'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:49 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour  'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted  'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour  'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GettingStarted  'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GuidedTour  'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:29 MaxSem: Ran sync-common on mw1041 - looked outta sync
* 18:51 paravoid: authdns-update: change ns1's IP to new service IP; glues were updated
* 18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:40 logmsgbot: reedy synchronized wmf-config/
* 18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf11
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf12
* 18:24 logmsgbot: reedy Finished syncing Wikimedia installation... :
* 18:18 Jeff_Green: authdns-update to change IPs for db1008
* 18:06 logmsgbot: reedy Started syncing Wikimedia installation... :
* 17:54 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 17:44 logmsgbot: reedy synchronized php-1.22wmf11/cache/l10n/
* 17:30 logmsgbot: reedy Started syncing Wikimedia installation... :
* 17:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to wmf11
* 17:12 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 17:11 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Jul 25 17:10:36 UTC {{Gerrit|2013}}
* 16:42 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 16:40:47 UTC {{Gerrit|2013}}
* 16:30 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 16:29:28 UTC {{Gerrit|2013}}
* 16:14 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
* 15:55 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
* 15:30 logmsgbot: reedy synchronized docroot and w
* 15:11 logmsgbot: reedy synchronized php-1.22wmf12
* 12:49 hashar: gallium cleared out /etc/update-motd.d/05* which adds the roles in motd.  They got out of sync.  Rerunning puppet
* 08:25 paravoid: running swift->ceph thumb sync
* 07:58 paravoid: upgrading ceph to 0.67-rc2
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 25 02:47:27 UTC {{Gerrit|2013}}
* 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 02:29:13 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 02:15:32 UTC {{Gerrit|2013}}
* 00:15 Ryan_Lane: upgrading wikitech-static to 1.22wmf11
 
== July 24 ==
* 23:55 logmsgbot: krinkle synchronized php-1.22wmf11/extensions/VisualEditor  'Idc7b094f8eb2788c48'
* 23:50 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor  'Idc7b094f8eb2788c48'
* 23:15 Ryan_Lane: upgrading wikitech to 1.22wmf11
* 23:14 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
* 23:12 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
* 22:46 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
* 22:45 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
* 21:59 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:44 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:30 binasher: dropped all databases on s3 that were migrated to s7 [rt5506]
* 21:29 mwalker: updating fundraising civicrm from {{Gerrit|0834a16761081c2b8e39624eb905be89bd524bc7}} to {{Gerrit|820c54a659006a358e27e7451ed0274a3198f3cb}}
* 20:21 paravoid: authdns-update: change ns0's IP to new service IP; glues were updated
* 19:04 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor  'Fix copyright notice bug'
* 19:03 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor  'Fix copyright notice bug'
* 18:04 logmsgbot: catrope synchronized php-1.22wmf11/resources/mediawiki/mediawiki.notification.js  'Fix mw.notify positioning bug'
* 18:04 logmsgbot: catrope synchronized php-1.22wmf10/resources/mediawiki/mediawiki.notification.js  'Fix mw.notify positioning bug'
* 17:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs  'Fix FlaggedRevs fatal'
* 17:42 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs  'Fix FlaggedRevs fatal'
* 17:32 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs  'Fix excessive FlaggedRevs notices'
* 17:31 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs  'Fix excessive FlaggedRevs notices'
* 17:21 RobH: torrus deadlocked, fixing
* 17:10 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable VisualEditor on 8 more wikis'
* 17:09 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Show either visualeditor-enable or visualeditor-betatempdisable, not both'
* 16:48 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js  'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js  'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js  'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js  'touch'
* 16:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor  'Fix nowiki bug'
* 16:41 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor  'Fix nowiki bug'
* 16:32 RoanKattouw: Fixed /etc/dsh/group/bits ... again :(
* 16:28 logmsgbot: catrope Finished syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
* 16:25 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js  'touch'
* 16:22 logmsgbot: catrope synchronized php-1.22wmf10/resources/startup.js  'touch'
* 16:04 logmsgbot: catrope Started syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
* 15:32 logmsgbot: reedy synchronized wmf-config/
* 12:18 logmsgbot: mark synchronized wmf-config/squid.php
* 11:00 MaxSem: Aborted TTM migration, will file a bug report
* 10:25 MaxSem: Attempting to migrate TTM to zinc
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 24 02:26:44 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 24 02:15:04 UTC {{Gerrit|2013}}
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 24 02:08:46 UTC {{Gerrit|2013}}
* 00:12 ^demon: rebooting formey
* 00:07 ^demon: dist-upgrading formey lol
 
== July 23 ==
* 23:41 MaxSem: Rebuilding Solr index to catch up new schema
* 23:35 andrewbogott: done messing with sockpuppet, stafford, puppet-merge, etc for now.
* 21:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:56 mwalker: updating civicrm on aluminium from {{Gerrit|e4062d012d0ccfb6220fc3703fd42ad894151ec5}} to {{Gerrit|180ecd219da68a0b950dafb036078b7c285e2437}}
* 20:50 mwalker: kludging payments data -- adding 7 hours to contribution timestamps between IDs {{Gerrit|4380557}} and {{Gerrit|4605567}}
* 20:36 mwalker: paused all audit and QC jenkins jobs on the payments cluster in preparation for fixing timestamps
* 19:55 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:47 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:31 Jeff_Green: dist-upgrade & reboot for payments*
* 18:40 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:07 logmsgbot: reedy synchronized wmf-config/
* 18:03 logmsgbot: reedy synchronized database lists files:
* 18:01 logmsgbot: reedy synchronized wmf-config/
* 17:59 Reedy: Populated sites table on all wikivoyage wikis
* 17:41 logmsgbot: reedy synchronized wmf-config/
* 17:32 logmsgbot: reedy synchronized wmf-config/
* 17:28 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 17:27 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 17:14 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php  '{{Gerrit|26162fbc97ebbe25fee84bb333b0b0b2efe4c447}}'
* 16:41 Jeff_Green: authdns-update to propagate fundraisingdb-* cname changes
* 15:59 Jeff_Green: stop icinga notifications for db1025 mysql
* 15:58 Jeff_Green: stop icinga notifications for db1008
* 15:22 Jeff_Green: dist-upgrade &reboot db1025
* 15:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:14 logmsgbot: reedy synchronized database lists files:
* 15:14 Jeff_Green: dist-upgrade & reboot payments2-4
* 14:58 Jeff_Green: dist-upgrade & reboot payments2
* 14:48 ottomata: stopping puppet on gadolinium, looking into filters causing socat process packet loss
* 14:23 hashar: Jenkins: lanthanum has been added as a Jenkins slave node though NO job should be running there.e
* 10:48 logmsgbot: mark synchronized wmf-config/squid.php  'Add ssl1005/ssl1006'
* 05:45 paravoid: pooling ssl1005/{{Gerrit|1006}} for https/ipv6
* 05:35 paravoid: authdns-update for cr1-ulsfo <-> cr2-eqiad, dns-auth-lb -> nsN(-new), ssl100x fixed IPv6
* 02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 23 02:21:20 UTC {{Gerrit|2013}}
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 23 02:12:36 UTC {{Gerrit|2013}}
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 23 02:08:03 UTC {{Gerrit|2013}}
* 01:02 Ryan_Lane: enabled hyperthreading on ssl1005
* 00:40 Ryan_Lane: depooling ssl1005/6 due to ipv6 issues
* 00:40 Ryan_Lane: pooled new https node ssl1006.wikimedia.org (with hyperthreading enabled)
* 00:40 Ryan_Lane: pooled new https node ssl1005.wikimedia.org
 
== July 22 ==
* 22:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/UploadWizard
* 22:05 cmjohnson1: restarting ns2
* 22:01 cmjohnson1: dns update
* 21:48 LeslieCarr: restarting powerdns on ns1
* 21:05 logmsgbot: asher synchronized wmf-config/StartProfiler.php  'only randomly profile web reqs in eqiad'
* 20:40 sbernardin: hot swapping disk 13 on db78 storage array
* 20:21 paravoid: deploying bmc-config fix across all pmtpa/eqiad mgmt
* 19:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everyhting else no wikipedia to 1.22wmf11
* 19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikinews and wikibooks to 1.22wmf11
* 19:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikivoyage and wiktionary to 1.22wmf11
* 19:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, private, fishbowl and closed to 1.22wmf11
* 19:26 paravoid: restarting & repooling cp1047/cp1059
* 19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move 3 wikis with UW to 1.22wmf11
* 16:04 hashar: Jenkins back up
* 15:53 hashar: jenkins accidentally killed Jenkins :(
* 15:18 hashar: gallium / jenkins : blacklisting a bunch of user agent {{gerrit|{{Gerrit|75105}}}}
* 14:42 apergos: (btw docs would be nice, is that really the right way to kick it?)
* 14:42 apergos: shot and restarted gitblit:  on antinomy, cd /var/lib/gitblit,  java -jar gitblit.jar &  (see bug {{Gerrit|51769}})
* 12:47 paravoid: depooling cp1047/cp1059 for further investigation
* 12:29 paravoid: restarting cp1046/cp1060's varnish-frontend
* 11:22 hashar: Jenkins: deleting -merge jobs ({{bug|{{Gerrit|51395}}}} sequentially to avoid deadlocking jenkins {{bug|{{Gerrit|51802}}}}
* 11:20 hashar: jenkins restarted
* 11:02 hashar: restarting Jenkins, some threads are deadlocked ( see {{bug|{{Gerrit|51802}}}} )
* 09:27 paravoid: adding new ns0/ns1 service ip static routes to dobson/linne on cr1-sdtpa/cr2-pmtpa
* 07:19 paravoid: restarting pybal and manually ipvsadm removing dns_auth services from lvs1/lvs5/lvs1002/lvs1005
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 22 02:17:55 UTC {{Gerrit|2013}}
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 22 02:10:16 UTC {{Gerrit|2013}}
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 22 02:06:01 UTC {{Gerrit|2013}}
 
== July 21 ==
 
* 20:20 hashar: crashed Jenkins while debugging it :(
* 19:29 hashar: Jenkins deleting remaining -merge jobs (not triggered anymore since that is handled by Zuul now {{bug|51395}})
* 19:28 hashar: logging bot is dead, no idea for how long though.
 
== July 20 ==
* 14:05 Nemo_bis: antimony was overloaded (100 % CPU, around 15 load) again, still serving internal errors
* 08:46 netsplit kills morebots
* 07:01 paravoid: apt: including dclass 2.2.2-1
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 20 02:30:42 UTC {{Gerrit|2013}}
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 20 02:18:54 UTC {{Gerrit|2013}}
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 20 02:10:26 UTC {{Gerrit|2013}}
 
== July 19 ==
* 20:32 logmsgbot: hashar synchronized wikiversions-labs.dat
* 20:31 logmsgbot: hashar synchronized all-labs.dblist
* 20:21 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php  'Slightly increase Parsoid dequeue rate'
* 19:44 sbernardin: shutting down spence for decommissioning
* 19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'restore Parsoid to previously deployed newer code'
* 19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/Parsoid.php  'restore Parsoid to previously deployed newer code'
* 19:37 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'restore Parsoid to previously deployed newer code'
* 19:36 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/Parsoid.php  'restore Parsoid to previously deployed newer code'
* 18:56 gwicke: updated Parsoid to {{Gerrit|d5fe6c9}}
* 14:51 mark: Upgraded mobile caches to varnish 3.0.3~rc1-wm14
* 14:40 hashar: continuous integration : slightly adjusted main page https://integration.wikimedia.org/  (see integration/docroot.git )
* 10:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Disable ShortUrl on testwiki'
* 10:12 logmsgbot: reedy synchronized php-1.22wmf11/includes/Setup.php
* 09:47 logmsgbot: reedy synchronized wmf-config/
* 07:56 paravoid: stopping radosgw across the ceph cluster, post-upgrade issues
* 02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 19 02:49:50 UTC {{Gerrit|2013}}
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Fri Jul 19 02:33:48 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 19 02:15:29 UTC {{Gerrit|2013}}
 
== July 18 ==
* 23:55 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
* 23:53 Tim: deploying new php5-fss, to fix segfault bug
* 23:41 springle: restarting db45 slave threads
* 23:39 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
* 23:18 springle: completed wikidatawiki wb_terms.term_weight OSC
* 22:48 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
* 22:31 logmsgbot: mflaschen Started syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
* 22:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Remove bigdelete from sysyop'
* 22:30 Ryan_Lane: dist-upgrading all virt nodes
* 21:57 superm401: Scap failed due to Wikibase issue
* 21:54 Aaron|home: Finally running "mwscriptwikiset maintenance/purgeDeletedFiles.php all.dblist --starttime {{Gerrit|20130529000000}}"
* 20:27 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:59 springle: starting wikidatawiki wb_terms.term_weight OSC
* 19:53 springle: delaying slave db45 for wikidatawiki wb_terms OSC duration
* 19:15 notpeter: adding elasticsearch v. 0.90.2 package to brewster
* 18:21 LeslieCarr: adding new links into cr1/cr2-eqiad asw-c-eqiad bundle
* 18:14 Reedy: Added term weight to wb_terms on testwikidatawiki
* 18:13 Reedy: Created wb_property_info on testwikidatawiki
* 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwikidatawiki to 1.22wmf11
* 18:07 Reedy: Ran extensions/UploadWizard/maintenance/migrateCampaigns.php against testwiki
* 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, wikidatawiki and loginwiki to 1.22wmf11
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.22wmf11
* 17:58 cmjohnson1: authdns update
* 17:58 logmsgbot: reedy synchronized wmf-config/
* 16:49 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
* 16:35 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
* 16:28 logmsgbot: reedy synchronized docroot and w
* 16:23 logmsgbot: reedy synchronized php-1.22wmf11/  'Initial code sync'
* 15:55 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth  'Fix SUL issue - wmf9'
* 15:53 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth  'Fix SUL issue'
* 14:33 ^demon: zuul back up too
* 14:32 ^demon: gerrit service back online
* 14:04 cmjohnson1: shutting down manganese
* 14:04 ^demon: stopped gerrit & zuul services on manganese and gallium respectively
* 14:03 cmjohnson1: removing/swapping sdb2 on manganese
* 12:28 mark: Inserted varnish 3.0.3plus~rc1-wm14 packages into the precise-wikimedia APT repository
* 02:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 18 02:43:17 UTC {{Gerrit|2013}}
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 18 02:28:28 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 18 02:15:04 UTC {{Gerrit|2013}}
 
== July 17 ==
* 22:17 awight: chongo
* 22:02 mwalker: updating payments-listeners from {{Gerrit|b2ac16f5284c56ea1ba475aba9dae9cc476f0f6d}} to {{Gerrit|90ddcc35ccab7366a0fc6ba12f457a513d4515e3}}
* 20:43 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php
* 20:26 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth  'hooks for SUL2 - wmf9'
* 20:23 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth  'hooks for SUL2'
* 20:19 Ryan_Lane: deploying OpenStackManager commit {{Gerrit|46f3be1}} to virt0
* 19:54 Ryan_Lane: clarification (those two hashes for OpenStackManager)
* 19:54 Ryan_Lane: deploying {{Gerrit|5e1289b}} and {{Gerrit|40ca623}} to virt0
* 18:54 ^d: killed *another* pywikibot/compat clone. Please stop cloning this right now, you're gerrit cry.
* 18:43 awight: payments cluster updated from from {{Gerrit|2a9169765b94e0}} to {{Gerrit|855aa0f8c0bbc5}}
* 18:03 ^d: killed a clone of pywikibot/compat. Until I figure out why it's soooo big, please don't kill gerrit trying to clone it
* 17:55 ^d: force running jgit gc on pywikibot/compat
* 17:41 ^d: restarting gerrit
* 16:58 logmsgbot: reedy synchronized wmf-config/
* 16:52 logmsgbot: reedy synchronized wmf-config/
* 16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Revert live hack of setting testwiki to zh'
* 16:24 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'debugging over'
* 16:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'testwiki wgLanguageCode to zh'
* 16:10 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'debug'
* 15:52 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'debug'
* 14:32 akosiaris: adding bigtop-tomcat package to apt for analytics
* 14:13 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'debug'
* 11:32 paravoid: updating gdnsd in apt to 1.9.0-1~precise1
* 08:41 akosiaris: added oozie and hue packages to apt.wikimedia.org as per analytics team request
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 17 02:47:27 UTC {{Gerrit|2013}}
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 17 02:31:39 UTC {{Gerrit|2013}}
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 17 02:17:00 UTC {{Gerrit|2013}}
* 00:09 logmsgbot: krinkle synchronized php-1.22wmf9/extensions/VisualEditor  'I12f52719ecafd7488bb00419'
* 00:07 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor  'I12f52719ecafd7488bb00419'
 
== July 16 ==
* 23:38 logmsgbot: krinkle Finished syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
* 23:31 logmsgbot: krinkle Started syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
* 23:15 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo  'Update Echo to master'
* 23:14 logmsgbot: bsitu synchronized php-1.22wmf10/extensions/Echo  'Update Echo to master'
* 21:00 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 20:58 springle: finished rechecking tag_summary enwiki re bug {{Gerrit|51254}} comment 11
* 20:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 18:04 springle: rechecking tag_summary enwiki re bug {{Gerrit|51254}} comment 11
* 17:54 Ryan_Lane: deployed LdapAuthentication {{Gerrit|fdb01877}} to virt0 and purged memcache to remove bad memcache entries for users
* 17:10 ^demon: gitblit restarted to grab a bugfix
* 16:59 mwalker: updated payments-listeners from {{Gerrit|8a38a49c694b333cf04df16f8c405fbad847d954}} to {{Gerrit|b2ac16f5284c56ea1ba475aba9dae9cc476f0f6d}}
* 16:33 cmjohnson1: that's sdb2 on manganese
* 16:31 cmjohnson1: swapping sdb1 on manganese (software raid1)  rt5442
* 15:34 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 14:54 hashar: zuul / jenkins  -merge jobs are no more triggered by Zuul since it detects whether a patch apply cleanly on the tip of the branch. That saves up a bunch of overhead {{bug|{{Gerrit|51395}}}} and {{gerrit|{{Gerrit|73947}}}}
* 12:58 mark: Sending all European mobile traffic to esams
* 12:21 mark: Sending dutch mobile traffic to esams
* 12:06 mark: Created georecord m.wikimedia.org
* 10:30 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php  ' now uses ['bits'] {{gerrit|{{Gerrit|71777}}}}'
* 10:29 logmsgbot: hashar synchronized wmf-config/CommonSettings.php  ' now uses ['bits'] {{gerrit|{{Gerrit|71777}}}}'
* 10:22 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php  '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
* 10:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php  '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
* 09:55 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'ULS: Bug {{Gerrit|51019}}'
* 09:53 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/  'ULS to master'
* 09:49 logmsgbot: nikerabbit synchronized php-1.22wmf10/extensions/UniversalLanguageSelector/  'ULS to master'
* 09:42 ori-l: Resumed reporting of MediaWiki fatals & exceptions to Ganglia
* 09:28 hashar: upgrading packages on gallium
* 09:13 apergos: restarting gerrit
* 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 16 02:43:55 UTC {{Gerrit|2013}}
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 16 02:28:30 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 16 02:15:23 UTC {{Gerrit|2013}}
* 01:01 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VisualEditor to master again
* 00:53 logmsgbot: catrope Started syncing Wikimedia installation... : Update VisualEditor to master again
 
== July 15 ==
* 23:40 cmjohnson1: reinstalling carbon
* 23:06 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth  'Fix sul2 regression'
* 23:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php  'Slightly increase Parsoid dequeue rate'
* 22:31 logmsgbot: kaldari synchronized php-1.22wmf10/extensions/WikiLove/WikiLove.hooks.php  'Fixing WikiLove regression on wmf10'
* 21:51 ori-l: All EventLogging services start/running; data looks good.
* 20:47 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable VisualEditor for all users (anons too) on enwiki'
* 20:45 Krinkle: Graceful reload of Zuul to fast-forward deployment to Ia53d412b029205
* 20:41 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 20:30 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 20:26 ori-l: Disabled MediaWiki errors Ganglia module on vanadium
* 20:03 ori-l: Rebooting vanadium to complete kernel upgrade to 3.2.0-49.
* 19:49 ori-l: Shutting down EventLogging services on vanadium ahead of Iba8cc5d7b deployment.
* 18:59 hashar: labstore3 system CPU rocketed from ~10% to ~60%
* 18:43 cmjohnson1: powercycling mw1163
* 18:42 cmjohnson1: depooling mw1163 to troubleshoot DIMM error
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non wikipedia to 1.22wmf10
* 18:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.22wmf10
* 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikivoyage, wikiversity to 1.22wmf10
* 18:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, closed, special to 1.22wmf10
* 17:55 cmjohnson1: powering down mw1173
* 17:36 cmjohnson1: depooling mw1173 to troubleshoot DIMM Failure
* 17:10 RobH: Fixing blog setup of themes, theme may reset to defaults for next few minutes as i tinker with it.
* 15:28 MaxSem: Recreating Solr index
* 12:41 apergos: rebooting kaulen to pick up some upgrades, per rt {{Gerrit|5460}}
* 02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 15 02:42:20 UTC {{Gerrit|2013}}
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 15 02:27:57 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul 15 02:14:48 UTC {{Gerrit|2013}}
 
== July 14 ==
* 18:44 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php  'Throttle titles per template update job further from 10 to 6'
* 17:50 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php  'Re-enable Parsoid updates after throttling template update rate'
* 16:55 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Gadgets/
* 16:54 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Gadgets/
* 14:05 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Resync for boxes that came back up'
* 14:02 mark: Repooled stolen apaches back in the appserver cluster
* 14:00 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js  'touch'
* 13:51 mark: Depooled stolen apaches from the api cluster
* 13:21 mark: Upping PyBal weight to 15 for stolen appservers
* 13:20 RoanKattouw: Hand-syncing CommonSettings.php with dsh to set $wgParsoidSkipRatio = 0
* 13:18 mark: Stealing 10 app servers, for the API pool
* 13:06 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Set $wgParsoidSkipRatio to 1 to let the API cluster breathe'
* 12:55 RoanKattouw: Restarted Apache on mw1118, had to stop by hand using killall -9 (/etc/init.d/apache stop didn't work)
* 12:39 RoanKattouw: Restarted pybal on lvs1003
* 12:38 RoanKattouw: Restarted pybal on lvs1006 (set api depool threshold to .4 from .6)
* 06:39 apergos: powercycled mc1005, unreachable via console, from ganglia guess it was in swap death
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 14 02:07:10 UTC {{Gerrit|2013}}
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sun Jul 14 02:02:31 UTC {{Gerrit|2013}}
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul 14 02:01:43 UTC {{Gerrit|2013}}
 
== July 13 ==
* 02:30 mutante: kernel upgrade, rebooting zirconium
* 02:30 mutante: copied etherpad-lite from lucid to precise
* 02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 13 02:22:51 UTC {{Gerrit|2013}}
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 13 02:13:55 UTC {{Gerrit|2013}}
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul 13 02:07:36 UTC {{Gerrit|2013}}
* 00:17 mutante: swift install on iron (swift cleaner & utils only) had a depends issue  python-swift (= 1.5.0-3) but 1.7.4-0ubuntu2+wmf1 is installed, upgraded anyway with -f
* 00:07 mutante: installing package upgrades on iron
 
== July 12 ==
* 23:40 gwicke: updated Parsoid to {{Gerrit|f6d3742}}
* 23:38 springle: restarted db32 slave
* 23:32 springle: finished tag_summary rebuild
* 22:40 logmsgbot: aaron cleared profiling data
* 22:34 mutante: completely stopped and then started squid front and back on cp1001 (had socket already in use errors)
* 22:26 springle: rebuilding tag_summary on enwiki
* 22:25 mutante: restarting squid on cp1001
* 22:07 mutante: restarting job runners
* 21:07 springle: stopped db32 mysql slave threads
* 19:58 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'updating Parsoid'
* 19:57 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php  'updating Parsoid'
* 18:45 RobH: authdns-update all ulsfo now 4XXX
* 18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Remove wgOldChangeTagsIndex on all remaining wikis'
* 18:00 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'updating Parsoid master'
* 17:59 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php  'updating Parsoid master'
* 16:27 Krinkle: Graceful reload of Zuul to forward deployment to I4de1670e11d58e
* 16:10 logmsgbot: reedy synchronized docroot and w
* 15:51 logmsgbot: reedy synchronized docroot and w
* 15:35 logmsgbot: reedy synchronized docroot and w
* 15:33 logmsgbot: reedy synchronized docroot and w
* 15:31 logmsgbot: reedy synchronized docroot and w
* 15:26 logmsgbot: reedy synchronized docroot and w
* 15:20 Krinkle: Resetting [[commons:User:CommonsDelinker]] account e-mailaddress (bug {{Gerrit|51016}})
* 10:43 mark: Sending European wikidata and wikivoyage traffic to esams
* 05:13 mutante: restarting ircecho on neon
* 03:03 mwalker: and again... payments from {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}} to {{Gerrit|2a9169765b94e036c4d2ee0cada362e4de6e212e}}
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 12 02:47:23 UTC {{Gerrit|2013}}
* 02:41 mwalker: attempting again on the payments cluster! from {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}} to {{Gerrit|7a5a3c376ed0b7f25c5fa0e9f14d6cc67e543982}}
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 12 02:34:20 UTC {{Gerrit|2013}}
* 02:21 mwalker: aaand nope; didn't work, reverting payments back to {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}}
* 02:18 mwalker: trying again -- updating payments from {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}} to {{Gerrit|8a801e809de37bc8595b19262faa9b9e47bfbd6e}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul 12 02:16:00 UTC {{Gerrit|2013}}
* 01:40 mwalker: aaaahhhhh reverting payments to {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}} because banners are sending the wrong gateway (and I was ignoring it before!)
* 01:33 mwalker: updating payments (including the formchooser) from {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}} to {{Gerrit|1a2130bdba1157b095fde21d77aebb0a8b1ac88e}}
* 00:49 springle: paused updating change_tags indexes on remaining wikis in bug {{Gerrit|40867}} comment 6 (adds done, drops todo)
* 00:11 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
* 00:09 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/GeoData/solrupdate.php
 
== July 11 ==
* 23:46 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VE to master
* 23:38 logmsgbot: catrope Started syncing Wikimedia installation... : Update VE to master
* 22:53 springle: updating change_tags indexes on remaining wikis in bug {{Gerrit|40867}} comment 6
* 22:42 logmsgbot: tstarling synchronized wmf-config
* 22:41 logmsgbot: tstarling restarted twemproxy on all servers
* 22:40 logmsgbot: tstarling synchronized wmf-config/mc.php
* 22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy.yaml
* 22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy-pmtpa.yaml
* 22:33 logmsgbot: springle synchronized wmf-config/InitialiseSettings.php  'simplewiki change_tags indexes updated, bug {{Gerrit|40867}}'
* 21:49 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php  'Config changes for VE survey'
* 21:38 RobH: move internal ip, not wiki. bleh
* 21:38 RobH: authdns update for zinc to move internal wiki
* 21:38 Ryan_Lane: deployed squid change to switch test to mw1017
* 21:30 Ryan_Lane: depooling mw1017 to make it the new test.wm.o
* 21:23 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
* 21:22 RobH: authdns-update for tmc1/2 and mgmt
* 21:12 logmsgbot: spage Started syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
* 20:21 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 20:21 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 19:11 LeslieCarr: clearing out stuck icinga processes , except higher than normal icinga spam and do not worry
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki testwiki testwikidatawiki mediawikiwki to 1.22wmf10
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all 'pedias to 1.22wmf9
* 17:22 cmjohnson1: commenting out mc1-16 in node groups
* 17:17 sbernardin: bringing down ms-be5 for hardware replacement
* 16:56 gwicke: updated Parsoid to {{Gerrit|6d17a80}}
* 15:13 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
* 15:00 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
* 14:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf10 and rebuild l10n cache
* 14:46 logmsgbot: reedy synchronized docroot and w
* 14:45 logmsgbot: reedy synchronized php-1.22wmf10  'initial sync of 1.22wmf10'
* 10:21 Krinkle: Graceful reload of Zuul (forward to I3fc6baba8c6d65)
* 07:39 hashar: gallium : fix permissions for android-common nightly build (need to be jenkins owned, not jenkins-slave) {{bug|{{Gerrit|51137}}}}
* 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 11 02:51:16 UTC {{Gerrit|2013}}
* 02:38 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul 11 02:37:40 UTC {{Gerrit|2013}}
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 11 02:17:47 UTC {{Gerrit|2013}}
* 00:26 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Update VE to pick up cssText fix'
* 00:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Update VE to pick up cssText fix'
 
== July 10 ==
* 23:39 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Update VE to master'
* 23:38 RoanKattouw: [00:23] gwicke !updated Parsoid to {{Gerrit|f571475}}
* 23:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Update VE to master'
* 23:30 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth  'Updating CentralAuth on wmf9 for SUL'
* 22:27 mutante: reprepro copying etherpad from wikimedia-lucid to wikimedia-precise
* 21:36 mutante: removing old racktables Apache site and other remnants from hooper
* 21:03 mutante: package upgrades on hooper (etherpad)
* 18:29 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/  'ULS perf fix'
* 18:25 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/  'ULS perf fix'
* 18:10 gwicke: updated Parsoid to {{Gerrit|8ea8ab0}}
* 17:58 mutante: reverting removing spence, NTP server issue
* 17:47 mutante: DNS update - killing spence
* 17:21 Jeff_Green: dist-upgrade and reboot aluminium, db78
* 17:21 Jeff_Green: grosley switched to frack puppetmaster
* 17:21 Jeff_Green: db78 switched to frack puppetmaster
* 16:35 mutante: upgrading ruby and other packages on sockpuppet and stafford
* 14:40 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/{{Gerrit|72957}}'
* 14:36 apergos: restarted gerrit, its queue was backed up
* 14:24 hashar: gerrit being super slow, it has 32 items in its queue ( lot of  git-upload-pack p/mediawiki/core.git ). Ganglia for manganese does not show anything obvious.
* 14:22 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/GeoData/solrupdate.php
* 14:20 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
* 11:34 mark: Upgraded and restarted eqiad mobile caches (front/back)
* 09:48 mark: Upgrading Varnish on bits servers
* 09:17 mark: Inserted varnish 3.0.3plus-rc1-wm12 packages into the precise-wikimedia APT repository
* 03:03 logmsgbot: tstarling Finished syncing Wikimedia installation... : ProofreadPage update for bug {{Gerrit|51085}}
* 02:59 logmsgbot: tstarling Started syncing Wikimedia installation... : ProofreadPage update for bug {{Gerrit|51085}}
* 02:53 logmsgbot: tstarling synchronized php-1.22wmf9/extensions/ProofreadPage/ProofreadPage.body.php
* 02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 10 02:42:32 UTC {{Gerrit|2013}}
* 02:39 logmsgbot: tstarling synchronized php-1.22wmf8/maintenance/showJobs.php
* 02:38 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueueRedis.php
* 02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueue.php
* 02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/MappedIterator.php
* 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul 10 02:29:28 UTC {{Gerrit|2013}}
* 02:22 Tim: on tin: attempted to do a git pull in wmf8 but it updated a whole lot of extensions for some reason. Checking out the old versions manually.
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 10 02:15:40 UTC {{Gerrit|2013}}
* 01:16 gwicke: finished Parsoid config update with latest dependencies
* 01:14 gwicke: starting Parsoid config update with latest dependencies
 
== July 9 ==
* 23:17 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/ContributionReporting  'Updating ContributionReporting for {{Gerrit|2013}} fundraiser'
* 22:16 mwalker: updating payments.wm.o from {{Gerrit|eee8689ab6466d51f9991c90f73004ad6c9d0b5b}} to {{Gerrit|1c360e2cde650d1195f04aee294ddd3e0f5f9052}}
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/MobileFrontend
* 22:03 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
* 21:42 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:27 Jeff_Green: db78 switched to mariadb
* 21:21 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:13 Robh: authdns-update for bastion5001.mgmt
* 20:13 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'activating Disambiguator extension on all WMF wikis'
* 18:26 LeslieCarr: authdns-update on ns0
* 18:21 logmsgbot: mlitn synchronized wmf-config/  'Enable ArticleFeedbackv5 auto-archive'
* 18:02 logmsgbot: mlitn synchronized wmf-config/  'Update ArticleFeedbackv5 & Echo config'
* 17:58 logmsgbot: mlitn synchronized php-1.22wmf9/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 17:57 logmsgbot: mlitn synchronized php-1.22wmf8/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 16:50 logmsgbot: anomie synchronized php-1.22wmf9/extensions/AbuseFilter  'Update AbuseFilter to master for global filter bug fixes'
* 16:39 cmjohnson1: going to tinkering with dysprosium...will see some icinga alerts
* 15:10 ottomata: re-starting puppet on analytics1006,{{Gerrit|1008}},{{Gerrit|1009}}
* 15:01 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 14:40 ottomata: stopping puppet on analytics1006, {{Gerrit|1008}} and {{Gerrit|1009}} to manually apply udp2log puppetization
* 13:20 paravoid: upgraded ceph to 0.66
* 13:04 logmsgbot: hashar synchronized php-1.22wmf9/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
* 13:04 hashar: touched and synced extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js on php-1.22wmf8 and php-1.22wmf9
* 13:04 logmsgbot: hashar synchronized php-1.22wmf8/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
* 11:48 paravoid: apt: updating to ceph 0.66
* 09:34 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'Bug {{Gerrit|51019}}'
* 09:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/  'ULS to master'
* 09:30 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:56 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'ULS phase 5'
* 08:53 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:44 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/  'ULS to master'
* 04:17 logmsgbot: tstarling synchronized php-1.22wmf8/includes/api/ApiPurge.php
* 04:16 logmsgbot: tstarling synchronized php-1.22wmf8/includes/WikiPage.php
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul  9 02:38:31 UTC {{Gerrit|2013}}
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul  9 02:26:54 UTC {{Gerrit|2013}}
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul  9 02:14:12 UTC {{Gerrit|2013}}
* 01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/api/ApiPurge.php
* 01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/WikiPage.php
* 00:46 gwicke: updated Parsoid to {{Gerrit|d0e3603}}
 
== July 8 ==
* 23:56 mutante: graceful'ing Apaches
* 23:53 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VE and AbuseFilter
* 23:43 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VE and AbuseFilter
* 22:48 mutante: restarting pdns on ns0
* 22:47 mutante: DNS update - point download.mw away from empty dir on kaulen
* 21:36 Robh: updating blog
* 19:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf9
* 19:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf9
* 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf9
* 18:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikimania and wikivoyage to 1.22wmf9
* 18:36 logmsgbot: reedy synchronized php-1.22wmf8/extensions/TimedMediaHandler/
* 18:35 logmsgbot: reedy synchronized php-1.22wmf9/extensions/TimedMediaHandler/
* 17:31 mark: Gave row A and row C upload caches equal share of traffic again
* 17:16 logmsgbot: csteipp synchronized php-1.22wmf9/includes/SkinTemplate.php  'Deploy missing function for SUL2'
* 16:44 mutante: installing package upgrades on kaulen (bugzilla)
* 15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:38 paravoid: apt: installing gdnsd 1.8.3-1~precise1
* 12:44 hashar: jenkins: reduced number of executors on master from 2 to 0.
* 10:28 paravoid: (re)installing colby, rubidium, eeden
* 09:09 hashar: srv281 and srv1173 have ssh timeout when synchronizing configuration.
* 09:09 logmsgbot: hashar synchronized images/wikimedia-button.png  'added to git as of {{gerrit|{{Gerrit|71775}}}}'
* 08:36 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php  'touch config for $wgLoadScript beta/prod variance {{gerrit|{{Gerrit|70322}}}}'
* 08:35 logmsgbot: hashar synchronized wmf-config/CommonSettings.php  '$wgLoadScript beta/prod variance {{gerrit|{{Gerrit|70322}}}}'
* 08:33 hashar:  Testing out $wgLoadScript beta/prod variance {{gerrit|{{Gerrit|70322}}}}, pulled change and running sync-common on srv193
* 04:44 Tim: on enwiki: discarded refreshLinks2 entries for templates Yesno, If_pagename, Category_handler/numbered, Category_handler, by setting their root job timestamp to a fake new timestamp. See bug {{Gerrit|50785}}
* 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul  8 02:15:21 UTC {{Gerrit|2013}}
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul  8 02:09:13 UTC {{Gerrit|2013}}
 
== July 7 ==
* 23:29 Tim: on fenari: removed /a/common directory so that mwscript will work
* 23:28 Tim: restarted morebots
* 02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul  7 02:14:42 UTC {{Gerrit|2013}}
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul  7 02:09:01 UTC {{Gerrit|2013}}
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jul  7 02:05:09 UTC {{Gerrit|2013}}
 
== July 6 ==
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul  6 02:38:04 UTC {{Gerrit|2013}}
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul  6 02:27:28 UTC {{Gerrit|2013}}
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jul  6 02:14:24 UTC {{Gerrit|2013}}
 
== July 5 ==
* 21:55 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/  'Update VE to master in wmf8 (did wmf9 earlier)'
* 21:25 mutante: graceful Apaches, activate "ontology" alias for wikidata
* 21:10 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor/  'Update VE to master in wmf9'
* 20:56 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/{{Gerrit|72144}}'
* 18:34 sbernardin: bringing down colby to remove extra drives
* 13:54 hashar:  {{bug|{{Gerrit|50228}}}}
* 13:48 hashar: Jenkins: reduced # of executors on [master] from 4 to 2.  Will bring it to 0 next monday (see {{bug|{{Gerrit|50228}}}})
* 05:13 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 05:09 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 03:01 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 02:58 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul  5 02:51:29 UTC {{Gerrit|2013}}
* 02:45 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul  5 02:44:41 UTC {{Gerrit|2013}}
* 02:40 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul  5 02:38:54 UTC {{Gerrit|2013}}
* 02:34 Tim: testing l10nupdate
* 02:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul  5 02:03:10 UTC {{Gerrit|2013}}
* 02:04 logmsgbot: LocalisationUpdate failed (1.22wmf9) at Fri Jul  5 02:03:09 UTC {{Gerrit|2013}}
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul  5 02:03:09 UTC {{Gerrit|2013}}
 
== July 4 ==
* 22:10 LeslieCarr: icinga restarted, was down due to bad check
* 16:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 13:58 pp-pdf1: restarted all services
* 13:57 pp-pdf1: - update mwlib to 0.15.10
* 12:01 akosiaris: uploaded buck version 0+git20130612-0+wmf1 at apt.wikimedia.org
* 09:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 09:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:07 logmsgbot: tstarling synchronized README
* 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul  4 02:34:13 UTC {{Gerrit|2013}}
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul  4 02:27:40 UTC {{Gerrit|2013}}
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul  4 02:14:32 UTC {{Gerrit|2013}}
* 01:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Fix <a href bug'
* 01:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Fix <a href bug'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/collapsibleTabs.js  'touch'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/vector.js  'touch'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js  'touch'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf9/resources/Resources.php  'collapsibleTabs fix'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js  'touch'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/vector.js  'touch'
* 01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js  'touch'
* 01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/Resources.php  'collapsibleTabs fix'
* 00:52 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Latest fixes'
* 00:51 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Latest fixes'
* 00:46 RoanKattouw: Add cp1056, cp1057, cp1069 and cp1070 to /etc/dsh/group/bits , they were missing and so the purge-varnish script didn't actually work
 
== July 3 ==
* 23:30 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Update VE again'
* 23:30 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Update VE again'
* 23:25 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 23:18 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 22:12 ^d: restarting gerrit for firefox workaround
* 20:42 mark: Trippled PyBal weights of row A upload caches to reduce traffic on row C
* 20:38 ottomata: restarting gmetad on nickel to reload an rrd file
* 20:37 hashar: reloading Zuul to lower gerrit debug messages {{gerrit|{{Gerrit|71916}}}}
* 19:47 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/CentralNotice  'Updating CentralNotice to master on wmf9 (mobile stability and emergency messaging)'
* 19:46 logmsgbot: mwalker synchronized php-1.22wmf8/extensions/CentralNotice  'Updating CentralNotice to master on wmf8 (mobile stability and emergency messaging)'
* 19:45 binasher: migrated ishmael to neon (eqiad)
* 19:45 binasher: pdns hung on ns2, restarted
* 19:41 gwicke: updated Parsoid to {{Gerrit|a53e198}}
* 18:07 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
* 17:59 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 17:32 gwicke: updated Parsoid to {{Gerrit|732c3e6}}
* 17:28 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 17:25 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
* 17:18 Ryan_Lane: deploying change {{Gerrit|60783}} to virt0
* 17:10 RobH: authdns-update rt {{Gerrit|4547}}
* 16:56 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 15:39 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|71375}}/'
* 15:30 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|70597}}/'
* 15:26 logmsgbot: maxsem synchronized wmf-config/mobile.php  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|71813}}/'
* 14:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Wikibase changes'
* 14:12 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Wikibase changes'
* 14:06 logmsgbot: demon synchronized php-1.22wmf9/extensions/Wikibase/  'Updating wikibase to {{Gerrit|5a6511760562dc5b8fae033492af704599141640}}'
* 13:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Wikibase: I2e9d03b2'
* 13:37 logmsgbot: maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/{{Gerrit|70557}}/'
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul  3 02:38:05 UTC {{Gerrit|2013}}
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul  3 02:28:35 UTC {{Gerrit|2013}}
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul  3 02:15:00 UTC {{Gerrit|2013}}
* 00:02 ^d: manganese back up
* 00:00 ^d: rebooting manganese (gerrit)
 
== July 2 ==
* 22:47 logmsgbot: bsitu synchronized php-1.22wmf8/extensions/Echo  'update Echo to master'
* 22:46 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo  'update Echo to master'
* 20:57 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 20:46 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:41 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
* 19:32 mwalker: updated production civi from {{Gerrit|6e78674302b3e4189248ba8234deaf3d2ee35c54}} to {{Gerrit|5bcbab5c7d374473f05bf49b41ffd2c531541510}}
* 19:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
* 18:08 bblack: upgraded vhtcpd to 0.0.8 on brewster
* 17:48 sbernardin: replacing disks slot5 & slot6 on db45
* 17:38 sbernardin: replacing disk on db44
* 17:29 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: bumping loginwiki to wmf9
* 16:30 cmjohnson1: disabling puppet freshness check
* 16:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:16 sbernardin: rebooting mw80 for bios update
* 16:11 cmjohnson1: depooling mw80
* 16:10 bblack: updated vhtcpd package on brewster to 0.0.7-1
* 16:09 Jeff_Green: authdns update, switching mx records for donate.wikimedia.org to mchenry/sodium
* 16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:15 paravoid: apt: include jmxtrans 242-1
* 14:59 pp-pdf1: restarted all services
* 14:59 pp-pdf1: - update mwlib to 0.15.9
* 10:18 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:10 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:03 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:01 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 09:27 hashar: jenkins upgraded.
* 09:15 apergos: reinstalling dataset1001
* 09:01 hashar: gallium : upgraded Jenkins from 1.509.1 to 1.509.2  {{bug|50522}}
* 08:46 hashar: gerrit : added Tntnet user to the 'openzim' group.  Grants him push/submit rights on the openzim git repository.  Requested by Kelson.
* 08:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:17 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php  'ULS on enwiki part 2'
* 08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'ULS on enwiki part 1'
* 08:11 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/  'ULS to master'
* 05:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul  2 05:25:52 UTC {{Gerrit|2013}}
* 05:22 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul  2 05:21:48 UTC {{Gerrit|2013}}
* 05:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul  2 05:14:00 UTC {{Gerrit|2013}}
* 05:06 Tim: testing LU with a real shell (not /bin/false)
* 02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul  2 02:23:13 UTC {{Gerrit|2013}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul  2 02:15:57 UTC {{Gerrit|2013}}
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul  2 02:09:03 UTC {{Gerrit|2013}}
* 01:22 gwicke: updated Parsoid to {{Gerrit|65bd7ae9}}
* 00:56 logmsgbot: ori testing dologmsg. mutante ftw.
* 00:06 mutante: restarted logmsgbot
 
== July 1 ==
* 23:51 ori-l: logsmsgbot's cidr-filtering was broken by https://gerrit.wikimedia.org/r/#/c/{{Gerrit|69624}}/, manifested now because of neon restart earlier
* 22:49 mutante: restarted snmptt and snmptrapd on neon
* 22:43 mutante: deleting files from snmptt spool dir on neon
* 22:18 LeslieCarr: restarting neon for new kernel
* 21:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable VisualEditor for all logged-in users on enwiki'
* 21:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Plumbing for VE settings'
* 21:34 gwicke: updated Parsoid to {{Gerrit|613287b}}
* 21:24 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for i18n changes in VE
* 21:14 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for i18n changes in VE
* 21:04 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/  'Update VE to master in wmf8 (did wmf9 earlier)'
* 21:02 logmsgbot: catrope synchronized php-1.22wmf8/extensions/TimedMediaHandler/  'Deploy TMH API cherry-pick to wmf8 (already in wmf9)'
* 20:55 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Updating VE to master in wmf9; for real this time'
* 20:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor  'Updating VE to master in wmf9'
* 20:36 Krinkle: Zuul was clogging up with ~ {{Gerrit|350}} queued up events. Added 1 extra executor on Jenkins master and slave.
* 20:11 gwicke: updated Parsoid to {{Gerrit|55d011d}}
* 18:02 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php  'enablign silent redirects for test wikis'
* 18:00 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php  'pushing config flag for centralauth silent login'
* 17:22 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth  'Updating wmf9 to latest CentralAuth'
* 16:00 cmjohnson1: reseating disk 3 db44
* 14:58 ^demon: gerrit: brought offline, removed /var/lib/gerrit2/review_site/cache/*, brought back up
* 13:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 12:09 mark: Depooled old eqiad mobile cache servers
* 11:25 mark: Depooled old eqiad upload varnish servers
* 10:54 apergos: rebooting labstore3, /srv reporting errors, couldn't xfs_check it
* 10:08 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/Campaigns  'I58a424c70: Restore extensions/Campaigns submodule; was accidentally removed by change I2cafcb595.'
* 03:53 Tim: restarted morebots
* 03:33 TimStarling: testing new scap
 
== June 29 ==
* 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 29 02:11:56 UTC {{Gerrit|2013}}
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jun 29 02:06:08 UTC {{Gerrit|2013}}
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 29 02:03:41 UTC {{Gerrit|2013}}
* 00:00 jvandavier: jvandavier is the best
 
== June 28 ==
* 23:59 K4-713: updated payments cluster to {{Gerrit|eee8689ab646}}
* 20:55 gwicke: updated Parsoid to {{Gerrit|454f87a}}
* 19:20 mutante: attempting clean stop and restart of snmptt service on neon
* 19:08 mutante: deleting /var/spool/snmptt/ files on neon running out of inodes
* 17:57 ^demon: restarting gerrit, bringing back missing documentation lost during upgrade
* 17:33 mutante: upgrading binutils and libldap on bast1001
* 17:20 mutante: installing some pkg upgrades on fenari
* 16:56 paravoid: cause of brief outage: operator error caused restart of varnish on all upload caches eqiad, cascaded into a swift overload, cascaded into imagescaler outage & page
* 16:39 LeslieCarr: restarting snmptrapd and smptt on neon
* 16:28 hashar: created operations/debs/jenkins-debian-glue and force pushed upstream sources from https://github.com/mika/jenkins-debian-glue
* 15:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf9
* 14:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 14:05:42 UTC {{Gerrit|2013}}
* 13:58 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 13:56:55 UTC {{Gerrit|2013}}
* 13:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 13:51:49 UTC {{Gerrit|2013}}
* 13:44 logmsgbot: reedy Finished syncing Wikimedia installation... : Scaptastic
* 13:42 logmsgbot: reedy Started syncing Wikimedia installation... : Scaptastic
* 13:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything back from wmf7 to wmf8
* 13:39 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
* 13:22 logmsgbot: reedy Started syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
* 13:21 logmsgbot: reedy synchronized wmf-config/
* 13:16 hashar: jenkins: reducing # of executor on master from 4 to 2
* 12:24 hashar: jenkins: migrating mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests from master to slaves
* 09:57 hashar: gallium : updating some files under /srv/org to belong to jenkins-slave:jenkins-slave instead of jenkins.
* 09:24 hashar: jenkins: made the .erb linting jobs to be voting {{gerrit|{{Gerrit|70994}}}}
* 06:28 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to wmf8
* 05:07 Krinkle: Note, anons were eventually getting HTTP {{Gerrit|500}} too (even for enwiki Main Page) as action=view threw as well squid cache got purged somehow and apaches unable populate it back. And then LVS depooled stuff for apache serving {{Gerrit|500}} on action=view for main page.
* 05:07 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php  're-enabling GeoData'
* 05:05 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
* 05:00 Tim: reverting to 1.22wmf7
* 04:58 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php  'disable GeoData'
* 04:55 RoanKattouw: sync-dir php-1.22wmf8/cache/l10n
* 04:55 Krinkle: catrope Removing that extension from extension-list
* 04:54 Krinkle: Cause from GeoData "coordinates" magic word. Deployed through broken i18n stuff from WikibaseDataModel deployment earlier that caused fatal during the scap just now
* 04:54 Krinkle: About 20 minutes ago the site went down for logged in users (and anon users hitting an uncached page) throwing HTTP {{Gerrit|500}}
* 04:54 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
* 04:53 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf9.php
* 04:51 logmsgbot: catrope Finished syncing Wikimedia installation... : Trying to fix the site
* 04:47 logmsgbot: catrope Started syncing Wikimedia installation... : Trying to fix the site
* 04:47 RoanKattouw: Running scap in an attempt to fix things
* 04:46 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
* 04:31 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 04:22 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 02:18:38 UTC {{Gerrit|2013}}
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 02:10:26 UTC {{Gerrit|2013}}
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 02:07:05 UTC {{Gerrit|2013}}
* 01:53 mwalker: updating payments from {{Gerrit|bf6cbd63cb3cc6148f8abf35767c7dba4ce92b0e}} to {{Gerrit|9d7748c2e65ce8208be9ba07dfb0558bd3152bbe}}
* 00:57 gwicke: updated Parsoid to {{Gerrit|68a086a}}
* 00:00 mutante: graceful Apaches, activate wiikipedia and wekipedia
 
== June 27 ==
* 22:31 gwicke: updated Parsoid to {{Gerrit|8b38dcc}}
* 20:52 hashar: jenkins: migrating mediawiki-core-phpunit-parser from master to slaves
* 20:44 logmsgbot: spage synchronized php-1.22wmf8/extensions/Campaigns  'updating Campaigns in 1.22wmf8'
* 20:43 hashar: deployment-prep jenkins: migrating mediawiki-core-phpunit-misc from master to slaves
* 20:42 logmsgbot: spage synchronized php-1.22wmf8/extensions/GuidedTour  'updating GuidedTour in 1.22wmf8'
* 20:36 logmsgbot: spage synchronized php-1.22wmf9/extensions/GuidedTour  'updating GuidedTour in 1.22wmf9'
* 20:33 hashar: jenkins: migrating mediawiki-core-phpunit-databaseless form master to slaves
* 20:21 hashar: jenkins: migrating mediawiki-core-phpunit-api from master to slaves
* 18:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Add Extension:NewUserMessage to de.wikiversity'
* 18:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:35 Reedy: Created Echo tables on enwikivoyage
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and testwikidatawiki to 1.22wmf9
* 18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedias to 1.22wmf8
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf8
* 17:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf9
* 17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/Wikibase  'Updating to master of 1.22wmf6 branch'
* 17:37 logmsgbot: reedy synchronized php-1.22wmf7/extensions/Wikibase  'Updating to master of 1.22wmf6 branch'
* 17:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf8
* 17:21 hashar: zuul: removing --force-message from layout {{gerrit|{{Gerrit|70849}}}} and reloading zuul. Caused {{bug|{{Gerrit|50300}}}}
* 17:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 17:15:49 UTC {{Gerrit|2013}}
* 17:13 hashar: Zuul/Jenkins merging bug is {{bug|{{Gerrit|50300}}}}
* 17:12 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jun 27 17:11:43 UTC {{Gerrit|2013}}
* 17:11 hashar: Jenkins/Zuul is no more merging changes due to  --force-message options no more being available in Gerrit :/
* 17:02 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 17:01:20 UTC {{Gerrit|2013}}
* 17:01 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 17:00:37 UTC {{Gerrit|2013}}
* 16:45 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
* 16:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
* 16:28 mark: Corrected IPv6 problem, repooled the new servers, depooled the old bits servers
* 16:19 logmsgbot: reedy synchronized php-1.22wmf9/extensions/WikibaseDataModel
* 16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase/
* 16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Diff
* 16:14 logmsgbot: reedy synchronized php-1.22wmf9/extensions/DataValues
* 16:12 logmsgbot: reedy synchronized php-1.22wmf9/
* 16:03 mark: Depooled row C bits caches, repooled old bits caches to investigate IPv6 problem
* 15:56 logmsgbot: reedy synchronized docroot and w
* 15:47 mark: Depooled old eqiad bits caches
* 15:41 mark: Pooled new eqiad bits caches
* 14:50 logmsgbot: mark synchronized wmf-config/squid.php
* 14:50 logmsgbot: mark synchronized wmf-config/squid.php
* 14:11 ^demon: gerrit: running puppet and restarting service
* 09:39 mark: Pooled new eqiad upload caches with 1% load
* 03:09 Tim: on fenari: moved away the outdated source copy that was at /home/wikipedia/common . Will update noc.
* 02:56 Tim: added springle to wmf and ops LDAP groups
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 02:19:05 UTC {{Gerrit|2013}}
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 02:13:40 UTC {{Gerrit|2013}}
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 02:07:37 UTC {{Gerrit|2013}}
* 01:35 gwicke: Roan cleared Parsoid caches
* 01:33 gwicke: updated Parsoid to {{Gerrit|091ebece}}
* 00:24 gwicke: updated Parsoid to {{Gerrit|eccca39}}
* 00:17 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js  'touch'
* 00:17 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js  'touch'
* 00:10 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for VE update, contained i18n changes
* 00:03 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for VE update, contained i18n changes
 
== June 26 ==
* 23:48 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess/
* 23:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Update VE to master'
* 23:47 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor  'Update VE to master'
* 23:46 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 23:22 gwicke: updated Parsoid to {{Gerrit|8c83524}}
* 22:22 LeslieCarr: killed slow query on db1025
* 21:15 Jeff_Green: changing fundraising db config to point to service aliases
* 18:37 Jeff_Green: commented out cron-msp queue flush in stat1:/etc/cron.d/sendmail to stop pointless cronspam, host is running exim
* 18:13 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess
* 18:10 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess
* 16:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Update VisualEditor to master'
* 16:38 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor  'Update VisualEditor to master'
* 16:08 gwicke: updated Parsoid to {{Gerrit|e79f70a}}
* 14:26 logmsgbot: reedy synchronized php-1.22wmf8/extensions/VisualEditor/modules/ve/ve.js
* 14:24 logmsgbot: reedy synchronized php-1.22wmf7/extensions/VisualEditor/modules/ve/ve.js
* 14:01 logmsgbot: reedy synchronized php-1.22wmf7/resources
* 14:00 logmsgbot: reedy synchronized php-1.22wmf8/resources
* 13:59 logmsgbot: reedy synchronized wmf-config  'touch 1'
* 12:08 hashar: Jenkins let jobs roam on gallium slave node and keep master only for tied jobs.
* 10:39 hashar: jenkins moving unit test jobs for mw extensions to use the slave tmpfs and let them roam on slaves
* 08:42 MaxSem: recreating Solr index
* 08:40 hashar: jenkins installed Simple Theme Plugin
* 08:35 hashar: jenkins reducing number of executors on (master) from 10 to 4.  Most jobs are now running on the 'gallium' slave node.
* 08:35 hashar: jenkins: restarted it twice to finally have it up and running.
* 08:15 hashar: restarting Jenkins to apply plugins updates.
* 07:13 Tim: deploying squid config change to block API DoS attack
* 06:41 paravoid: upgrading ceph/ms-fe.eqiad.wmnet to 0.65, ignore warnings/alerts for the next few hours
* 03:59 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Updating VisualEditor to master'
* 03:58 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor  'Updating VisualEditor to master'
* 02:57 RoanKattouw: Updated Parsoid config for bug {{Gerrit|50173}}
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 26 02:27:09 UTC {{Gerrit|2013}}
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 26 02:13:32 UTC {{Gerrit|2013}}
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jun 26 02:07:35 UTC {{Gerrit|2013}}
* 01:41 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js  'I6e277a9503a1e6003bc7bf9f9468ed7b35552e60'
 
== June 25 ==
* 23:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor  'Updating VisualEditor to master'
* 23:25 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor  'Updating VisualEditor to master'
* 21:47 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving pool4 and 5 search traffic back to eqiad for realz'
* 21:34 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving pool2 and 3 search traffic back to eqiad for realz'
* 21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:25 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving en and prefix search traffic back to eqiad for realz'
* 21:19 LeslieCarr: restarted pybal on lvs1003
* 21:16 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:34 Ryan_Lane: test
* 19:32 Ryan_Lane: test
* 19:26 logmsgbot: reedy synchronized .
* 19:26 Ryan_Lane: upgrading adminbot to 1.7.4 on wikitech-static
* 19:04 logmsgbot: reedy synchronized docroot and w
* 18:34 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving en and prefix search traffic back to pmtpa'
* 18:27 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving en and prefix search traffic back to eqiad'
* 18:14 logmsgbot: reedy synchronized php-1.22wmf8/extensions/UploadWizard
* 17:19 logmsgbot: aaron cleared profiling data
* 16:57 logmsgbot: reedy synchronized wmf-config/
* 15:25 hashar: jenkins: raising # of executors on 'gallium' slave node from 4 to 8
* 15:03 gwicke: updated Parsoid to 5ef05e6c9
* 14:54 paravoid: deluser icinga; rm -rf /home/icinga on all hosts but neon
* 14:45 paravoid: restarting nrpe on all hosts
* 14:25 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 12:37 mark: Pooled the new eqiad mobile frontend caches with lowest weight in PyBal
* 12:27 hashar: jenkins: all mediawiki extension jobs got updated.
* 12:12 logmsgbot: mark synchronized wmf-config/squid.php
* 11:45 mark: Depooled old parsoid cache servers - parsoidcache migration complete
* 11:17 hashar: jenkins: migrating extensions jobs to use gallium slave ( lint , phpcs-HEAD )  or be bound on master (test extensions)
* 10:56 mark: Pooled cp1045 and cp1058 Varnish frontends in the parsoid cache pool
* 08:13 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'ULS phase3'
* 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 25 02:26:25 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 25 02:13:22 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jun 25 02:07:28 UTC 2013
* 01:51 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/  '{{gerrit|If96bed727ba7af7}}'
* 01:49 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/  '{{gerrit|If96bed727ba7af7}}'
* 01:23 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php  'Iac6a3f2f4d: wmgMFLogEvents => true on beta labs'
 
== June 24 ==
* 23:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'I82fe6063a: Disable AFTv4 (Bug 43892)'
* 23:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted  'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
* 23:06 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted  'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
* 22:44 Krinkle: Graceful reload of Zuul to deploy {{gerrit|Ie51c7ce9}}
* 20:54 mutante: DNS update - switching RT over to magnesium and 4.x
* 19:53 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 19:47 gwicke: updated Parsoid to 6ab78c8dd8a
* 19:46 mutante: moving backups from db9 to tridge
* 19:39 mutante: temp shutting down webserver on streber (RT scheduled downtime)
* 18:22 gwicke: updated Parsoid to b3a872fa7
* 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiooks to 1.22wmf8
* 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikiversity, wikinews, wiktionary to 1.22wmf8
* 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf8
* 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia and closed to 1.22wmf8
* 17:09 apergos: er, "shot" many dead snmptt processes
* 17:09 apergos: while kicking icinga into working again on neon, also cleared out a pile of inodes in /var/spool/snmptt again, show many many dead processes too
* 16:55 logmsgbot: krinkle synchronized wmf-config/CommonSettings.php  'I94d74fd8abeba465f'
* 16:55 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php  'I94d74fd8abeba465f'
* 16:48 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/  'I73522f63d9a49a4b7'
* 16:47 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/  'I73522f63d9a49a4b7'
* 15:54 logmsgbot: reedy synchronized wmf-config/
* 15:50 ^demon: restarting gerrit
* 15:49 Reedy: Created WikiLove tables on enwikivoyage
* 14:51 hashar: jenkins: migrating pyflakes and pep8 jobs to slaves having the label 'hasContintPackages'
* 14:09 hashar: jenkins raising 'gallium' slave # of executors from 1 to 4. I have migrated some jobs to run on that slave.
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 24 02:24:40 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Mon Jun 24 02:12:57 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 24 02:07:08 UTC 2013
 
== June 23 ==
* 07:06 logmsgbot: krinkle Finished syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 06:39 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 06:32 RoanKattouw: Made ExtensionMessages-1.22wmf8.php group-writeable
* 06:28 Krinkle: Aborted scap due to php fatal errors with 1.22wmf8 (Permission error on ExtensionMessages-1.22wmf8.php; Missing files: Narayam.i18n.php, WebFonts.i18n.php and WebFonts.alias.php)
* 06:27 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 23 02:16:43 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jun 23 02:09:09 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 23 02:05:29 UTC 2013
 
== June 22 ==
* 15:46 Reedy: tin is down, dead to SSH
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 22 02:19:26 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 22 02:13:00 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 22 02:07:29 UTC 2013
 
== June 21 ==
* 22:47 binasher: created a wb_entity_per_page view in labsdb wikidatawiki_p
* 21:24 mutante: deleting /home/daniel on fenari (confirmed not needed)
* 20:31 mutante: installing upgrades on bast1001
* 20:27 mutante: cleaning up /home/daniel/ (the other Daniel) ssh keys on fenari and letting puppet recreate them
* 19:31 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js  'Gerrit change I3db3cf47b / bug 49967'
* 19:30 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js  'Gerrit change I3db3cf47b / bug 49967'
* 18:53 gwicke: updated Parsoid to 21e1e7d
* 18:09 mutante: installing package upgrades on kaulen (bugzilla)
* 18:03 gwicke: updated Parsoid to 37cd852f
* 14:38 hashar: jenkins: updating all phpcs-HEAD jobs to rely on git-changed-in-head script {{gerrit|69863}}
* 10:38 hashar: upgrading packages on gallium
* 06:09 gwicke: updated Parsoid to 90515ab5
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 21 02:24:03 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 21 02:13:50 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 21 02:07:39 UTC 2013
* 00:05 gwicke: updated Parsoid to 393a3263
 
== June 20 ==
* 23:30 gwicke: updated Parsoid to 6dadde3
* 22:58 gwicke: updated Parsoid to 6240c19
* 22:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour  'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
* 22:02 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour  'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
* 21:01 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'I24bbe0d4e: Update EventLogging config to utilize API'
* 20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted  'GettingStarted to bf05656766'
* 20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour  'GuidedTour to a6fdf3c910'
* 20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted  'GettingStarted to bf05656766'
* 20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour  'GuidedTour to a6fdf3c910'
* 20:28 Krinkle: Gracefully reloading Zuul to deploy Ie7513f356cd8
* 20:24 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 20:24 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/EventLogging  'EventLogging to 2351b4ccbb'
* 20:21 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/EventLogging  'EventLogging to 2351b4ccbb'
* 20:20 Krinkle: Gracefully reloading Zuul to deploy I8c0ac58d9498979b
* 20:11 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/CoreEvents  'CoreEvents to d291b64248'
* 20:11 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents  'CoreEvents to d291b64248'
* 20:07 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php  'I83fcaa5ec: enable NavigationTiming on beta cluster'
* 19:49 logmsgbot: maxsem synchronized php-1.22wmf7/extensions/MobileFrontend
* 19:47 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
* 19:23 logmsgbot: reedy synchronized wmf-config/
* 18:06 gwicke: updated Parsoid to b206b54
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.22wmf7
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, test2wiki and testwikidatawiki to 1.22wmf8
* 17:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 17:57:01 UTC 2013
* 17:55 logmsgbot: reedy synchronized wmf-config/
* 17:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 20 17:51:50 UTC 2013
* 17:41 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 17:40:46 UTC 2013
* 17:41 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 17:39:59 UTC 2013
* 17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/  'Sync Campaigns and CentralAuth'
* 17:34 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
* 17:10 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
* 16:56 logmsgbot: reedy synchronized docroot and w
* 16:54 logmsgbot: reedy synchronized php-1.22wmf8/  'initial sync'
* 16:09 Jeff_Green: nuked neon puppet.log b/c /var/log was 99% full
* 15:55 gwicke: updated Parsoid to bf8d3df
* 14:17 hashar: jenkins updating all mediawiki extensions unit testing jobs ( mwext-.*-testextensions-master'
* 13:28 cmjohnson1: dns update
* 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 02:22:16 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 02:12:59 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 02:07:13 UTC 2013
* 00:13 gwicke: updated Parsoid to 45944a0
 
== June 19 ==
* 21:06 RobH: rebooting tridge due to updates
* 20:57 Ryan_Lane: updating OpenStackManager on virt0 to 3e92211313ed8b73ee44fb395230bafb14dc9bdc
* 20:34 ottomata: analytics1019 and analytics1020 are fully puppetized.  rebalancing them back into the hadoop cluster
* 18:42 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/  'ULS to master again'
* 18:35 gwicke: updated Parsoid to ddc47d8
* 18:35 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/  'ULS to master'
* 18:32 logmsgbot: asher synchronized wmf-config/db-pmtpa.php  'pull db64'
* 18:29 mutante: sync-apache and graceful for votewiki rewrites (change 67203)
* 18:15 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/  'ULS to master'
* 15:02 cmjohnson1: authdns update
* 14:58 cmjohnson1: powering down and relocating searchidx1001
* 14:55 Jeff_Green: update & reboot boron
* 14:36 cmjohnson1: powering down mw1171 to replace hard drive and reinstalling
* 13:52 cmjohnson1: powering down carbon to replace bad disk
* 10:01 logmsgbot: hashar synchronized wmf-config  '{{gerrit|69466}} beta: $wmg for AbuseFilterCentralDB'
* 10:00 logmsgbot: hashar synchronized wmf-config/CommonSettings.php  '{{gerrit|69466}} beta: $wmg for AbuseFilterCentralDB'
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 19 02:18:52 UTC 2013
* 02:16 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js  'touch'
* 02:16 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js  'touch'
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 19 02:14:07 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 19 02:07:42 UTC 2013
* 02:05 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 01:50 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 00:57 gwicke: updated Parsoid to 0445108
 
== June 18 ==
* 22:52 logmsgbot: olivneh synchronized wmf-config/CommonSettings-labs.php  'Iaade2591b: Configure $wgEventLoggingBaseUri & $wgEventLoggingFile on beta cluster'
* 22:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php  'Ibeb07a4b9: Re-enable EventLogging on beta cluster'
* 22:30 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'I7fc1a5ad4: Ensure forward-compatibility by setting '
* 22:16 mutante: scheduled 24 hour downtime for eqiad search-pools to stop paging
* 22:04 mutante: rebooting iron to finish upgrade
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
* 21:55 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
* 21:50 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:39 mutante: apache-graceful-all for redirects
* 21:37 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:41 LeslieCarr: added temp logging filter to cr2-eqiad analytics4 filter
* 20:34 mutante: upgrading iron to precise
* 19:44 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'Turn Disambiguator on on en.wiki'
* 19:17 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 18:58 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:50 mutante: rebooting iron
* 18:44 mutante: installing kernel and package upgrades on iron
* 18:33 mwalker: updating payments cluster (listeners) from aa0039104e71417845ec14a5ad666c2de464cfab to 8ca174296d570b6fda365cab37b2222d81c47c3d for adyen USD capture bug
* 18:12 greg-g: it's better than bad, it's good!
* 18:09 RobH: log log log log
* 18:09 mutante: LVS monitoring pages caused by neon recovering
* 18:08 RobH: ignore eqiad based search errors for now, as tampa is hosting all search traffic while hardware is repaired
* 18:07 RobH: search is based in tampa
* 17:40 cmjohnson1: dns update
* 17:28 cmjohnson1: wtp1008 is going down for main board replacement
* 17:09 mutante: reloading squid frontends in squid_esams_text and eqiad.text
* 17:07 mutante: reloading squid frontends in pmtpa.text
* 14:01 cmjohnson1: powering down search1001-1024 to relocate to row c5
* 08:26 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'ULS phase 2'
* 08:16 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:14 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/  'ULS to master'
* 07:19 ori-l: Force-confirmed e-mail for 160 accounts. full list in fluorine:/home/olivneh/users-forced-confirm-email-18-Jun-2013.list
* 07:12 ori-l: Marking e-mail addresses of accounts created in last 24h with no password as "confirmed" on all wikis on which VE is enabled.
* 04:13 ori-l: Manually marked the e-mail address of 54 enwiki users with e-mail address configured but no password (due to bug 49727) as confirmed.
* 03:41 logmsgbot: mflaschen synchronized php-1.22wmf7/extensions/VisualEditor/VisualEditor.hooks.php  'Hotfix to fix account creation'
* 03:15 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.hooks.php  'Emergency hotfix to fix account creation'
* 03:10 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.php  'Emergency hotfix to fix account creation'
* 03:10 superm401: Started hotfix for account creation on wmf6.
* 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 18 02:25:42 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 18 02:13:24 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 18 02:07:31 UTC 2013
* 01:04 Krinkle: Gracefully reloading Zuul to deploy {{gerrit|I5695a3b988e9ec3138f}}
* 00:51 mutante: reloading squid config on pmtpa.txt
* 00:48 mutante: reloading squid config on esams text squids
* 00:40 mwalker: updating the payments cluster (listeners) from 5f67ceddd9d495cffd22f6b303443aba58041d47 to aa0039104e71417845ec14a5ad666c2de464cfab for additional smashpig content
* 00:11 mutante: touch /etc/exim4/defer_domains on streber
 
== June 17 ==
* 23:52 mutante: restarting exim on streber
* 23:52 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 23:40 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 23:36 ^demon: restarting gerrit
* 23:31 RoanKattouw: Running scap to update VisualEditor
* 23:11 mutante: install package upgrades on streber
* 23:06 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'Change I95a8785c7 / Bug 49312: Add 'Programs' NS to MetaWiki'
* 23:05 Krinkle: Gracefully reloading Zuul to deploy {{gerrit|I773e84e6f75512}}
* 22:22 gwicke: updated Parsoid to b2a7b101e
* 22:20 mutante: built package and upgrading to php-luasandbox 1.7-1
* 19:38 Jeff_Green: dist-upgrade and reboot indium
* 19:29 gwicke: updated Parsoid to d1ae92720c4e
* 18:50 hashar: upgrading packages on gallium
* 18:41 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'bookkeeping changes for pmtpa dbs'
* 18:16 logmsgbot: aaron synchronized php-1.22wmf7/includes/db/DatabaseMysql.php  'ac676a2d83af283701dee111a28a52ea1730d657'
* 18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non 'pedia to 1.22wmf7
* 17:15 notpeter: rebooting es7 for lvm snapshot cleanup (again)
* 17:05 apergos: neon should be back in business
* 16:41 apergos: neon  out of inodes again, clearing out /var/spool/snmpttp, will return to service shortly
* 13:11 apergos: need to find out why sfo-intranet.corp.wikimedia.org can't be resolved in order to have opendj back and restore mail service
* 12:40 apergos: restarting opendj on sanger
* 10:50 hashar: manually rebased mediawiki-config on fenari 7d10d3f --> 1d2093e
* 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 17 02:12:33 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 17 02:06:18 UTC 2013
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 17 02:03:45 UTC 2013
 
== June 16 ==
* 23:36 paravoid: restarting pybal on lvs1003/lvs1006 for ms-fe4.eqiad.wmnet ip change
* 02:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 16 02:13:03 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 16 02:06:29 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun 16 02:03:54 UTC 2013
 
== June 15 ==
* 08:38 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'Mobile host for Commons'
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 15 02:19:50 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 15 02:13:57 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun 15 02:08:17 UTC 2013
* 02:06 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 00:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 00:05 mutante: reloading squid config on cp1001-cp1020 (text eqiad) for RT-5296
 
== June 14 ==
* 23:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 22:27 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 22:26 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 22:06 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 21:57 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 21:56 logmsgbot: py synchronized wmf-config/db-eqiad.php  'repooling es1007 and es1010'
* 21:56 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 21:41 notpeter: reboot es1010 to deal with messed up lvm snapshot
* 21:36 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fixing UploadWizard in Chrome
* 21:33 notpeter: reboot es1007 to deal with messed up lvm snapshot
* 21:31 logmsgbot: py synchronized wmf-config/db-eqiad.php  'temp depooling es1007 and es1010 for reboots'
* 21:28 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fixing UploadWizard in Chrome
* 21:03 notpeter: rebooting es10 to deal with fucked up snapshot volume
* 19:33 mutante: deploying small changes to bugzilla login screen
* 19:30 notpeter: installin updates on es7 while I'm at it
* 19:28 notpeter: rebooting es7 to deal with messed up vlm snapshot
* 18:46 gwicke: updated Parsoid config and code back to latest after finding and fixing Varnish config bug
* 18:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Re-enable VisualEditor on the rest of the wikis'
* 18:01 logmsgbot: aaron synchronized php-1.22wmf7/maintenance/purgeDeletedFiles.php  '9e2ffededde6e7752a8bd64d2ae791af768213c0'
* 17:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Re-enable VE on mw.org'
* 15:44 cmjohnson1: powering down wtp1008 to swap cpu's
* 15:26 cmjohnson1: carbon going offline to run h/w test at Dell support request.
* 15:15 gwicke: rolled back Parsoid dependency update to investigate issues
* 15:14 cmjohnson1: working mw1041 updating bios
* 14:54 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php  'emergency disabling of Visual Editor, see bug 49577'
* 04:46 gwicke: updated Parsoid to 64921430b1
* 04:33 gwicke: updating Parsoid dependencies from config repository
* 02:20 cmjohnson1: dns update
* 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 14 02:17:44 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 14 02:11:02 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun 14 02:05:57 UTC 2013
* 01:25 paravoid: starting swift->ceph scripts on a screen on ms-fe1002
* 00:29 logmsgbot: catrope Finished syncing Wikimedia installation... : Fixup for VE deployment
* 00:20 logmsgbot: catrope Started syncing Wikimedia installation... : Fixup for VE deployment
* 00:10 RoanKattouw: Scapping to fix up VE deployment
 
== June 13 ==
* 23:59 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'I3728a4cd9'
* 23:42 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'Enabling Campaigns & CoreEvents (4/4)'
* 23:41 logmsgbot: olivneh synchronized wmf-config/extension-list  'Enabling Campaigns & CoreEvents (3/4)'
* 23:41 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'Enabling Campaigns & CoreEvents (1/4)'
* 23:40 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php  'Enabling Campaigns & CoreEvents (1/4)'
* 23:37 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents  'Adding CoreEvents extension to wmf7'
* 23:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 23:35:36 UTC 2013
* 23:31 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/Campaigns  'Adding Campaigns extension to wmf6'
* 23:31 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 23:30:06 UTC 2013
* 23:30 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/CoreEvents  'Adding CoreEvents extension to wmf6'
* 23:21 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 23:20:43 UTC 2013
* 23:15 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/resources/mw.LanguageUpWiz.js  'Opera UploadWizard fix'
* 23:14 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php  'Opera UploadWizard fix'
* 23:08 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.class.php  'Update LocalisationUpdate to master in wmf7'
* 23:07 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.php  'Update LocalisationUpdate to master in wmf7'
* 22:57 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js  'touch'
* 22:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Enable EventLogging for VE on enwiki'
* 22:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 22:49:02 UTC 2013
* 22:48 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable EventLogging for VE on enwiki'
* 22:43 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 22:42:42 UTC 2013
* 22:40 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
* 22:39 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:38:14 UTC 2013
* 22:34 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:33:09 UTC 2013
* 22:31 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
* 22:18 RoanKattouw: Fixing perms on php-1.22wmf7, not group-writable
* 22:10 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 22:08 LeslieCarr: restarted squid on cp1004 due to "Jun 13 21:40:34 cp1004 squid[1961]: commBind: Cannot bind socket FD 23 to *:0: (98) Address already in use"
* 22:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 21:55 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 21:15 logmsgbot: spage synchronized php-1.22wmf6/extensions/GuidedTour  'latest GuidedTour'
* 21:13 logmsgbot: spage synchronized php-1.22wmf6/extensions/EventLogging  'latest EventLogging'
* 20:51 LeslieCarr: mw1173 has bad memory, rt 5294
* 20:48 logmsgbot: spage synchronized php-1.22wmf7/extensions/GuidedTour  'latest GuidedTour'
* 20:48 LeslieCarr: repuppetizing mw1020 after a reinstall
* 20:41 Jeff_Green: authdns update
* 20:41 logmsgbot: spage synchronized php-1.22wmf7/extensions/EventLogging  'latest EventLogging'
* 20:19 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/resources/mw.LanguageUpWiz.js  'Ia09ace5dc: Fix UploadWizard's config variables (2/2)'
* 20:18 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php  'Ia09ace5dc: Fix UploadWizard's config variables'
* 20:16 logmsgbot: kaldari synchronized php-1.22wmf6/extensions/Thanks/modules/ext.thanks.thank.js  'fix IE Thanks bug'
* 20:16 logmsgbot: reedy synchronized php-1.22wmf5/cache/
* 20:07 gwicke: updated Parsoid to a33980cf
* 19:44 binasher: rebooted mc1009 as a firedrill, chaos monkey ahoy!
* 19:36 cmjohnson1: dns update
* 19:06 logmsgbot: kaldari synchronized php-1.22wmf7/extensions/Thanks/modules/ext.thanks.thank.js  'fix IE Thanks bug'
* 18:59 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving last of search traffic to pmtpa'
* 18:50 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving more search traffic to pmtpa'
* 18:33 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving search traffic to pmtpa for en, de, fr, and ja'
* 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki and mediawikiwiki to 1.22wmf7
* 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikipedia to 1.22wmf6
* 16:29 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
* 16:12 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
* 16:04 logmsgbot: reedy synchronized w
* 16:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:46 logmsgbot: reedy synchronized docroot
* 15:45 logmsgbot: reedy synchronized php-1.22wmf7  'Initial file sync'
* 07:33 apergos: the last of the c2100s in the swift pool has been replaced (and new rings pushed), finally. now we "only" have the H310 controllers to replace...
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 02:28:27 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun 13 02:14:58 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 02:08:12 UTC 2013
* 00:29 logmsgbot: asher synchronized wmf-config/mc-eqiad.php  'disable persistent conns to twmprxy'
* 00:24 paravoid: ceph readding osd.81, osd.125, osd.137, osd.141 after disk replacement (#5202, #5228, #5248, #5263)
* 00:22 logmsgbot: asher synchronized wmf-config/mc-eqiad.php  'trying twemproxy on all eqiad hosts'
* 00:14 logmsgbot: asher synchronized wmf-config/mc-eqiad.php  'large server_failure_limit, small retry_timeout for twemproxy test hosts'
 
== June 12 ==
* 23:57 logmsgbot: asher synchronized wmf-config  'revert twemproxy'
* 23:48 logmsgbot: asher restarted twemproxy on all servers
* 23:46 logmsgbot: asher synchronized wmf-config/twemproxy-pmtpa.yaml  'pull 10.0.12.15'
* 23:42 logmsgbot: aaron cleared profiling data
* 23:35 logmsgbot: asher synchronized wmf-config  'use twemproxy for memcached access from all hosts'
* 23:33 logmsgbot: tstarling cleared profiling data
* 23:10 paravoid: apt: updating libmemcached/php5-memcached (#5268)
* 23:00 logmsgbot: asher synchronized wmf-config/mc-eqiad.php  'set (server_failure_limit, retry_timeout) = 0 for twemproxy test hosts'
* 22:26 logmsgbot: aaron synchronized php-1.22wmf6/includes/objectcache/MemcachedPeclBagOStuff.php  '7f091934ceae0280b44292e5ce426b24c5a579a2'
* 22:24 logmsgbot: aaron synchronized php-1.22wmf5/includes/objectcache/MemcachedPeclBagOStuff.php  'f6287b2e2faba20aea7379dae84ee7c5dd84983f'
* 21:35 mutante: deploying new favicon on wikitech
* 21:22 LeslieCarr: restarting caesium
* 20:56 mutante: PXE booting analytics1020
* 18:51 logmsgbot: asher synchronized wmf-config/mc-eqiad.php  'use twemproxy on mw113* (10 api apaches)'
* 17:29 cmjohnson1: powering down wtp1008 for h/w problem/testing
* 17:23 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
* 17:21 yurik: mw1020 complained during depl about fingerprint 81:2e:56:16:83:fd:b1:65:08:7b:ad:f8:66:98:19:c9.
* 17:20 ori-l: mw1020: Warning: Remote Host Identification Has Changed
* 17:18 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess/
* 17:06 cmjohnson1: removing/replacing disk slot 9 ms-be1007
* 16:36 cmjohnson1: removing/replacing failed disks in slot5 and slot9 from ms-be1012
* 16:00 cmjohnson1: powering down carbon for h/w testing/reports
* 15:19 cmjohnson1: mw1041 powering down for h/w issue
* 15:10 gwicke: updated Parsoid to 058da0822
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 12 02:27:19 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed Jun 12 02:13:49 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 12 02:07:39 UTC 2013
* 00:21 Ryan_Lane: upgraded mediawiki on wikitech-static
* 00:11 Tim: installing the experimental libmemcached packages on srv193
 
== June 11 ==
* 22:58 cmjohnson1: wtp1008 rebooting to check bios setting
* 21:26 cmjohnson1: mw1171 powering down via racadm
* 21:21 logmsgbot: maxsem synchronized php-1.22wmf5/extensions/MobileFrontend/
* 21:19 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:11 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:33 mutante: deploying bugzilla frontpage css/html changes (bug 22170)
* 20:28 cmjohnson1: powering off asw-c8-eqiad and removing from rack
* 20:18 LeslieCarr: halting asw-c8-eqiad
* 20:17 LeslieCarr: disabling asw-c8-eqiad
* 19:46 gwicke: updated Parsoid to ab8d15596f
* 19:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:23 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:56 Aaron|home: Cleared rows for queued transcodes on commonswiki
* 18:41 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Enable Echo interaction schema and cohort study'
* 18:40 cmjohnson1: depooling and powering down wtp1001-1002 and wtp1005-1tp1008
* 18:40 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Enable Echo interaction schema and cohort study'
* 18:15 Aaron|home: Parsoid jobs pipeline fixed with 866e66104fa6d285911da1888b9f9dcece2f53a4, pipeline should be working now
* 17:21 logmsgbot: asher restarted twemproxy on all servers
* 17:20 logmsgbot: asher synchronized wmf-config
* 16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:03 logmsgbot: reedy synchronized docroot/bits/favicon/testwikidata.ico
* 15:56 logmsgbot: asher restarted twemproxy on all servers
* 15:22 hashar: cleaned up permissions on operations/debs/buck , that is now a regular ops repo (ops are owner + force push )
* 15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/DataValues/
* 15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
* 15:12 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
* 15:09 mark: Inserted varnish 3.0.3plus~rc1-wm11 packages into the precise-wikimedia APT repository
* 14:41 hashar: granted myself force push rights on operations/debs/buck to push in git://github.com/facebook/buck.git@master to 'upstream' branch
* 08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'Enable ULS'
* 08:14 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php  'Disable narayam and WebFonts'
* 08:10 Nikerabbit: Starting ULS deployment phase 1
* 07:21 Jamesofur: updating Board Election translations on cluster
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 11 02:30:24 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun 11 02:16:57 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 11 02:09:34 UTC 2013
* 01:31 paravoid: restarting all jobrunners per Aaron's request
* 01:07 logmsgbot: aaron synchronized php-1.22wmf6/includes/db/LoadMonitor.php  '2e62b818a01d714b8464689eec2394a993209a70'
* 00:56 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Wikibase
* 00:56 logmsgbot: catrope synchronized php-1.22wmf6/extensions/Parsoid
* 00:55 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid
* 00:46 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Added parsoid jobs to $wgJobTypesExcludedFromDefaultQueue'
* 00:39 RoanKattouw: mw1020: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
* 00:35 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Point Parsoid updates to LVS IP'
* 00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php  'Re-enabled TTM server message jobs.'
 
== June 10 ==
* 23:33 paravoid: disabling asw-c-eqiad ge-4/0/7, enabling ge-4/0/6, moving description ms-fe1004 from one to the other and fixing VLANs
* 22:50 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
* 22:48 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
* 22:10 mutante: restarting lucene on search prefix hosts
* 21:55 gwicke: updated Parsoid to 6e40256821
* 21:08 mutante: dpkg --configure -a on analytics1020 to fix interrrupted dpkg
* 21:02 mutante: puppet re-signing analytics1020
* 20:48 mutante: restarting lsearchd on all pool4 search nodes
* 19:39 cmjohnson1: dns update
* 19:36 RoanKattouw: Restarted Apache on gallium
* 19:34 logmsgbot: olivneh synchronized php-1.22wmf6/includes/content  'Reverting change Ibfb2cbefe49398'
* 19:34 mutante: reinstalling analytics1020
* 19:26 mutante: powercycling analytics1020
* 19:24 mutante: deleting and recreating /var/spool/snmptt on neon
* 19:08 cmjohnson1: powering down ms-fe1004 to relocate to different row
* 18:58 RoanKattouw: Restarted Varnish on cerium and titanium
* 18:47 mutante: deleting files from /var/spool/snmptt to fix neon again
* 18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.22wmf6
* 18:29 paravoid: depooling mw1171, broken hardware #5231
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf6
* 18:26 Reedy: mw1171 is running with readonly file system
* 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf6
* 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, private and fishbowl to 1.22wmf6
* 17:48 mutante: rebooting analytics1020
* 17:46 mutante: killed and restared lucene on search indexers after imports, starting incremental updater
* 17:38 mutante: restarting search indexers to let them know about new wikis, running import-db scripts for elwikivoyage, vecwiktionary, testwikidatawiki
* 17:22 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php  'Undo live hack'
* 17:15 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php  'Live hack for debugging'
* 17:14 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php  'Live hack for debugging'
* 16:29 ^demon: restarting apache on gerrit box
* 08:49 apergos: restarted gerrit again, poor thing
* 08:26 logmsgbot: nikerabbit synchronized php-1.22wmf5/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:10 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/  'ULS to master'
* 07:34 Jamesofur: updated Board Election translations on cluster
* 02:36 gwicke: updated Parsoid to 956117df0e
* 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 10 02:06:49 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 10 02:02:14 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun 10 02:01:29 UTC 2013
* 00:12 Krinkle: Gerrit has once again closed the Zuul socket, ssh event stream down
 
== June 9 ==
* 22:33 Nemo_bis: gitblit fixed as of 1h ago: ^demon> Yeah, I saw the e-mails.
* 18:03 gwicke: deploy Parsoid 836ae1e, hopefully fixing the hanging worker issue
* 16:11 gwicke: restarting Parsoid
* 05:20 ^demon: restarted gerrit, again
* 03:29 Krinkle: git.wikimedia.org is 503 Service Temporarily Unavailable
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun  9 02:07:43 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun  9 02:02:15 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun  9 02:01:31 UTC 2013
* 01:59 Krinkle: Zuul is halted on a broken Gerrit event stream again.
 
== June 8 ==
* 23:22 RoanKattouw: Restarted Parsoid on a few machines where it was stuck
* 23:02 Jamesofur: synced Board Election translations on cluster
* 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Actually enable EducationProgram on the swedish wikipedia'
* 10:40 Nemo_bis: git.wikimedia.org 503: Service Temporarily Unavailable
* 07:54 Jamesofur: synced Board Election translations on cluster
* 02:25 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 02:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun  8 02:19:43 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun  8 02:08:16 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun  8 02:05:56 UTC 2013
* 01:37 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 01:37 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 01:00 RoanKattouw: Restarting Parsoid on all backends
* 00:54 RoanKattouw: Restarted Apache on gallium AGAIN. Patch for the underlying issue is https://gerrit.wikimedia.org/r/#/c/67551/
 
== June 7 ==
* 23:09 RoanKattouw: Restarted Apache on gallium agai
* 22:47 gwicke: update Parsoid to 9ccf7a3365
* 22:46 RoanKattouw: and on cerium and titanium too
* 22:46 RoanKattouw: Rerunning puppet on nickel for parsoidcache change
* 22:40 LeslieCarr: restart snmptrapd and smptt on neon
* 22:24 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 22:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 21:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/EducationProgram/
* 21:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/EducationProgram/
* 21:49 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Enable REL1_21 for extdist'
* 21:45 cmjohnson1: auth-dns update
* 21:21 mutante: installing package upgrades on neon now that it has inodes again
* 21:08 cmjohnson1: auth-dns update removing tampa parsoid dns entries
* 21:06 mutante: neon ran out of inodes, culprit /var/spool/snmptt has >300k files, deleting -mtime +2 in batches
* 20:49 RoanKattouw: Actually restarting pybal on lvs4 now
* 20:48 RoanKattouw: Running puppet on lvs4
* 20:47 RoanKattouw: Restarting pybal on lvs4
* 20:32 RoanKattouw: Restarting pybal on lvs3
* 19:50 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Tweaked redis queue timeout and enabled gzip'
* 19:43 RoanKattouw: Restarting Parsoid manually because git-deploy is not cooperating
* 19:29 mutante: installing package upgrades on neon (Icinga, nagios-plugins)
* 19:28 mutante: installing package upgrades on bast1001
* 19:24 mutante: installing package upgrades on fenari
* 19:20 RoanKattouw: Restarted Apache on gallium (APC cache pollution)
* 19:13 cmjohnson1: shutting down wtp1001 and wtp1002 to relocate to c8-eqiad
* 19:06 RoanKattouw: Depooled wtp1008 for investigation, Chris saw some sort of error on it
* 19:04 RoanKattouw: Depooled wtp1001, wtp1002 so Chris can move them
* 18:52 cmjohnson1: restarting ns1
* 18:47 cmjohnson1: auth-dns update (removing/renaming db1013)
* 18:28 mutante: DNS update - removing singer
* 17:52 cmjohnson1: shutting down db1013 to move to frack and rename
* 17:45 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/eraseArchivedFile.php  '2aa62f6afbd48a3362b78788b81aed847e6e805f'
* 16:29 logmsgbot: reedy synchronized wmf-config/
* 16:22 logmsgbot: reedy synchronized wmf-config/
* 16:04 logmsgbot: reedy synchronized wmf-config/
* 15:59 Reedy: Created EducationProgram tables on hewiki
* 15:58 Reedy: Created EducationProgram tables on sewiki
* 15:56 Reedy: Creating wikilove tables on viwiki
* 14:17 hashar: Zuul was no more triggering Jenkins jobs {{bug|49294}} . Fixed by Chad who restarted Gerrit.
* 13:55 ^demon: restarting gerrit, queue is overloaded and it's killing the bridge to jenkins
* 13:44 ^demon: restarting zuul per https://bugzilla.wikimedia.org/show_bug.cgi?id=49294#c4
* 12:46 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 12:45 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun  7 02:27:05 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun  7 02:16:10 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri Jun  7 02:05:54 UTC 2013
* 00:48 logmsgbot: catrope synchronized php-1.22wmf6/extensions/VisualEditor  'Updating VisualEditor to master'
 
== June 6 ==
* 23:39 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 23:31 gwicke: updating Parsoid to 020c7b2
* 23:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Plumbing for $wmgVisualEditorExperimental'
* 23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Set $wmgVisualEditorExperimental = true on mediawiki.org'
* 23:20 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
* 23:17 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 23:15 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 23:08 logmsgbot: tstarling cleared profiling data
* 23:04 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/php/Parsoid.php  'fix hook call'
* 22:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'touch'
* 22:47 logmsgbot: catrope synchronized visualeditor.dblist  'Enabling VisualEditor on the remaining 270 Wikipedias'
* 22:44 logmsgbot: gwicke Finished syncing Wikimedia installation... :
* 22:36 logmsgbot: gwicke Started syncing Wikimedia installation... :
* 22:28 gwicke: deploying https://gerrit.wikimedia.org/r/#/c/67338/
* 22:22 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php  'enable SUL2 on testwikidatawiki'
* 22:14 RoanKattouw: WTF, puppet ran just fine on srv283
* 22:13 RoanKattouw: Running puppet on srv283 to figure out what broke
* 22:11 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
* 22:01 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
* 21:43 Krinkle: Zuul gave up on connecting to Gerrit. Restarting Zuul made it try again. Restarting Gerrit again made it connect and it seems back up.
* 21:40 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid  'Adding Parsoid to wmf5'
* 21:26 Krinkle: Restarting zuul on gallium
* 21:11 kaldari: completed updateSpecialPages.php on testwiki
* 21:10 kaldari: Running updateSpecialPages.php on testwiki to test new Disambigutor special pages
* 21:00 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess
* 20:58 RoanKattouw: Restarting zuul on gallium
* 20:57 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess
* 20:45 logmsgbot: reedy synchronized database lists files:
* 20:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Send all Parsoid requests to eqiad'
* 20:34 logmsgbot: reedy synchronized php-1.22wmf6/extensions/
* 19:42 logmsgbot: reedy synchronized php-1.22wmf6/extensions/WikimediaMaintenance/
* 19:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:35 logmsgbot: reedy synchronized wmf-config/interwiki.cdb  'Updating interwiki cache'
* 19:34 logmsgbot: reedy synchronized php-1.22wmf5/extensions/WikimediaMaintenance/
* 19:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php  'Set wgSecurePollScript'
* 19:01 logmsgbot: reedy synchronized w/auth-api.php
* 18:53 logmsgbot: reedy synchronized database lists files:
* 18:53 logmsgbot: reedy synchronized wmf-config/
* 18:44 logmsgbot: reedy synchronized database lists files:
* 18:39 logmsgbot: reedy synchronized wikidata.dblist
* 18:39 LeslieCarr: removed some extraneous packages from neon which had filled its root partition
* 18:29 Reedy: Created translate tables on testwikidatawiki
* 18:28 logmsgbot: reedy synchronized database lists files:
* 18:27 logmsgbot: reedy synchronized database lists files:
* 18:26 logmsgbot: reedy synchronized wmf-config/
* 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 18:20 logmsgbot: reedy synchronized database lists files:
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf6
* 17:25 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
* 17:19 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
* 17:09 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:58 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:50 logmsgbot: reedy synchronized w
* 16:49 logmsgbot: reedy synchronized docroot
* 16:47 logmsgbot: reedy synchronized w
* 16:46 logmsgbot: reedy synchronized docroot
* 16:45 logmsgbot: reedy synchronized php-1.22wmf6/
* 15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 13:44 ^demon: zuul seems to not be yelling about invalid json anymore
* 13:41 ^demon: rolling gerrit back to 2.6-rc0-144-gb1dadd2 (version before any updates). Too many rough edges we didn't catch in testing, zuul is not happy
* 12:53 ^demon: gerrit rollback to 2.7-rc1-420-g5d5c5c3, some of the new work wasn't quite ready for prime time
* 12:30 ^demon: gerrit's back, running 2.7-rc1-424-gef469ac
* 12:26 ^demon: bringing gerrit down for a few minutes to pick up some changes
* 07:28 apergos: wtp1008 rebooted itself a few minutes ago (hence the icinga whine), no idea why
* 07:22 apergos: and the rest of the ms-fes
* 07:18 apergos: restarted swift on most backends as well
* 07:15 apergos: swift init restart all on ms-fe1 and 3
* 05:16 Tim: fixed high enwiki master query load due to GuidedTour
* 05:15 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour
* 05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/GuidedTourHooks.php
* 05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/modules/tours/test.js
* 04:19 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php  'remove debug hack'
* 03:55 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php  'temp debug hack'
* 03:48 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php  'revert temp hack'
* 03:46 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php  'temp debug log for high master query load'
* 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun  6 02:15:33 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun  6 02:06:01 UTC 2013
* 02:06 Krinkle: Graceful reload of Zuul to deploy {{gerrit|I4e7978f8a2a770}}
* 00:56 mwalker: updating listeners from a9fdfd328a8d06d4efb5f4037c281cf4341e23ba to e5f30211ae89ba52a1234715a6d59a7a2d421169; updating payments from cf2d1639a9a8b2854d5302db2aba86199b4e4101 to bf6cbd63cb3cc6148f8abf35767c7dba4ce92b0e
* 00:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:10 Tim: deployed apache configuration for vote.wikimedia.org
* 00:09 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php  'Enabled message cache debug log'
 
== June 5 ==
* 23:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:57 logmsgbot: reedy synchronized wmf-config/
* 23:56 logmsgbot: reedy synchronized database lists files:
* 23:56 logmsgbot: reedy synchronized docroot
* 23:44 logmsgbot: spage Finished syncing Wikimedia installation... :
* 23:40 Tim: added vote.wikimedia.org to DNS
* 23:38 logmsgbot: spage Started syncing Wikimedia installation... :
* 23:29 logmsgbot: mlitn synchronized php-1.22wmf5/extensions/ArticleFeedback/  'Update ArticleFeedbackv5 to master'
* 23:13 mutante: enabling wikidata "entity" URLs like http://www.wikidata.org/entity/Q1
* 22:52 logmsgbot: maxsem synchronized wmf-config
* 22:21 Ryan_Lane: upgrading salt in labs and production
* 22:09 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:00 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:39 logmsgbot: dzahn synchronized docroot/testwikidata
* 21:23 RoanKattouw: Killed old process on wtp1004 that was hogging the port
* 21:22 mutante: shutting down mw1171
* 21:06 ^demon: killed gitweb, long live gitblit
* 20:27 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 20:20 mutante: attempting parsoid restart on wtp1004
* 20:17 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/resources/SecurePoll.js
* 20:05 mutante: enabling test.wikidata virtual host
* 19:56 logmsgbot: dzahn synchronized docroot/wikidata_test/
* 19:55 mutante: sync_dir docroot/wikidata_test/ on tin
* 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything over to 1.22wmf5
* 16:50 paravoid: upgrading Ceph to cuttlefish, ignore warnings for the next half/one hour
* 15:32 andrewbogott: upgraded adminbot on wikitech-static to 1.7.3.  Should be more likely to survive netsplits.
* 10:24 paravoid: starting icinga
* 03:58 apergos: restarted morebots
 
== June 4 ==
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun  4 02:36:22 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue Jun  4 02:16:08 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun  4 02:09:55 UTC 2013
 
== June 3 ==
* 23:08 mutante: DNS update - point contacts away from broken singer
* 22:58 logmsgbot: kaldari synchronized php-1.22wmf5/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayerSupport.js  'syncing mw.MediaWikiPlayerSupport.js for TimedMediaHandler bug'
* 22:26 binasher: replaced redis-server in precise-wikimedia with 2.6.13
* 21:18 binasher: upgrading php5-redis on all appservers to version bult from github master 2d0f29bdaf29b071aea29b8fb9ee4158c2b69d72
* 20:28 LeslieCarr: dist-upgrading vanadium
* 20:28 LeslieCarr: allowing salt out of analytics subnet
* 19:37 logmsgbot: aaron synchronized php-1.22wmf5/includes/diff/DifferenceEngine.php  '22a544be01809226c1069cff90cdc174df81b7e5'
* 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf5
* 19:16 logmsgbot: aaron synchronized php-1.22wmf5/includes/WebRequest.php  '7715ffc9e4216497edc696fa686103dcc9698e9d'
* 19:15 logmsgbot: aaron synchronized php-1.22wmf4/includes/WebRequest.php  '7f31f0029bdc4526b41ad09968e17e10838087a6'
* 17:56 binasher: dropped old rowiki db from s3, left over from the migration to s7 4+ years ago
* 17:01 cmjohnson1: ms-be1011 replacing disk at slot5
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun  3 02:07:36 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon Jun  3 02:02:19 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun  3 02:01:35 UTC 2013
 
== June 2 ==
* 23:01 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll
* 23:00 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun  2 02:07:28 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun Jun  2 02:02:14 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun  2 02:01:30 UTC 2013
 
== June 1 ==
* 15:56 paravoid: staggered reboot of ms-fe1001-4, standard maintenance
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun  1 02:07:32 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat Jun  1 02:02:25 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun  1 02:01:38 UTC 2013
 
== May 31 ==
* 21:34 hashar: Zuul / Jenkins had some delay most probably related to a lot of patchsets being sent at the same time. Zuul catching up right now.
* 21:15 binasher: completed echo event_page_id schema migrations
* 20:50 binasher: restarted gerrit
* 19:53 AaronSchulz: Running a terbium script to copy missing files to swift as well as ceph
* 19:24 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/copyFileBackend.php  '82352634594d0a67d3b9f6597c3bb2c0911f8028'
* 19:23 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/copyFileBackend.php  '3d0e3f8e4d09d4a2462f583c25dd284e6ee3e466'
* 14:42 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
* 14:41 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 14:34 Reedy: Created bv2013_edit tables on all wikis
* 14:30 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
* 14:29 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 12:04 apergos: uh... it would help if I looked on tin, the actual deploy host so nm. anyways texvc now working
* 12:01 apergos: ran scap-recompile to get texvc built for 1.22wmf5 and btw this had to be done from fenari, there's no dsh on terbium
* 06:57 apergos: ceph osds 50 and 132 on ms-fe1002 are logging 'slow requests', don't know how to restart specific osds in bobtail though
* 06:43 Aaron|home: Running a script on terbium to list out files that were updated (overwritten) but are not synced on all wikis (it found none on commons/enwiki)
* 05:26 Tim: on msfe1001-1004: disabled apache_status ganglia module
* 04:18 Tim: on nickel: restarting gmetad to see if that fixes ceph cluster reporting
* 04:02 logmsgbot: aaron synchronized wmf-config/filebackend.php  'Disabled ceph backends and switched to just swift.'
* 02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 31 02:31:40 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 31 02:11:03 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri May 31 02:06:06 UTC 2013
* 00:11 mwalker: Updating payments deploy from dce20cbeb501a9945d97b2e64c59dbb4dd0a9620 to cf2d1639a9a8b2854d5302db2aba86199b4e4101 for RTBT attempted fix
* 00:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
 
== May 30 ==
* 23:58 LeslieCarr: fixing packages/security upgrades on db45
* 23:56 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 23:55 LeslieCarr: mw1171 appears to have a bad hard drive
* 22:08 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for E3 and E2
* 21:55 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for E3 and E2
* 21:23 LeslieCarr: powercycling mw16
* 20:35 cmjohnson1: ms-be106 replacing disk slot 6
* 20:11 cmjohnson1: es1001 replacing bad disk (slot 11)
* 20:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:55 LeslieCarr: enabling routing-engine on asw-c1-eqiad
* 19:54 LeslieCarr: asw-c1-eqiad successfully replaced
* 19:48 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:21 LeslieCarr: powering on new asw-c-eqiad switch -- possible disruption to row c connectivity (low risk)
* 19:20 LeslieCarr: replacing asw-c1-eqiad with new switch
* 18:15 bblack: updated vhtcpd deb to 0.0.6-1 on brewster, puppet will upgrade the varnishes
* 16:19 andrewbogott_afk: updated OpenStackManager on virt0 to git head.
* 14:59 paravoid: restarting ms-fe4 proxy
* 14:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable SecurePoll on testwiki'
* 13:21 RoanKattouw: (it's not broken, never mind, I'm just incapable of reading things)
* 13:21 RoanKattouw: Synced afe912f7a335ae037b8d8b0ba9616dc55848a2f0 for VisualEditor updates (logmsgbot seems to be broken)
* 13:20 hashar: gallium: killed stalled Jenkins threads that were producing high I/O.
* 13:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:15 logmsgbot: catrope synchronized visualeditor.dblist  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:11 hashar: gallium: blackholed some web crawler robot.
* 12:01 Krinkle: Gracefully reloading Zuul to deploy 'Ica1137438a1b9137f01276b4dfe49a82227de518'
* 10:19 logmsgbot: mark synchronized wmf-config/squid.php  'Add cp1037-1040 as text Varnish'
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 30 02:37:00 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 30 02:14:49 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu May 30 02:08:18 UTC 2013
 
== May 29 ==
* 23:36 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'enwikispecies -> specieswiki'
* 23:26 AaronSchulz: Tried running some metawiki TTMServerMessageUpdateJob jobs again, still very slow for some jobs (though at least not hours)
* 23:20 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (2/2)'
* 23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (1/2)'
* 23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php  'Labs: consolidate 'wgUseVFormCreateAccount' & 'wgUseVFormUserLogin' into 'wmgUseVForm''
* 19:53 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
* 19:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf5
* 18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase
* 18:56 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikinews to 1.22wmf5
* 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf5
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf5
* 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special and wikimedia to 1.22wmf5 (not wikidata)
* 18:10 logmsgbot: reedy synchronized database lists files: private and wikivoyage to 1.22wmf5
* 17:47 cmjohnson1: mw1085 powering down to reseat DIMM
* 17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 17:18 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 16:18 cmjohnson1: replacing failed disk on dataset1001 slot9
* 15:02 bblack: deployed vhtcpd puppet stuff via sockpuppet...
* 14:06 cmjohnson1: ms-be4..fixing raid cfg
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 29 02:33:19 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed May 29 02:10:48 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 29 02:05:47 UTC 2013
 
== May 28 ==
* 21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:17 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 18:54 logmsgbot: mlitn synchronized php-1.22wmf4/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 18:53 sbernardin: taking down ms-be1 for server replacement
* 18:23 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
* 18:14 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
* 17:28 sbernardin: taking down ms-be4 for server replacement
* 16:46 cmjohnson1: ms-be4 going down for h/w replacement
* 14:37 RoanKattouw: Gracefulled Apache on gallium, looks like there might have been APC corruption
* 14:23 notpeter: reimaging db1033 for testing
* 07:02 apergos: search23 apparently rebooted itself, nothing of use in the logs nor in atop
* 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 28 02:17:19 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue May 28 02:11:34 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 28 02:01:26 UTC 2013
 
== May 27 ==
* 20:30 logmsgbot: reedy synchronized php-1.22wmf5/extensions/UniversalLanguageSelector  'Revert back to 1.22wmf4 version of ULS'
* 20:19 apergos: restarted morebots
*18:41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
*18:39 reedy Finished syncing Wikimedia installation... : Take 3
*18:24 reedy Started syncing Wikimedia installation... : Take 3
*18:16 reedy Started syncing Wikimedia installation... : Take 2
*18:15 reedy synchronized php-1.22wmf5/cache/
*17:30 reedy synchronized wmf-config/squid.php  'Repush'
*17:15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
*17:13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
*17:12 reedy synchronized php-1.22wmf5/extensions/DataValues/
*17:11 reedy synchronized php-1.22wmf5/extensions/Diff/
*17:04 reedy synchronized docroot
*17:04 reedy synchronized w
*17:03 reedy synchronized php-1.22wmf5  'Initial file sync out'
*02.15 LocalisationUpdate ResourceLoader cache refresh completed at Mon May 27 02:15:05 UTC 2013
*02.03 LocalisationUpdate completed (1.22wmf4) at Mon May 27 02:03:50 UTC 2013
 
== May 26 ==
*18.41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
*18.39 reedy Finished syncing Wikimedia installation... : Take 3
*18.24 reedy Started syncing Wikimedia installation... : Take 3
*18.16 reedy Started syncing Wikimedia installation... : Take 2
*18.15 reedy synchronized php-1.22wmf5/cache/
*17.30 reedy synchronized wmf-config/squid.php  'Repush'
*17.15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
*17.13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
*17.12 reedy synchronized php-1.22wmf5/extensions/DataValues/
*17.11 reedy synchronized php-1.22wmf5/extensions/Diff/
*17.04 reedy synchronized docroot
*17.04 reedy synchronized w
*17.03 reedy synchronized php-1.22wmf5  'Initial file sync out'
* 15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:32 RobH: authdns-update for netmon1001
* 13:21 logmsgbot: mark synchronized wmf-config/squid.php  'Update cache list'
* 13:09 logmsgbot: mark synchronized wmf-config/squid.php  'Add amssq47-62'
* 12:44 Jeff_Green: authdns-update
* 12:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 12:31 logmsgbot: reedy synchronized wikidata.dblist
* 12:22 mutante: DNS update - add test.wikidata cname
* 12:17 ottomata: taking oxygen down for precise upgrade
* 09:43 LeslieCarr: upgrading some pmtpa mw* machines - may be a few alerts
* 09:18 logmsgbot: catrope synchronized wmf-config/throttle.php  'Raise account creation throttle for Amsterdam hackathon'
* 08:49 logmsgbot: tstarling cleared profiling data
* 06:11 Aaron|home: De-duplicated base refreshLinks2 jobs on enwiki, didn't do much though
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 26 02:27:21 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 26 02:06:36 UTC 2013
 
== May 25 ==
* 14:51 Jeff_Green: authdns-update
* 13:36 mutante: restarting pdns on ns0
* 13:11 LeslieCarr: rm -rf /home/wikipedia/common/php-1.21wmf12 on fenari
* 12:46 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
* 12:43 Jeff_Green: authdns-update
* 12:07 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
* 12:04 bblack: cp1029.eqiad : manually testing s/varnishhtcpd/vhtcpd/
* 10:08 Krinkle: Gracefully reloading Zuul to deploy I8c95150a688973
* 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 25 02:17:03 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 25 02:04:49 UTC 2013
 
== May 24 ==
* 23:38 logmsgbot: aaron synchronized php-1.22wmf4/maintenance  '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
* 23:37 logmsgbot: aaron synchronized php-1.22wmf4/includes/profiler/Profiler.php  '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
* 23:03 logmsgbot: reedy synchronized wmf-config/trusted-xff.cdb
* 20:39 logmsgbot: aaron synchronized php-1.22wmf4/includes/job/jobs/RefreshLinksJob.php  '88ffcabd2b0039d18b805bd1e5d8d350c26588e1'
* 14:05 mutante: dzahn graceful'ed apaches and api servers , add redirect for be-x-old.wikisource
* 13:56 RobH: tmh1 back online, tmh2 rebooting for kernel updates
* 13:43 RobH: tmh1 rebooting for kernel updates
* 13:01 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php  'testwiki captcha'
* 12:47 logmsgbot: reedy synchronized wmf-config/
* 10:42 Jeff_Green: authdns-update again...
* 10:28 hashar: jenkins: updating all phplint jobs to get rid of the old ant building script
* 10:22 Jeff_Green: authdns-update to add fundraisingdb service aliases
* 10:21 mutante: installing upgrades on fenari
* 10:07 RobH: tridge upgrading, will require reboot
* 08:29 K4-713: updated payments cluster to dce20cbeb501a
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 24 02:28:20 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 24 02:07:17 UTC 2013
 
== May 23 ==
* 21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Enable echo email bundling on enwiki'
* 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted  'E3 latest GettingStarted'
* 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour  'E3 latest GuidedTour'
* 19:17 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo/modules  'syncing Echo modules for css updates'
* 18:54 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
* 18:48 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
* 18:31 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Add new eventlogging schema: EchoPrefUpdate'
* 12:19 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 23 02:25:13 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 23 02:06:32 UTC 2013
 
== May 22 ==
* 23:40 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/runJobs.php  '2188c14239b74d202b2109933612259193f6fa41'
* 22:44 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js  'touch, attempt to fix VisualEditor cache snafu on dewiki'
* 22:33 paravoid: removing users sara & mwang from ldap/ops
* 19:12 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 19:05 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 18:51 apergos: restarted gerrit
* 18:42 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Swithced remaining wikis to 1.22wmf4
* 18:26 logmsgbot: kaldari synchronized php-1.22wmf4/includes/Preferences.php  'syncing preferences.php for bug 48693'
* 18:02 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 18:00 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 17:46 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 07:50 apergos: restarted swift proxy on ms-fe1
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 22 02:35:44 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 22 02:14:48 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 22 02:08:27 UTC 2013
* 00:21 Ryan_Lane: deployed change 64875 to virt0
 
== May 21 ==
* 21:11 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
* 21:04 logmsgbot: awjrichards Started syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
* 19:27 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:22 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:31 mwalker: (previous log referring to fundraising jenkins!) now upgrading fundraising jenkins for auth plugins
* 18:31 mwalker: restarted jenkins for rekeying of jobs dir; then updated non auth plugs
* 18:17 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Disable AFTv5 talk page links'
* 18:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Disable AFTv5 talk page links'
* 17:49 cmjohnson1: powering down db26 for last time
* 17:45 Jeff_Green: rebooting several fundraising hosts for kernel upgrades
* 17:14 cmjohnson1: replacing disk10 (L11) on ms-be1009
* 14:38 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php  'Disable $wgCentralAuthLoginWiki for testwiki'
* 13:59 logmsgbot: anomie synchronized wmf-config/CommonSettings.php  'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
* 13:59 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php  'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
* 13:14 logmsgbot: anomie synchronized php-1.22wmf4/extensions/CentralAuth  'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
* 13:13 logmsgbot: anomie synchronized php-1.22wmf3/extensions/CentralAuth  'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
* 09:41 hashar: made Jeroen De Dauw a maintainer of https://packagist.org/packages/mediawiki/data-values
* 08:10 hashar: zuul: running git gc on all Zuul git repositories:  <tt>find /var/lib/zuul/git -maxdepth 4 -type d -name .git -print -exec git --git-dir={} gc \;</tt>
* 08:02 hashar: Running git gc on Zuul git repositories (gallium: /var/lib/zuul/git/operations/puppet )
* 07:18 LeslieCarr: allowed labs access to the new labsdb hosts
* 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 21 02:26:30 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 21 02:09:52 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 21 02:06:41 UTC 2013
* 01:31 cmjohnson1: cycling titanium to check bios settings
 
== May 20 ==
* 23:59 mutante: dist-upgrading celsus
* 23:36 logmsgbot: mflaschen synchronized php-1.22wmf4/extensions/GettingStarted/  '-m Sync GettingStarted for E3 lightning deploy'
* 22:26 mutante: rebooting Tampa parsoid servers, one by one
* 22:20 mutante: dist-upgrading, rebooting kuo
* 22:11 mutante: dist-upgrading, rebooting arsenic
* 22:04 mutante: dist-upgrading, rebooting strontium
* 21:48 mutante: dist-upgrading, rebooting palladium
* 21:31 mutante: dist-upgrading, rebooting niobium
* 20:58 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js  'touch, fixup cache snafu'
* 20:51 Ryan_Lane: temporarily disabling nova-api ratelimiting
* 20:46 akosiaris: added cloudera debian packages hadoop, zookeeper, bigtop-jsvc, bitop-utils to apt.wikimedia.org
* 20:07 Ryan_Lane: rebooting all pmtpa virt nodes
* 20:07 Ryan_Lane: shutdown all labs instances
* 19:12 mutante: dist-upgrading cp1(sq51)
* 19:09 logmsgbot: csteipp synchronized php-1.22wmf4/includes/  'Security fix'
* 19:08 logmsgbot: csteipp synchronized php-1.22wmf3/includes  'Security fix'
* 19:04 logmsgbot: reedy synchronized w/
* 19:01 logmsgbot: reedy synchronized wmf-config/wgConf.php
* 19:00 logmsgbot: reedy synchronized wmf-config/wgConfVHosts.php
* 18:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable translate on commonswiki'
* 18:46 Reedy: Created translate tables on commonswiki
* 18:44 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js  'touched'
* 18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache
* 18:21 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache
* 18:15 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
* 18:14 logmsgbot: reedy synchronized php-1.22wmf4/extensions/ProofreadPage/
* 18:08 logmsgbot: reedy synchronized wmf-config/  'Enable TemplateData on all wikis'
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4
* 17:49 MaxSem: Reindexing solr
* 17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044
* 17:35 mutante: kernel upgrade cp1043
* 17:00 mutante: resurrecting icinga-wm and nsca
* 16:59 mutante: ping logmsgbot
* 16:52 mutante: rebooting neon for upgrade
* 16:51 cmjohnson1: shutting down lanthanum to add ssd rt5074
* 15:49 ottomata: rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel
* 14:55 ottomata: rebooting emery to upgrade to 3.2.0-43-generic kernel
* 14:53 ottomata: upgraded all analytics* machines to linux 3.2.0-43-generic kernel
* 02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013
 
== May 19 ==
* 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 19 02:06:55 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 19 02:01:54 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 19 02:01:19 UTC 2013
 
== May 18 ==
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013
* 01:23 mutante: singer boot issue - affects service contacts.wm (but no others)
* 00:13 mutante: dist-upgrading yvon
* 00:09 mutante: dist-upgrading nitrogen
 
== May 17 ==
* 23:48 mutante: dist-upgrading singer
* 23:46 mutante: dist-upgrading nickel
* 23:45 RobH: all tampa based apaches have had kernel upgrades
* 23:40 mutante: dist-upgrading praseodymium
* 23:33 RobH: upgrading all srv*.pmtpa.wmnet via dist-upgrade in salt.
* 23:28 mutante: dist-upgrading gurvin
* 23:13 RobH: rebooting all pmtpa mw servers
* 23:13 mutante: dist-upgrading hydrogen, manutius
* 23:12 RobH: still doing pmtpa mw upgrades, ignore all icinga alarms for now
* 23:05 mutante: dist-upgrading chromium
* 23:02 RobH: using salt to dist-upgrade all the tampa apaches... nothing could go wrong....right?
* 22:58 akosiaris: upgrading and rebooting cp1001-1020 (3 at a time)
* 22:55 mutante: dist-upgrading capella ( IPv6 tunnel relay)
* 22:34 mutante: shutting down zinc
* 22:23 mutante: !!log doesn't log
* 22:21 Reedy: [23:18:29] <binasher> !!log running hotbackup of db71 to pre-labsdb for s1
* 22:14 mutante: rebooting spence for upgrades
* 22:10 logmsgbot: reedy synchronized w
* 22:08 logmsgbot: reedy synchronized docroot
* 22:08 mutante: rm -rf php-1.21wmf12 on tin per reedy
* 22:03 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 21:58 apergos: mw80 memtest failure dimm1, all other image scalers in pmtpa updated
* 21:53 mutante: dist-upgrading zhen (mobile vumi)
* 21:49 mutante: DNS update - kill storage1 and 2
* 21:34 akosiaris: about to reboot fenari
* 21:13 hashar: gallium Manually blackholed some web engine crawler (via ip route)
* 19:55 mutante: Wikimedia IRC server working again
* 19:46 Ryan_Lane: rebooted ssl1/2 ssl3001 ssl1001/1002
* 19:42 hashar: Jenkins restarted successfully.
* 19:40 Ryan_Lane: depooling ssl3001 ssl1/2 ssl1001/2
* 19:40 Ryan_Lane: repooling ssl1003/4 ssl3/4
* 19:31 Ryan_Lane: repooling ssl3002/3
* 19:30 hashar: restarted Jenkins
* 19:29 Ryan_Lane: rebooted ssl3/4 ssl3002/3 ssl1003/1004
* 19:29 Ryan_Lane: depooled ssl3/4 ssl3002/3 ssl1003/1004
* 19:19 RobH: blog back up, whew.
* 19:16 RobH: holmium (blog server) rebooting for kernel upgrade
* 19:12 cmjohnson1: shutting titanium down to add disks (take 2)
* 19:04 apergos: eqiad imagescalers done
* 19:04 mutante: rebooting gallium
* 19:03 mutante: wikimedia irc is back
* 19:00 hashar: gallium: stopping jenkins and zuul
* 18:56 RobH: cerium rebooting for kernel upgrade
* 18:56 Ryan_Lane: dist-upgrading all ssl boxes
* 18:55 mutante: dist-upgrading ekrem (IRC)
* 18:55 cmjohnson1: shutting down titanium for add'l disk
* 18:50 RobH: i hate you morebots.
* 18:50 RobH: caesium rebooting
* 18:47 mutante: dist-upgrading calcium
* 18:42 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php  'Removing france as a redirect country for fundraising (7649276ddb3f20678de265c37fe75e1d5e642956)'
* 18:31 hashar: Jenkins: created a Jenkins slave on gallium using jenkins-slave as user and /srv/ssd/jenkins-slave as working directory.
* 18:17 RobH: carbon rebooting for kernel update, tftp in eqiad will be offline for a few
* 17:50 apergos: updating and rebooting image scalers one by one
* 17:46 cmjohnson1: shutting down cerium to add add'l disk
* 17:39 apergos: rebooted snapshots 1001-4 (new kernel)
* 17:35 mutante: bugzilla is back
* 17:31 ^demon: rebooting formey (svn) in 2 minutes...why are you still using SVN?
* 17:30 mutante: rebooting kaulen
* 17:24 mutante: rebooting kaulen (Bugzilla) in 5 minutes, please save your bugs and expect 5 minutes downtime
* 16:49 ^demon: rebooting antimony
* 15:00 paravoid: setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend
* 13:31 hashar: Zuul: applying project templates {{gerrit|63674}}
* 12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 12:29 logmsgbot: reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php
* 05:06 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013
* 00:56 K4-713: Woohoo! Logs!
* 00:51 logmsgbot: reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php
 
== May 16 ==
* 23:48 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/AccountAudit/  'Updating AccountAudit to master. Adding aa_method'
* 23:45 logmsgbot: pgehres synchronized php-1.22wmf4/extensions/AccountAudit/  'Updating AccountAudit to master. Adding aa_method'
* 23:43 logmsgbot: reedy synchronized database lists files:
* 23:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:39 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 22:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:29 logmsgbot: mwalker synchronized php-1.22wmf4/extensions/CentralNotice/  'Updating CentralNotice to master'
* 22:28 logmsgbot: reedy synchronized wmf-config/
* 22:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:02 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 22:00 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
* 21:53 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
* 21:50 superm401: Scap failed due to wrong ssh configuration
* 21:49 Ryan_Lane: rebooting virt1005 and virt1007
* 21:49 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 deployment
* 21:48 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 deployment
* 21:47 Ryan_Lane: rebooting virt1000
* 21:40 superm401: Started scap for E3 deployment of GettingStarted and GuidedTour
* 21:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 21:29 binasher: running accountaudit_login aa_method schema migrations on all wikis
* 21:20 Ryan_Lane: rebooting virt0
* 20:56 Ryan_Lane: running dist-upgrade on virt cluster
* 20:51 logmsgbot: reedy synchronized wmf-config/
* 20:40 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
* 20:35 paravoid: restarting a ceph-osd osd.3 for testing
* 20:03 logmsgbot: reedy synchronized wmf-config/  'Updating interwiki cache'
* 19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:58 logmsgbot: reedy synchronized database lists files:
* 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:49 logmsgbot: reedy synchronized database lists files:
* 19:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:01 logmsgbot: reedy synchronized wmf-config/
* 18:56 logmsgbot: reedy synchronized wmf-config/
* 18:55 logmsgbot: reedy synchronized wmf-config/session.php
* 18:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'EP on dewikiversity'
* 18:41 mwalker|0_0: updating payments from fdf308c8fa92ca75adbb3154b8b31a9f7237e885 to head
* 18:34 gwicke: updated Parsoid to b0229051eb08c8
* 18:29 mutante: magnesium (racktables): upgrading kernel and rebooting
* 18:21 mutante: zirconium (planet): upgrading kernel and rebooting
* 18:17 paravoid: restarting a ceph-osd osd.0 for testing
* 18:02 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
* 17:51 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
* 17:27 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 17:13 logmsgbot: reedy synchronized php-1.22wmf4/extensions/
* 16:53 ottomata: rebooting stat1 for kernel upgrade
* 16:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pooling db1043'
* 16:18 gwicke: Parsoid update to current master: done
* 16:16 gwicke: Parsoid update to current master: start
* 15:57 Krinkle: Graceful reload of Zuul deploying Ic9194c079fa7d46
* 15:28 Krinkle: Graceful reload of Zuul deploying I7565eff83a7a128e
* 13:50 Krinkle: Graceful reload of Zuul deploying I0ea73ca1439c8aa6
* 07:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/  '{{gerrit|I9d5afff9c328}}'
* 07:47 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/GeoData/  '{{gerrit|I9d5afff9c328}}'
* 07:30 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 06:42 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 16 02:30:28 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 16 02:11:55 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 16 02:06:34 UTC 2013
* 00:37 mutante: puppetstoredconfigclean.rb ocg3.pmtpa.wmnet
* 00:14 mutante: puppetstoredconfigclean.rb ms2.pmtpa.wmnet
* 00:06 logmsgbot: reedy synchronized database lists files:
* 00:05 logmsgbot: reedy synchronized database lists files:
 
== May 15 ==
* 23:44 mutante: pushing wikimedia-task-appserver 2.9-1
* 23:32 logmsgbot: reedy synchronized database lists files:
* 23:30 logmsgbot: reedy synchronized database lists files:
* 22:30 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js  'touched'
* 22:01 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 21:54 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 20:57 Ryan_Lane: OpenStackManager is reenabled on virt0. login to wikitech is working again
* 20:56 Ryan_Lane: openstack in pmtpa now at the folsom release
* 20:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
* 20:51 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend/
* 20:15 Ryan_Lane: force running puppet on all virt nodes to bring them back up
* 20:14 Ryan_Lane: upgrading schema for keystone, then glance, then nova
* 20:13 Ryan_Lane: backing up keystone, nova, and glance databases
* 20:12 logmsgbot: aaron cleared profiling data
* 20:12 logmsgbot: aaron cleared profiling data
* 20:10 Ryan_Lane: upgrading glance, keystone and nova-scheduler on virt0
* 20:09 Ryan_Lane: upgrading nova services on compute nodes
* 20:07 logmsgbot: aaron synchronized php-1.22wmf4/includes/specials/SpecialActiveusers.php  '{{gerrit|df79a829457752d21b04e641fda0041e09d13a9f}}'
* 20:05 Ryan_Lane: disabling OpenStackManager
* 20:05 Ryan_Lane: stopping all openstack services. it won't be possible to log into wikitech during this process
* 19:44 paravoid: running refreshFileHeaders.php on terbium
* 19:25 logmsgbot: aaron cleared profiling data
* 19:19 Ryan_Lane: adding temporary ubuntu-cloud archive apt changes in a non-puppetized way on pmtpa virt nodes
* 19:18 Ryan_Lane: disabling puppet on pmtpa virt cluster
* 19:09 logmsgbot: reedy synchronized wmf-config/  'touch!'
* 19:09 logmsgbot: reedy synchronized php-1.22wmf4/resources/  'touch!'
* 18:42 Ryan_Lane: cherry-picked change 62100 on virt0 until permanent fix for jquery.chosen minimum width is added
* 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf4
* 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf3 due to broken UploadWizard
* 18:35 ^demon: restarting gerrit
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aaaand wikisource to 1.22wmf4
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf4
* 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikibooks to 1.22wmf4
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf4
* 18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, special, wikimedia and private to 1.22wmf4
* 18:01 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
* 17:58 RobH: authdns update, killin dataset1 entries
* 17:56 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 17:50 Ryan_Lane: deploying new project filter on virt0
* 17:44 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 16:57 RobH: authdns update, cleaning out old decom entries
* 15:35 Jeff_Green: rebooting boron
* 11:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 11:15 logmsgbot: reedy synchronized wmf-config/squid-labs.php
* 11:14 logmsgbot: reedy synchronized wmf-config/squid.php
* 11:13 logmsgbot: reedy synchronized wmf-config/squid-labs.php  'wmf-config/squid.php'
* 11:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 15 02:34:37 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 15 02:14:58 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 15 02:08:15 UTC 2013
* 00:49 mutante: graceful apache on stat1001
* 00:31 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/63806/'
* 00:29 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/63806/'
 
== May 14 ==
* 23:57 mwalker: lies -- went from d895730067e3325fd630d09196ee81eeba24b5ee back to head on civi because it was working -- the tester was just broken
* 23:42 mwalker: rolling back civi from deploy head to d895730067e3325fd630d09196ee81eeba24b5ee because thank you isn't working
* 23:38 logmsgbot: aaron synchronized wmf-config/StartProfiler.php
* 23:23 logmsgbot: maxsem synchronized php-1.22wmf3/resources/startup.js
* 22:38 RobH: authdns-update for a bunch of decom updates
* 22:32 mutante: DNS update - removing owa servers
* 22:23 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php
* 22:17 Ryan_Lane: finished upgrading wikitech
* 22:10 Reedy: Running sync-common on srv193
* 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:03 Ryan_Lane: upgrading mediawiki to 1.22wmf4 on wikitech
* 22:03 mwalker: upgrading fundraising civi (d895730067e3325fd630d09196ee81eeba24b5ee --> head)
* 21:53 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:30 RoanKattouw: Repooled wtp1004, added wtp1005-wtp1024 to the pool
* 21:01 sbernardin: changing HD in ms-be5
* 20:52 binasher: s4 now all mariadb, new master info = MASTER_LOG_FILE='db1059-bin.000012', MASTER_LOG_POS=473448023
* 20:45 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/63596/'
* 20:44 sbernardin: changing drives on ms-be10
* 20:39 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/Gadgets/  'https://gerrit.wikimedia.org/r/#/c/60954/'
* 20:33 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 20:32 mutante: apply umask 0002 for wikidev users fix on tin and terbium, class generic::wikidev-umask
* 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'setting s4 to writeable on new master'
* 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'setting s4 to read-only for a sec'
* 20:22 logmsgbot: maxsem synchronized docroot/bits/mobile/W.png
* 20:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 19:53 RoanKattouw: Running salt-call state.highstate on wtp1005-24
* 19:51 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php  'd58dc596a25607deaecf0aa1c3be877f834da533 '
* 19:47 RoanKattouw: Accepted salt keys for wtp1005-1024 on sockpuppet
* 19:22 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
* 19:12 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
* 18:52 mutante: gracefulling apaches with dsh
* 18:46 Ryan_Lane: running an apache-graceful-all
* 18:44 ottomata: deployed puppet change to make nginx send webrequest logs back to main multicast firehose.  NOTE: These logs are not sent to the udp2log instance on emery.
* 18:35 sbernardin: replacing bad drive in ms-be11
* 18:34 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo  'syncing Echo for 1.22wmf4'
* 18:31 logmsgbot: reedy synchronized wmf-config/
* 18:30 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php  '0cf6b27aa43ad2b65725081ee4d36ccb0f8a34c6'
* 18:28 logmsgbot: reedy synchronized wmf-config/
* 18:27 mutante: powercycling frozen mw1173
* 18:25 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo  'syncing Echo for 1.22wmf4'
* 18:12 logmsgbot: reedy synchronized wmf-config/
* 17:56 logmsgbot: reedy synchronized wmf-config/
* 17:53 logmsgbot: reedy synchronized wmf-config/
* 17:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php  'Enable upload of XML files on Incubator (for MediaWiki dumps)'
* 17:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Namespace configuration for ur.wikipedia'
* 17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:18 logmsgbot: reedy synchronized wmf-config/
* 15:14 Reedy: Created EducationProgram tables on dewikiversity
* 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'thwiki collation'
* 13:53 cmjohnson1: running dns update
* 13:53 hashar: Zuul: applying a patch to prevent it from fetching a change multiple time.
* 10:51 apergos: re-enabling puppet on ssl1001 for a bit of real traffic
* 10:42 apergos: repooled ssl1
* 09:58 apergos: puppet temp disabled on all ssl terminators except ssl1 in pmtpa
* 09:37 apergos: depooled ssl1 in pmtpa for testing
* 08:51 logmsgbot: tstarling synchronized php-1.22wmf4/LocalSettings.php  'remove testwiki special case'
* 08:51 logmsgbot: tstarling synchronized php-1.22wmf3/LocalSettings.php  'remove testwiki special case'
* 08:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 08:26 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 08:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:44 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:16 logmsgbot: tstarling synchronized w/MWVersion.php
* 07:15 logmsgbot: tstarling synchronized refresh-dblist
* 07:14 logmsgbot: tstarling synchronized multiversion
* 07:12 logmsgbot: tstarling synchronized docroot/noc/db.php
* 06:40 logmsgbot: tstarling synchronized README
* 04:34 Tim: moved scap to tin. Copied source from NFS to tin. Fixed up broken git submodule config using sed.
* 04:03 logmsgbot: tstarling synchronized cgi-bin
* 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 14 03:09:49 UTC 2013
* 02:31 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 14 02:31:15 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 14 02:12:10 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
* 01:13 logmsgbot: tstarling synchronized README
* 01:00 logmsgbot: tstarling synchronized README
* 00:20 akosiaris: added myself to root email alias at mchenry
* 00:17 akosiaris: installing package-upgrades on zirconium
 
== May 13 ==
* 23:19 logmsgbot: catrope synchronized php-1.22wmf4/extensions/VisualEditor/  'VE bug fix'
* 22:46 RobH: authdns-update for wikipedia.is
* 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 21:11 hashar: jenkins: updating all phpcs-HEAD jobs to make them ignore i18n.php files {{bug|48420}} / {{gerrit|63571}}
* 21:02 logmsgbot: reedy synchronized wmf-config/
* 20:34 logmsgbot: aaron cleared profiling data
* 20:33 cmjohnson1: morebots back
* 14:40 gallium: deleted an ip route blacklist
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013
 
== May 12 ==
* 14:59 Krinkle: Graceful reload of Zuul to deploy Ic22d7c6cb811fd7e3
* 14:31 jeremyb: imported missing !log entries from channel log
* 14:23 bblack: restarted adminbot on wikitech-static
* 14:22 bblack: fixed typo in rt.wikimedia.org vhost hostname, was breaking RT http stuff
* 14:19 bblack: morebots is broken, someone should fix that when they read this
* 09:32 logmsgbot: olivneh synchronized wmf-config/throttle.php  '(Bug 48301) Add throttle exception for Haifa University workshop'
* 03:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 12 03:03:03 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 12 02:13:17 UTC 2013
* 01:28 logmsgbot: aaron synchronized php-1.22wmf3/maintenance/copyFileBackend.php  '3e103e93f01c24d7e454aaa5db07418405d273c3'
* 01:22 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendMultiWrite.php  '308ebddc237e76a9602b20585386fc26876f0bee'
 
== May 11 ==
 
* 13:24 hashar: restarted Jenkins. Got hit by a memory bug again.
* 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 11 03:48:00 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 11 02:16:23 UTC 2013
 
== May 10 ==
* 23:20 mutante: DNS update - remove various old Tampa hosts
* 22:49 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/  'Small little update to maintenance script'
* 21:46 mutante: shutdown -h blondel .. bye
* 21:39 RobH: switch port disabled for blondel & bellin
* 21:27 RobH: decom of bellen/blondel : setting network ports to disabled
* 21:27 AaronSchulz: Running listing based copy-sync scripts in terbium for all originals+transcodes on all wikis for swift->ceph to fix any outdated/missing files
* 21:05 mutante: fixing some more varnishncsa monitoring
* 20:58 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendStore.php  '2717d876ba2507df26ac2c5ab746ad7895af1bbf'
* 20:31 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend
* 20:06 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS
* 19:57 logmsgbot: aaron synchronized php-1.22wmf3/maintenance
* 19:56 logmsgbot: aaron synchronized php-1.22wmf3/includes  '6c01414775bb9c5269b391bee7ffd1611f9ff74f & d1940caa1f4a11e7c7d8d2ff5b178da7de8e2a92'
* 18:50 mutante: killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa
* 18:23 mutante: powercycling srv291
* 18:17 RobH: authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server
* 17:42 RobH: authdnsupdate - removing racktables2 entry
* 17:19 hashar: jenkins: cleared all occurrences of an old plugin in build histories. {{bug|48025}}.  Jenkins should be fine now. Will monitor tonight.
* 13:20 mark: Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access
* 12:28 mark: Removed knsq23-knsq30 from the text squid config
* 12:25 hashar: jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 )
* 12:09 mark: Depooled knsq23-knsq30 text squid frontends
* 11:52 mark: Moved amslvs1 and 2 from csw1-esams to csw2-esams
* 11:52 mark: Started PyBal on amslvs1
* 11:51 mark: Stopped PyBal on amslvs1
* 11:49 mark: Reenabled PyBal on amslvs2
* 11:44 hashar: restarting jenkins
* 11:44 mark: Disabled Puppet and stopped PyBal on amslvs2
* 11:27 mark: Moved ssl3001-3004 from csw1-esams to csw2-esams
* 10:09 hashar: jenkins: upgrading plugins and restarting.
* 09:43 hashar: restarted jenkins twice to get it to finally serve something :(
* 09:27 hashar: killed -9 jenkins :(
* 09:25 hashar: restarting jenkins
* 09:18 hashar: gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508
* 08:34 hashar: Jenkins hit by the evil bug again {{bug|48025}}. Taking as much trace as possible before restarting it
* 04:46 paravoid: swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance
* 03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013
* 00:15 logmsgbot: pgehres synchronized wmf-config  'Actually enabling wgCentralAuthAutoMigrate'
 
== May 9 ==
* 23:34 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php  'Enabling siteNotice div on all alpha and beta mobile sites'
* 23:27 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php  'Moving one more CentralAuth log line'
* 23:22 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/  'Updating CentralNotice to master for some bugfixes'
* 23:07 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/  'Updates to CentralAuth'
* 21:55 hashar: Jenkins / Zuul is back up (for now).
* 21:45 hashar: restarted Jenkins to enable access logs in /var/log/jenkins/access.log {{gerrit|63068}}
* 21:42 hashar: forced Zuul to stop. All events have been dropped :(
* 21:31 hashar: restarting Zuul to make it query Jenkins directly without SSL + apache proxy {{gerrit|63069}}.
* 21:29 Jeff_Green: pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed
* 21:28 ^demon: gerrit repositories will now recursively merge
* 21:17 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/WikimediaMessages  'Updating WikimediaMessages to master'
* 20:48 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/GuidedTour  'Actually updating GuidedTour to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging  'Updating GuidedTour to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging  'Updating EventLogging to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/ConfirmEdit  'Updating ConfirmEdit to master'
* 20:21 hashar: restarting Jenkins.
* 20:20 hashar: Jenkins died :/
* 19:17 mutante: fixing file permissions in /a/e3/E3Analysis repo on stat1001
* 19:09 mutante: restarted Apache and then puppetmaster on stafford, that revived it
* 18:56 mutante: attempting to restart broken puppetmaster on stafford
* 18:27 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php  'Enabling CentralAuth debug log'
* 18:00 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php  'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth'
* 15:56 cmjohnson1: powering down barium /relocate server to c1
* 15:27 logmsgbot: anomie synchronized wmf-config/CommonSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 15:26 logmsgbot: anomie synchronized wmf-config/InitialiseSettings-labs.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 11:25 hashar: upgrading Zuul to latest master (makes report times friendlier)
* 08:54 hashar: Jenkins apparently working fine.  I have upgraded a few plugins, that seems to have fixed the Gerrit jobs.
* 08:22 hashar: restarting Jenkins again
* 08:21 hashar: upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris)
* 07:39 hashar: restarting jenkins. All its http thread are locked :/
* 07:25 hashar: Jenkins HTTP interface died :/
* 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May  9 03:39:55 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May  9 02:15:54 UTC 2013
 
== May 8 ==
* 22:27 RobH: authdns update to migrate racktables from hooper to magnesium.  both servers are already using db1001 as master, which is why hooper is slow to load racktables data
* 22:24 logmsgbot: mholmquist synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php  'Fixing Special:Preferences for wikis with UploadWizard'
* 21:45 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
* 21:37 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
* 21:07 Krinkle: Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276)
* 20:21 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/  'CentralNotice security bugfix 48255'
* 20:15 Jeff_Green: authdns-update to swap in grosley for dead aluminium
* 19:41 RobH: magnesium is future racktables host, disregard errors for now
* 19:41 cmjohnson1: accidently removed power from magnesium
* 19:00 Reedy: Ensuring securepoll database tables exist on all wikis
* 18:35 logmsgbot: reedy synchronized php-1.22wmf3/extensions/CentralAuth
* 18:31 Jeff_Green: updating and rebooting grosley
* 18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:17 logmsgbot: reedy synchronized wmf-config/
* 18:16 logmsgbot: reedy synchronized database lists files:
* 18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable EducationProgram on mkwiki'
* 18:07 Reedy: Created EducationProgram tables on mkwiki
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3
* 18:00 RobH: authdnsupdate for racktables2
* 17:46 apergos: )
* 17:46 apergos: about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script
* 17:42 Jeff_Green: kernel updates and reboot on pappas
* 17:29 logmsgbot: yurik Finished syncing Wikimedia installation... : Zero extension
* 17:18 RobH: restarting all mw jobrunners
* 17:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php  'Disabled TTMServerMessageUpdateJob jobs'
* 17:12 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php  'revert hack'
* 17:11 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php  'revert hack'
* 17:08 logmsgbot: yurik Started syncing Wikimedia installation... : Zero extension
* 15:01 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
* 14:22 paravoid: ceph osd out 39, primary for stuck unclean pg 3.5c6
* 14:21 paravoid: ceph osd out 128, primary for stuck unclean pg 3.19e1
* 13:59 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
* 13:59 mark: Removing mc1009 from the eqiad memcached pool
* 06:14 apergos: restarted adminbot (morebot) on wikitech-static
 
== May 7 ==
* 23:42 logmsgbot: catrope synchronized php-1.22wmf3/resources/startup.js  'touch'
* 23:42 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js  'touch'
* 23:39 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:35 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/  'Appling update to fix CentralNotice message editing bug'
* 23:30 mutante: running addWiki.php for iegcomwiki
* 23:26 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php
* 23:23 mutante: depool mw1041 in pybal
* 23:22 mutante: removed mw1041 from dsh groups mediawiki-installation and apaches
* 23:20 mutante: mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
* 23:17 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/  'E3 Lightning deploy for GettingStarted'
* 23:17 LeslieCarr: upgrading srv* machines
* 22:48 Tim: on marmontel: disabled logrotate
* 22:36 logmsgbot: aaron synchronized wmf-config/mc-pmtpa.php  'removing mc15 for hw maint'
* 22:34 mutante: upgrading PHP_CodeSniffer on gallium
* 22:31 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/62744/'
* 22:10 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/62650/'
* 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
* 21:47 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:09 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
* 20:53 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php  'Live hack to avoid translationNotificationJob jobs.'
* 20:44 logmsgbot: reedy synchronized database lists files:
* 20:30 andrewbogott: copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
* 20:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 20:11 logmsgbot: reedy synchronized docroot/login/
* 20:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
* 19:56 logmsgbot: reedy synchronized wmf-config/
* 19:55 logmsgbot: reedy synchronized database lists files: loginwiki
* 19:37 mutante: DNS update - switch old secure.wm
* 19:30 mutante: apache-graceful-all, move old secure.wm redirects
* 18:54 cmjohnson1: authdns update
* 18:42 logmsgbot: dzahn synchronized docroot
* 18:42 mutante: sync-docroot for login.wm
* 18:38 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'adding db1042,1059 to s4'
* 18:03 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css  'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
* 17:59 logmsgbot: awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css  'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
* 17:43 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/  'Updating FundraiserLandingPage for WMF fundraising in FR'
* 17:38 kaldari: killed copyExistingEmailPreference.php on enwiki
* 17:29 Reedy: Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
* 17:26 logmsgbot: mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
* 17:19 mutante: killed hanging pdns related processes on ns1 and restarted pdns
* 17:15 kaldari: begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
* 17:02 mutante: DNS update - add login.wikimedia and login.m.wikimedia
* 16:57 logmsgbot: mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
* 16:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:35 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:31 RobH: professor being worked on, network is offline =(
* 16:29 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:19 gwicke: updated Parsoid to current master
* 16:03 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:46 logmsgbot: reedy synchronized docroot
* 14:47 cmjohnson1: authdns update
* 14:33 cmjohnson1: authdns update
* 13:09 mark: Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues
* 09:15 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php  'New ULS config part 2'
* 09:14 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'New ULS config part 1'
* 09:09 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/  'ULS to master'
* 08:21 paravoid: ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
* 07:59 paravoid: partially upgrading ceph to 0.56.6
* 07:27 Tim: on holmium: reverted firewall, too hard
* 07:22 Tim: on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
* 07:00 Tim: on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
* 06:28 Tim: on marmontel: disabled apache logrotate, stopped apache, disabled apache init script
* 04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May  7 04:19:39 UTC 2013
* 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue May  7 02:25:11 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May  7 02:16:06 UTC 2013
* 01:29 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php  'Two fixes to User::getNewMessageRevisionId() from kaldari'
* 01:29 logmsgbot: reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
* 00:38 logmsgbot: reedy synchronized php-1.22wmf3/includes/User.php  'Live hack to stop fatals'
* 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/extensions/Echo  'Update Echo to master'
* 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/docs/hooks.txt  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/includes/WikiPage.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/Skin.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:02 logmsgbot: bsitu synchronized php-1.22wmf3/includes/OutputPage.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
 
== May 6 ==
* 23:25 mutante: restarting prelabsdbdb on db1054
* 22:48 mutante: apache-graceful-all, add iegcom virtual host
* 21:00 mutante: blog is back up
* 21:00 mutante: holmium: bring up interface lo, comment "search" line in resolv.conf, restart varnish, ...
* 20:46 paravoid: swift replication scripts running on ms-fe1002 under screen -- kill if they have any effect on production
* 20:15 mutante: DNS update - sync with ns2
* 20:04 mutante: restarting parsoid on wtp1001
* 20:03 mutante: restarting pdns on ns0
* 20:01 mutante: killing parsoid server.js on wtp1001
* 19:59 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php  'f257ddac39194f8bfcfd145104999fdba83d33f5'
* 19:59 mutante: DNS update - switch all blog cnames over to holmium
* 19:30 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 19:25 LeslieCarr: moved gateway from csw1-esams to cr1-esams
* 19:18 Krinkle: Reloading Zuul to deploy I259585e78ee
* 19:18 LeslieCarr: rebooting ns2
* 19:16 logmsgbot: reedy synchronized wmf-config/
* 19:13 binasher: rebuilt db1016 from hotbackup of db1001; pmtpa side of m1 needs to be rebuilt as well
* 18:42 RobH: ns2/nescio appears offline, investigating
* 18:37 RobH: ns2 appears offline
* 18:37 RobH: authdns-update crashed on ns2, rerunning
* 18:33 RobH: holmium
* 18:33 RobH: authdns-update for new server homium
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf3
* 17:18 LeslieCarr: marmontel's interface is now in the sandbox vlan
* 17:16 LeslieCarr: turned off marmontel's network port
* 16:12 logmsgbot: reedy synchronized docroot
* 13:55 hashar: gallium: installed manually libcidr0-dev libanon0-dev . Puppet change is {{gerrit|62434}}
* 09:51 paravoid: upgrading Ganglia Web to 3.5.7
* 08:25 hashar: Zuul upgrading to upstream version 7191ee8 + 1 wmf hack.
* 06:45 Nemo_bis: Translate search on vanadium keeps being broken https://toolserver.org/~nemobis/tmp/SearchTranslations.log (now RT-5085)
* 06:15 paravoid: pushing new swift rings: ms-be1 weight 50->0, ms-be2 33->75
* 02:19 paravoid: restarting gerrit
* 02:18 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
 
== May 5 ==
* 10:56 Nemo_bis: "Search service is temporarily unavailable" since unknown time https://meta.wikimedia.org/wiki/Special:SearchTranslations?query=amen&uselang=en
* 03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May  5 03:36:17 UTC 2013
* 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun May  5 02:25:44 UTC 2013
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May  5 02:17:22 UTC 2013
 
== May 4 ==
* 22:42 logmsgbot: reedy synchronized wmf-config/interwiki.cdb  'Updating interwiki cache'
* 21:44 Reedy: Manually force ran TorBlock/loadExitNodes.php
* 21:43 logmsgbot: reedy synchronized php-1.22wmf3/extensions/TorBlock/
* 16:03 logmsgbot: reedy synchronized wmf-config/
* 16:02 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
* 04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May  4 04:36:09 UTC 2013
* 02:21 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat May  4 02:20:30 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May  4 02:12:01 UTC 2013
* 01:17 Krinkle: binasher restarted gerrit
 
== May 3 ==
* 23:49 Krinkle: Gerrit is completely unresponsive (all http and ssh time out, serving HTTP 500). Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061
* 22:19 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'adding new pmtpa slave'
* 21:10 cmjohnson1: db10 is decommissioned...powering down for the final time
* 20:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
* 19:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
* 07:52 MaxSem: OMG logmsgbot is a terminator!
* 06:16 logmsgbot: tstarling synchronized wmf-config/missing.php
* 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May  3 04:03:12 UTC 2013
* 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri May  3 02:26:02 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May  3 02:15:45 UTC 2013
* 01:25 hashar: Zuul processed all its events, seems to be fine now. *sleep*
* 00:33 logmsgbot: dzahn synchronized docroot
* 00:32 mutante: sync-docroot for iegcom wiki
* 00:25 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php  '6512451451ac6617bedbfce7be17045bed794d70'
* 00:16 mutante: DNS update - add iegcom
 
== May 2 ==
* 23:20 mutante: restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between)
* 23:18 hashar: yeah after hours and hours of fighting, Jenkins is finally working again.
* 23:11 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
* 22:59 hashar: restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason
* 22:59 mutante: repooling search1016
* 22:40 mutante: gallium, run puppet, graceful Apache to deploy split log files
* 22:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1020 for upgrade'
* 22:32 mutante: depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis)
* 22:29 mutante: repooling search1015
* 22:23 hashar: Zuul restarted.  The bug about slowness is {{bug|48025}}
* 22:15 hashar: restarting Zuul
* 22:13 hashar: Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. {{gerrit|62095}}
* 22:04 mutante: depooling search1015 in pybal
* 21:43 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GuidedTour/  'Sync GuidedTour to 1.22wmf3 for E3 deployment'
* 21:42 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GuidedTour/  'Sync GuidedTour to 1.22wmf2 for E3 deployment'
* 21:38 mutante: importing wikimania2014wiki into search indexers
* 21:35 mutante: restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki
* 21:28 logmsgbot: mflaschen synchronized php-1.22wmf3/skins/common/shared.css  'Sync font-size change for edit section links'
* 21:15 logmsgbot: spage synchronized php-1.22wmf2/extensions/ConfirmEdit  'update 1.22wmf2 to wmf3 version of ConfirmEdit'
* 21:14 LeslieCarr: adding GEANT via fiberring to avoid-paths
* 20:21 logmsgbot: reedy synchronized php-1.22wmf3/includes/
* 20:15 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
* 19:51 Jeff_Green: authdns-update to move db1025 to frack.eqiad.wmnet
* 19:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb  'Updating interwiki cache'
* 19:40 ottomata: rebooting oxygen
* 19:33 ottomata: re-enabling puppet on cp1031, it was administratively disabled.  running puppet there.
* 19:24 RobH: professor was already powered down (why?)  starting it back up now
* 19:23 ottomata: all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay
* 19:22 RobH: rebooting professor
* 19:16 Jeff_Green: moving db1025 into frack-fundraising1-c-eqiad
* 19:14 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo  'sync Echo ext for en.wiki'
* 19:14 ottomata: deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay
* 19:11 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo  'sync Echo ext'
* 19:09 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo  'sync Echo ext'
* 18:52 ottomata: removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke')
* 18:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb  'Updating interwiki cache'
* 18:41 ottomata: varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay
* 18:30 Krinkle: Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring>
* 17:59 Krinkle: Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs.
* 17:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 17:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:48 logmsgbot: reedy synchronized wmf-config/
* 17:37 Krinkle: Jenkins keeps clogging up. Starting an emergency restart.
* 17:00 Krinkle: Jenkins is nearing 100% CPU on gallium, what is Jenkins doing?
* 16:59 Krinkle: Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating...
* 16:52 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/  'ULS to master'
* 15:13 hashar: Jenkins made Jenkins to instantly restart (was {{bug|47120}}) ) by deleting the downstream-buildview  plugin.
* 15:11 hashar: restarted Zuul
* 14:12 hashar: Upgraded Jenkins from 1.480.3 to LTS 1.509.1.  Restarted it.
* 14:10 hashar: shutting down Zuul
* 14:09 hashar: upgrading Jenkins (unscheduled maintenance).
* 13:01 logmsgbot: maxsem synchronized wmf-config/mobile.php  'https://gerrit.wikimedia.org/r/#/c/61941/'
* 12:59 mark: Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams
* 09:55 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:53 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:50 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php  'https://gerrit.wikimedia.org/r/#/c/61962/'
* 06:04 apergos: powercycle snapshot1001, upgrade to precise
* 05:46 apergos: powercycle snapshot1004, upgrade to precise
* 05:28 apergos: snapshot1003 powercucle, upgrading to precise
* 05:28 Tim: on professor: manually started carbon-cache.py
* 05:15 Tim: professor is down, no response on serial console, rebooting
* 05:11 logmsgbot: tstarling synchronized wmf-config/jobqueue-pmtpa.php
* 05:08 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
* 04:47 Tim: on rdb1001-1002, set requirepass and masterauth in soft state
* 04:44 Tim: on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state
* 04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May  2 04:39:48 UTC 2013
* 04:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:57 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
* 03:47 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 03:40 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
* 02:46 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu May  2 02:32:10 UTC 2013
* 02:22 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May  2 02:21:30 UTC 2013
* 02:21 Tim: on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces)
* 02:21 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 01:20 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js  'Poking bits to try and get the new banner controller deployed for CentralNotice'
* 01:17 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js  'Poking bits to try and get the new banner controller deployed for CentralNotice'
* 00:16 mutante: @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet
 
== May 1 ==
* 23:28 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
* 23:23 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 23:18 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 22:42 mutante: powercycling frozen mw1041
* 22:14 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
* 22:06 logmsgbot: dzahn synchronized ./multiversion/MWMultiVersion.php
* 22:02 logmsgbot: dzahn synchronized docroot
* 21:44 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
* 21:32 logmsgbot: dzahn gracefulled all apaches
* 21:25 ottomata: syncing new redirects.conf to apachces to redirect wikipediazero.org
* 21:14 Jeff_Green: stopping mysql on db78 for cloning
* 20:50 ottomata: adding new wikipediazero.org zone alias, running authdns-update on dobson
* 19:45 logmsgbot: reedy synchronized php-1.22wmf3/cache/l10n/
* 19:15 mutante: DNS update - remove project2, anthony and yongle :p
* 19:05 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata
* 18:36 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May  1 18:36:02 UTC 2013
* 18:32 mutante: DNS update - add wikimania2014 entries
* 18:31 RobH: disabled owa1-3 switchports, starting remote wipe
* 18:21 RobH: owa1/2/3 offlinging and wiping
* 18:20 logmsgbot: reedy synchronized php-1.22wmf3/extensions/  'Update extensions for wikibase deploy'
* 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3
* 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3
* 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3
* 17:58 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php  'syncing Hooks in Echo'
* 17:33 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php  'syncing Hooks in Echo'
* 17:20 LeslieCarr: puppet working on bits caches again after gerrit change 61793
* 17:01 RobH: torrus compiling, manutilus apache offline
* 16:53 RobH: torrus deadlocked, fixin
* 15:54 mark: Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2
* 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 10:30 logmsgbot: reedy synchronized php-1.22wmf2/extensions/ParserFunctions/  'Update to master to fix common fatals'
* 10:08 mark: Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces
* 03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May  1 03:49:55 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May  1 02:27:30 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed May  1 02:16:40 UTC 2013
* 02:09 logmsgbot: aaron synchronized wmf-config/filebackend.php  'Turned autoResync back on again for multiwrite backend'
* 00:46 logmsgbot: tstarling synchronized wmf-config/PrivateSettings.php
 
== April 30 ==
* 23:52 Krinkle: Gracefully restarted Zuul on gallium
* 23:52 Krinkle: Ryan_Lane restarted the gerrit service
* 23:44 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php  'Syncing CommonSettings for Echo'
* 23:34 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'Syncing InitialiseSettings for Echo'
* 23:28 logmsgbot: kaldari synchronized /php-1.22wmf3/extensions/Echo  'Updating Echo for some minor bugfixes'
* 23:18 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/  'Revert MobileFrontend'
* 23:15 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/  'Revert MobileFrontend'
* 23:12 logmsgbot: tstarling synchronized wmf-config/missing.php  'rewrite (I17989fc4)'
* 23:01 Krinkle: Graceful restart of Zuul on gallium
* 22:56 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php  'Enabling Extension:AccountAudit on all wikis and wmgEchoHelpPage'
* 22:09 Jeff_Green: rolling back db1025 change b/c "mysqld: unknown option '--federated'"
* 22:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/  'grrr'
* 22:00 logmsgbot: maxsem synchronized wmf-config
* 21:58 Jeff_Green: restarting mysqld on db1025 to enable federated engine for testing
* 21:56 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/  'Weekly mobile deployment'
* 21:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/  'Weekly mobile deployment'
* 20:04 logmsgbot: kaldari synchronized echowikis.dblist  'syncing echowikis.dblist for cron jobs'
* 19:59 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'turning on Echo for English Wikipedia'
* 19:49 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:19 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:03 cmjohnson1: removing db10 certs from puppet
* 17:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php  'Syncing CommonSettings for Echo deployment'
* 17:45 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'Syncing InitialiseSettings for Echo deployment'
* 13:57 cmjohnson1: jenkins seems to not be working -- restarting
* 11:27 Nikerabbit: refresh-translatable-pages.php finished for mediawikiwiki and metawiki
* 04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 30 04:06:11 UTC 2013
* 02:34 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue Apr 30 02:33:50 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 30 02:15:03 UTC 2013
* 02:04 paravoid: cleaning up neon's /var/log (100% full), restarting icinga...
* 00:59 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/  'Update VE'
* 00:44 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php  'Touching InitialiseSettings'
* 00:32 logmsgbot: pgehres Finished syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
* 00:00 logmsgbot: pgehres Started syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
 
== April 29 ==
* 23:20 logmsgbot: asher synchronized wmf-config/db-pmtpa.php  'updated s5 pmtpa topology'
* 23:09 binasher: s5 pmtpa master changed to db73 - MASTER_LOG_FILE='db73-bin.000090', MASTER_LOG_POS=978211000
* 23:08 AaronSchulz: finished creating AccountAudit tables
* 23:06 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php  'Re-enable WikiLove, bug fix is deployed'
* 23:04 logmsgbot: catrope Finished syncing Wikimedia installation... :
* 22:40 logmsgbot: catrope Started syncing Wikimedia installation... :
* 22:28 anomie: (Including backport of gerrit change 61494 in the scap)
* 22:27 RoanKattouw: Running scap
* 22:23 RoanKattouw: Running scap again
* 22:19 logmsgbot: catrope Started syncing Wikimedia installation... :
* 22:13 hashar: gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend.
* 22:03 hashar: jenkins web server is no more responding on port 8080.  Might have to restart it.
* 22:00 hashar: gallium / jenkins : blackholed a web drawer using  ip route add blackhole <IP>/32
* 21:54 hashar: Jenkins is in trouble
* 21:52 logmsgbot: reedy synchronized php-1.22wmf3/includes/DefaultSettings.php  'Fix wgVersion'
* 21:31 notpeter: upgrading db45 to precise
* 21:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enabling VisualEditor on viwiki and test2wiki'
* 20:31 AaronSchulz: Doing another original/math/timeline sync/copy run
* 20:28 logmsgbot: aaron synchronized wmf-config/filebackend.php  'Enabled multiwrites for ceph/swift'
* 19:19 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php
* 19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3
* 19:06 logmsgbot: reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
* 18:46 logmsgbot: reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
* 18:37 logmsgbot: reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove
* 18:21 logmsgbot: reedy synchronized php-1.22wmf3/extensions/
* 18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
* 18:03 notpeter: deploying new frontend squid conf
* 17:51 LeslieCarr: restarting icinga-wm
* 17:35 cmjohnson1: removing asw-c1 for replacement
* 17:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
* 17:09 mutante: fix NRPE checks on ms-be2, running as "weird user id" bug
* 17:05 mutante: removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p
* 17:05 logmsgbot: csteipp synchronized php-1.22wmf2/includes/
* 17:05 logmsgbot: reedy synchronized live-1.5
* 17:02 mutante: attempt to fix NRPE checks on ms-be9, restart as correct user
* 16:49 logmsgbot: reedy synchronized docroot
* 16:44 logmsgbot: reedy synchronized php-1.22wmf3/  'initial sync of 1.22wmf3 files'
* 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013
 
== April 28 ==
* 13:14 hashar: restarting Zuul, it has a finished build stuck in the `postmerge` pipeline.
* 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013
 
== April 27 ==
* 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013
* 00:08 mutante: make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen
 
== April 26 ==
* 23:58 mutante: upgrading mysql packages on marmontel (blog)
* 23:53 mutante: make SSL on the Blog use SSLCACertificatePath and c_rehash certs on marmontel
* 20:11 hashar: killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died
* 19:19 hashar: jenkins died again sorry :(  But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds!
* 18:04 mwalker: Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067
* 15:56 hashar: jenkins: enabled ExtraSettings.php loading {{gerrit|61022}}.  That enables a bunch of MediaWiki settings for all mediawiki related jobs, namely displaying errors, moving the temp dir to tmpfs, showing exceptions details and enabling development warnings.
* 13:49 mark: Rebooting hooft
* 13:48 mark: dist upgrade of hooft to precise
* 13:48 mark: Moved udpmcast from hooft to nescio
* 13:16 ^demon: gerrit replication freaked out, restarting
* 06:38 hashar: Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :(  Would be up in roughly 40 minutes :(
* 03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013
* 00:57 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'temporarily disabling WikiLove per bug 47457'
* 00:53 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/  'Update VisualEditor to master'
* 00:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Clean up VisualEditor config'
* 00:33 Tim: on fenari: fixed permissions on /etc/ssh/ssh_known_hosts
* 00:33 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Remove wmgVisualEditorHide'
 
== April 25 ==
* 23:09 logmsgbot: mwalker synchronized docroot/bits/favicon/  'Pushing new favicons for incubator, wiktionary, wiktionary.en, wikibooks, wikinews, and wikispecies'
* 22:32 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
* 22:22 ^demon: new ssl cert (gerrit.wikimedia.org) live on gerrit.
* 20:55 notpeter: reimaging db55
* 20:45 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 20:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 20:07 awight: payments cluster updated from 7f70ccd580d to fdf308c8f
* 19:12 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php  'updating for new schema number'
* 18:52 logmsgbot: reedy synchronized docroot
* 18:19 LeslieCarr: cleaning up old home directories on fenari
* 17:00 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Removing oldskin notice'
* 16:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php  'Enable VisualEditor on 14 more Wikipedias'
* 16:38 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/  'Update VisualEditor to master'
* 16:20 hashar: Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/  Should take roughly 40 minutes (no idea when it started)
* 16:09 hashar: Zuul git operations are slow again :/
* 14:57 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php  'rm debug hack'
* 14:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Turn mwsearch debug log back off, its not phps fault'
* 14:53 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Turn mwsearch debug log on'
* 14:52 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php  'Debugging'
* 11:56 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php  'I2f0990fa087a'
* 11:56 logmsgbot: krinkle synchronized docroot
* 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013
* 00:45 mutante: upgrading packages on manutius
* 00:30 mutante: DNS update - removing internproxy, removing tesla (minus mgmt)
 
== April 24 ==
* 23:04 logmsgbot: dzahn gracefulled all apaches
* 22:39 logmsgbot: dzahn gracefulled all apaches
* 22:39 LeslieCarr: upgrading mw's
* 22:38 LeslieCarr: replaced smokeping targets for sdtpa c3 and sdtpa d3
* 22:26 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php  'New location for icons - Ib5554c54729'
* 22:26 mutante: sync-apache, add coffee
* 22:24 logmsgbot: krinkle synchronized docroot/bits/apple-touch/  'New location for icons - Ib5554c54729'
* 21:47 mutante: DNS update - remove survey, remove wikitech-old, remove argon
* 21:47 logmsgbot: demon synchronized php-1.22wmf2/extensions/Wikibase  'b4b5fb4046175ca29834a4669c253c97ac5ccd07'
* 21:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
* 21:34 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Turn mwsearch log back on'
* 21:33 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
* 21:26 mutante: DNS update - readding voip
* 21:17 cmjohnson1: removing sq33 from dsh node groups and pybal cfg
* 20:32 RobH: racktables on hooper upgraded to recent stable
* 20:26 RobH: racktables flap again, working on it
* 20:16 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch
* 20:04 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch  'Deploying I48480cb6'
* 20:03 logmsgbot: demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php  'Deploying Ib7abe8d5'
* 19:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php  'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e'
* 19:17 mutante: deploying Bugzilla wikimedia extension changes for improved wikilinking
* 19:16 RobH: racktables going offline a moment for upgrade
* 18:45 logmsgbot: reedy synchronized wmf-config/  'wikidata stuffs'
* 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2
* 18:09 mutante: DNS update - removing unused remnants
* 18:09 sbernardin: replacing bad disk on db59
* 17:31 sbernardin: taking down ms-be2 for hw replacement
* 15:09 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php  'bug 47597'
* 15:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php  'bug 47597'
* 09:05 hashar: restarted Zuul. Upgraded to e403579 (latest master)
* 08:58 hashar: upgrading Zuul
* 06:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013
* 00:03 logmsgbot: awjrichards synchronized wmf-config/mobile.php  'Disable mobile sitenotice everywhere but testwiki'
* 00:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php  'Disable mobile sitenotice everywhere but testwiki'
 
== April 23 ==
* 23:38 mutante: cleaning up all unused Apache sites on singer, gzip, move to /root for backup
* 23:33 mutante: disabling old techblog apache site config on singer
* 23:32 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 23:15 logmsgbot: mwalker synchronized docroot/bits/favicon/  'Pushing wikidata, wikimania, wikiquote, and wikiversity favicons from Isarra'
* 23:12 logmsgbot: dzahn gracefulled all apaches
* 23:07 logmsgbot: maxsem synchronized wmf-config  'Mobile banners again, hopefully fixed'
* 23:05 mutante: sync-apache, add wikepdia.org and .com typo domains
* 22:28 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
* 22:21 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score
* 22:12 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php  'debugging'
* 22:05 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php  'debugging'
* 22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'rv debugging'
* 22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Score'
* 21:59 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php  'debugging'
* 21:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable score debug log'
* 21:40 logmsgbot: maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/60512/'
* 21:23 logmsgbot: maxsem synchronized wmf-config
* 21:17 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
* 21:15 logmsgbot: maxsem synchronized wmf-config  'Mobile banners everywhere - https://gerrit.wikimedia.org/r/#/c/60362/'
* 21:14 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php  'temp logging'
* 21:13 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/  'Weekly mobile deployment'
* 21:12 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/  'Weekly mobile deployment'
* 21:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/  'Weekly mobile deployment'
* 21:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/  'Weekly mobile deployment'
* 20:56 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
* 19:36 logmsgbot: anomie synchronized php-1.22wmf2/extensions/Cite  'Update to master for bug 47291'
* 18:44 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php  'Re-enable AFTv5 on enwiki'
* 17:49 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 17:49 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 16:34 logmsgbot: reedy synchronized php-1.22wmf2/extensions/TorBlock
* 16:34 logmsgbot: reedy synchronized php-1.22wmf1/extensions/TorBlock
* 12:34 paravoid: reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia
* 09:57 pp-pdf1: restarted all services
* 09:56 pp-pdf1: - alow numprocs to be configured via initenv.sh
* 09:16 pp-pdf1: restarted all services
* 09:12 pp-pdf1: restarted all services
* 09:10 pp-pdf1: restarted all services
* 08:46 logmsgbot: maxsem synchronized docroot/  'https://gerrit.wikimedia.org/r/#/c/60375/'
* 08:41 logmsgbot: maxsem synchronized wmf-config  'Beta changes, no-op for production'
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013
* 00:02 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'bookkeeping for pmtpa slaves (this time for real)'
* 00:01 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'bookkeeping for pmtpa slaves'
 
== April 22 ==
* 23:51 notpeter: reimaging db44
* 22:57 binasher: mysql query logs / ishmael migrated from db9 to db1001
* 21:41 LeslieCarr: upgrading even more mw servers in tampa
* 20:12 Reedy: Running namespaceDupes on kowikiversity
* 19:57 Reedy: Running namespaceDupes on zhwikibooks
* 19:55 Reedy: Running namespaceDupes on uzwiki
* 19:55 logmsgbot: reedy synchronized wmf-config/
* 19:50 logmsgbot: reedy synchronized images/
* 19:50 logmsgbot: reedy synchronized docroot
* 19:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'Enable Score extension on the cluster'
* 18:49 logmsgbot: catrope synchronized echowikis.dblist
* 18:44 LeslieCarr: sq33 is going to be decomissioned… long live…. nothing else in tampa
* 18:39 mutante: changing #mediawiki to #wikimedia-dev in wikibugs startup script (just in case, should not even be needed by puppetized ircecho)
* 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:26 LeslieCarr: still doing package upgrades on mw* machines
* 18:20 LeslieCarr: rebooting nonresponsive sq33
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf2
* 17:37 LeslieCarr: dist-upgrading some mw machines, should be low/no impact
* 16:51 mutante: restarting wikibugs on mchenry
* 16:43 logmsgbot: mark synchronized wmf-config/InitialiseSettings.php
* 16:37 mutante: added abaso to wmf-deployment group on gerrit (new mediawiki deployer)
* 16:11 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'New caching everywhere'
* 16:02 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
* 15:57 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  '$wgMFVaryResources on enwiki, https://gerrit.wikimedia.org/r/#/c/60286/'
* 14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 22 02:14:58 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 22 02:08:56 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 22 02:05:30 UTC 2013
* 00:13 Reedy: Running namespaceDupes.php --fix via foreachwiki in screen session on terbium
 
== April 21 ==
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 21 02:37:21 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 21 02:16:56 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 21 02:10:46 UTC 2013
 
== April 20 ==
* 16:54 ottomata: resetarting apache2 on stafford in attempt to fix puppetmaster
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 20 02:35:55 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 20 02:14:31 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 20 02:09:07 UTC 2013
* 00:09 logmsgbot: aaron synchronized php-1.22wmf1/includes/job/JobQueueRedis.php  'deployed c9e702e7f73f41f7bcd2deb045085689d5cb4229'
* 00:03 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueRedis.php  'deployed d2e4fdfed03148fdd1a564925ce351bec6f9b935'
 
== April 19 ==
* 23:07 Coren: brief reboot of labstore3 -> test to see that everything starts up as expected.
* 20:39 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php  'Setup redis job queue debug log'
* 18:36 Jeff_Green: increased php memory limit for fundraising civicrm instance
* 17:55 LeslieCarr: restarting etherpad on hooper
* 17:15 binasher: running enwiki.abuse_filter_log migration - drop afl_user index, added user_timestamp
* 16:10 LeslieCarr: flushed celsus varnish cache
* 06:31 Tim: deploying apache conf change for www.wikidata.org redirect (I7bb872fd)
* 03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013
* 02:27 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013
* 02:26 logmsgbot: aaron cleared profiling data
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013
* 01:54 logmsgbot: aaron cleared profiling data
* 01:54 logmsgbot: aaron synchronized wmf-config/StartProfiler.php  'Removed cli profiling for now.'
* 00:29 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor  'Update VisualEditor to master'
* 00:29 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor  'Update VisualEditor to master'
 
== April 18 ==
* 23:40 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor  'Update VisualEditor to master'
* 23:40 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor  'Update VisualEditor to master'
* 23:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php  'Deploy 5c5455c81bc8dee0ae2a0410655da639e2f84e87 ($wgVisualEditorParsoidProblemReportURL)'
* 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/processEchoEmailBatch.php  'More echo db fix'
* 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/DbEchoBackend.php  'More echo db fix'
* 22:51 logmsgbot: awjrichards synchronized wmf-config/mobile.php  'Enable mobile sitenotice on testwiki'
* 22:51 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php  'Enable mobile sitenotice on testwiki'
* 22:48 notpeter: reimaging db35
* 22:46 logmsgbot: mwalker synchronized docroot/bits/favicon/wikisource.ico  'New and shiney ws.o favicon by Isarra'
* 22:45 logmsgbot: mwalker synchronized docroot/bits/favicon/mediawiki.ico  'New and shiney mw.o favicon by Isarra'
* 22:35 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
* 22:35 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'adjusting weights on pmtpa slaves'
* 22:26 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 22:15 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 22:09 notpeter: correction: that was db33 out, not db31
* 22:03 logmsgbot: py synchronized wmf-config/db-pmtpa.php  'db31 and db35 out, db72 and db73 in'
* 21:56 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/EchoDbFactory.php  'Db cache fix'
* 21:41 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
* 21:39 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 21:38 mutante: merging changes on bugzilla to now support RT in "See also" field
* 20:45 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GettingStarted/  'Sync GettingStarted 1.22wmf2 for E3 deploy'
* 20:43 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/  'Sync GettingStarted for E3 deploy'
* 20:36 logmsgbot: reedy synchronized wmf-config/
* 20:15 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php  'wgMaxImageArea and wgMaxAnimatedGifArea to 50MP'
* 19:52 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:14 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Echo email config + db cluster update to testwiki'
* 19:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Echo email config + db cluster update to testwiki'
* 18:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/jobs  'deployed 1e3eaed6fbe7fd91c15399cc17eac8602b34c9cd'
* 18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
* 17:57 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
* 17:50 RobH: authdns0update for git.w.o
* 17:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf2
* 17:06 logmsgbot: reedy synchronized php-1.22wmf2/extensions/
* 16:58 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Wikibase  '5d496b4ea2c8a68b9e1280a282530f7ce5232b68'
* 16:47 logmsgbot: ariel synchronized wmf-config/throttle.php
* 16:10 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 16:10 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 15:39 cmjohnson1: authdns update
* 15:10 mark: Reversed replication of volume fr_archive, now nas1001-a:fr_archive -> nas1-a:fr_archive
* 14:26 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:24 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:14 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 14:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php  'https://gerrit.wikimedia.org/r/#/c/59818/'
* 13:24 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 13:03 mark: Rebooting cp1041
* 08:48 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 07:57 logmsgbot: olivneh synchronized php-1.22wmf2/extensions/ConfirmEdit  'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
* 07:57 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ConfirmEdit  'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
* 07:43 hashar: jenkins raised max # of executors from 6 to 8
* 07:07 logmsgbot: maxsem synchronized wmf-config/mobile.php  'https://gerrit.wikimedia.org/r/#/c/59767/'
* 05:59 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 05:21 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
* 05:15 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
* 05:12 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php
* 05:09 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
* 03:57 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch
* 03:55 logmsgbot: tstarling synchronized php-1.22wmf2/includes/HttpFunctions.php
* 03:48 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch
* 03:46 logmsgbot: tstarling synchronized php-1.22wmf1/includes/HttpFunctions.php
* 03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 18 03:26:48 UTC 2013
* 02:24 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 18 02:23:54 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 18 02:14:26 UTC 2013
* 01:03 mutante: re-enabling Tomasz' RT account
* 00:42 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'setting db1004 to full weight'
* 00:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'adding db1004 at a warmup weight'
* 00:09 logmsgbot: demon synchronized docroot
 
== April 17 ==
* 23:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1004'
* 23:21 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting  'Finished update of CN and CR to latest'
* 23:20 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/
* 23:14 hashar: If jenkins/zuul goes wild overnight:  connect to gallium, killoff puppet in crontab then  /etc/init.d/zuul stop and drop me an email. Will take of it whenever I wake up.
* 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
* 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice  'Beginning update of CentralNotice and ContributionTracking to latest'
* 22:59 hashar: restarting Zuul
* 22:20 hashar: Jenkins repointing all jobs to use the new Zuul repo ( /srv/ssd/zuul/git ) {{gerrit|59744}}
* 22:06 hashar: jenkins restarted.  Restarting Zuul.
* 21:20 hashar: stopped Zuul. Will wait a bit till jenkins start up
* 21:19 hashar: restarted Jenkins
* 21:19 hashar: restarted Zuul
* 21:16 hashar: finished migrating zuul git repositories ( rsync -av /var/lib/zuul/git /srv/ssd/zuul  )
* 21:09 hashar: Zuul ETA: it is copying MobileFrontend extension right now so half way done
* 21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionaries to 1.22wmf2
* 21:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.22wmf2
* 21:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf2
* 21:00 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.22wmf1
* 20:59 hashar: stopping Jenkins that somehow restarted
* 20:59 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php  'moar'
* 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews.dblist and special wikis to 1.22wmf2
* 20:51 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php  'debugging'
* 20:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikimania and wikimedia to 1.22wmf2
* 20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some wikis to create some noise
* 20:42 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php  'debugging'
* 20:30 hashar: Switching zuul dir to /srv/ssd/zuul ( {{gerrit|58898}} and {{gerrit|58899}} )
* 20:18 hashar: stopping Zuul gracefully
* 20:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that due to large number of warnings...
* 20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and special to 1.22wmf2
* 19:57 hashar: gallium slowness is partly due to the ssd install.  The git repositories are on the slow raid where as the workspaces are on the ssd.  That means `git clone` can't do hardlink anymore :(
* 19:43 binasher: enwiki, dewiki, wikidata are fully on mariadb 5.5.30
* 19:33 hashar: jenkins: reducing number of executors from 10 to 6.
* 19:32 hashar: jenkins gallium has a huge queue of jobs. investigating.
* 19:31 hashar: jenkins / gallium has a huge queue of jobs. Investigating.
* 19:19 logmsgbot: asher synchronized wmf-config/db-pmtpa.php  's1/s5 writeable from pmtpa'
* 19:17 logmsgbot: asher synchronized wmf-config/db-eqiad.php  's1/5 writeable'
* 19:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php  's1/5 read-only'
* 19:12 logmsgbot: asher synchronized wmf-config/db-pmtpa.php  'preparing for master swaps'
* 19:03 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php  'deployed 1437b57bd475e5ecb06800f9f64b14ccffd0dedf'
* 19:00 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php  'deployed 1306792ad24be5b8d17355c58da655eb88c2fcc6'
* 18:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1050'
* 18:18 LeslieCarr: installing new linecard in cr2-eqiad , .1% chance of issues
* 18:16 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1050'
* 17:50 logmsgbot: aaron synchronized php-1.22wmf2/includes/User.php  'deployed 6a66776fd7afdf883ad6f82270453fc76833508b'
* 16:20 ^demon: running puppet on manganese (will restart gerrit)
* 13:05 apergos: restarted icinga on neon (the usual: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe)
* 13:03 hashar: jenkins: refreshing job mediawiki-core-phpcs-HEAD  (points to a wrong git repo path)
* 09:50 mark: Inserted libvpx 1.1.0-1+wmf1 packages into the precise-wikimedia APT repository
* 06:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 06:57:02 UTC 2013
* 06:51 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 06:51:28 UTC 2013
* 06:48 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 06:47:37 UTC 2013
* 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 03:13:56 UTC 2013
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 02:19:57 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 02:10:22 UTC 2013
* 00:38 mutante: blog: enabled new eventlogging by ori, removed old logging class, converted plugins/WMBlog from svn to git and sync with gerrit
* 00:31 Tim: on hume: changed UID for l10nupdate to be 10002, for consistency with puppet admins.pp
 
== April 16 ==
* 23:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'raising db1058 weight'
* 23:57 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1005, adding db1058 to s5'
* 23:55 logmsgbot: aaron cleared profiling data
* 23:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:48 logmsgbot: reedy synchronized wmf-config/abusefilter.php
* 23:46 logmsgbot: aaron synchronized wmf-config/StartProfiler.php  'Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID'
* 23:22 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/
* 23:22 logmsgbot: awjrichards synchronized php-1.22wmf1/extensions/MobileFrontend/
* 23:00 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting/  'Contribution reporting now pulls 2013 data'
* 22:59 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
* 22:51 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 22:51:08 UTC 2013
* 22:37 Tim: running l10nupdate manually on fenari to test change 58911
* 22:03 logmsgbot: anomie synchronized php-1.22wmf2/extensions/WikimediaMaintenance  'Update WikimediaMaintenance to master for l10nupdate fix'
* 22:02 logmsgbot: anomie synchronized php-1.22wmf1/extensions/WikimediaMaintenance  'Update WikimediaMaintenance to master for l10nupdate fix'
* 21:49 logmsgbot: maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/59449'
* 21:46 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/  'https://gerrit.wikimedia.org/r/#/c/59525/'
* 21:43 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:54 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php  'Updating CentralAuth/migrateAccount.php'
* 19:53 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php  'Updating CentralAuth/migrateAccount.php'
* 19:38 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php  'Updating CentralAuth/migrateAccount.php'
* 19:33 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php  'Updating CentralAuth/migrateAccount.php'
* 19:18 binasher: upgrading db1005 to precise + mariadb
* 19:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1005 from s5 for upgrade'
* 19:02 binasher: streaming s5 hotbackup from db45 to db1058
* 18:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1058 from s1'
* 18:34 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/HTMLCacheUpdate.php  'deployed 344ddf85b6d7b79c40a1b6b9ec7e0a1e05c0705b'
* 18:26 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 18:26 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 18:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Turning on footer contact link for en.wiki'
* 18:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Enable email bundling for test2wiki and mediawikiwiki'
* 18:13 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Enable email bundling for test2wiki and mediawikiwiki'
* 16:42 logmsgbot: aaron synchronized php-1.22wmf2/includes/WikiPage.php  'deployed 38d60cdafe036625a6ec8fa3a9e8a94ef450c85c'
* 16:41 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php  'deployed fa819f8e3579855f6d95552655de98f92ec1789c'
* 16:40 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/BacklinkCache.php  'deployed 1a94124385df4c452c68fadaa74bc2714a204761'
* 16:37 logmsgbot: aaron synchronized php-1.22wmf2/includes/OutputPage.php  'deployed 997ab92458946d10fed96ff4ba41c74eeaee0d20'
* 05:04 Tim: moving most cron jobs from hume to terbium
* 03:23 legoktm: test
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013
 
== April 15 ==
* 23:57 logmsgbot: csteipp synchronized php-1.22wmf2/includes/  'User gerrit versions of security fixes'
* 23:18 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice
* 23:13 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice/  'CentralNotice banner= fix'
* 22:31 logmsgbot: dzahn gracefulled all apaches
* 22:16 andrewbogott: Moved labs-morebots to tools project.
* 21:43 LeslieCarr: doing a few more random host upgrades of misc hosts
* 21:32 LeslieCarr: upgrading mw1070-mw1079
* 21:24 LeslieCarr: upgrading mw1060-mw1069
* 21:22 RobH: linode wikitech is now shut down \o/
* 21:13 logmsgbot: dzahn gracefulled all apaches
* 20:32 mutante: restarting pdns on ns1
* 20:30 mutante: DNS update - add wikpedia.org
* 19:37 logmsgbot: ariel gracefulled all apaches
* 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf2
* 15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
* 15:19 logmsgbot: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
* 14:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf2, scap to come
* 14:56 logmsgbot: reedy synchronized docroot
* 14:55 logmsgbot: reedy synchronized w
* 14:54 logmsgbot: reedy synchronized wmf-config/
* 14:49 logmsgbot: reedy synchronized php-1.22wmf2/  'Initial sync of 1.22wmf2'
* 12:24 mark: Depooled amssq47-amssq62 frontend squids
* 12:04 mark: Depooled knsq16-knsq22 frontend squids
* 09:20 hashar: jenkins died :(
* 08:22 hashar: updating Jenkins jobs to fetch ZUUL_COMMIT instead of ZUUL_BRANCH. That would solve a race condition between jobs ( {{bug|46723}} / {{gerrit|58865}} ).
* 05:50 logmsgbot: olivneh synchronized wmf-config/throttle.php  'Adding throttling exception for itwiki GLAM event which starts in four hours (Bug 47221, change Id9ae600ac)'
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 15 02:03:51 UTC 2013
 
== April 14 ==
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013
 
== April 13 ==
* 23:54 Coren: rebooting labstore3.  autofs is being silly
* 15:11 Coren: Fixed boot problem on labstore3; back to operating.
* 04:54 Coren: rebooted labstore3 again; kernel and lvm2 upgrade
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 13 02:12:27 UTC 2013
* 01:10 Coren: rebooted labstore3 again -> we hates lockd.  We hates it!
* 00:41 Coren: rebooting labstore3 -> silly lockd module
 
== April 12 ==
* 23:31 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
* 23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
* 23:14 Coren: rebooted labstore3 -> hung MegaCLI processes
* 22:41 notpeter: temp depooling search1015
* 20:53 mutante: install package upgrades on sanger
* 19:57 Ryan_Lane: finished opendj upgrade on virt0
* 19:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Disable mwsearch log'
* 18:54 mutante: restarting parsoid on wtp1004
* 18:25 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php  'Set pmtpa queue config to match eqiad'
* 18:17 logmsgbot: demon synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php  'Moar logging'
* 18:16 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php  'Moar logging'
* 18:00 Ryan_Lane: upgrade opendj on virt0
* 16:00 LeslieCarr: adding another link to asw-c-eqiad's uplink
* 15:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Debug group for mwsearch'
* 15:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php  'Debug group for mwsearch'
* 15:01 hashar: Changing Jenkins workspace directories from ${ITEM_ROOTDIR}/workspace to /srv/ssd/jenkins/workspace/${ITEM_FULLNAME} . My change has not been taken in account early on :(
* 14:54 hashar: jenkins is back up. Starting Zuul
* 14:44 cmjohnson1: analytics1007 has never worked..but working on it
* 14:21 hashar: restarting jenkins
* 14:03 hashar: gallium: created primary partition on /dev/sdb .  formatting using:  mkfs.xfs /dev/sdb1
* 13:36 hashar: gallium can't find the SSD, shut downing, will attempt another slot
* 12:59 hashar: gracefully shut-downing Jenkins and Zuul for scheduled maintenance.  Will shutdown server gallium just after.
* 09:39 hashar: jenkins: updated mediawiki-core-whitespaces job to use ZUUL_COMMIT as a refspec specifier (for {{bug| 46723}} )
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 12 02:15:30 UTC 2013
* 00:08 mutante: DNS update - point old bugzilla3 entry over to actual bugzilla server
 
== April 11 ==
* 23:35 LeslieCarr: upgrading mw1052 to mw1060
* 23:30 logmsgbot: csteipp synchronized php-1.22wmf1/extensions/RSS/
* 23:30 LeslieCarr: upgrading mw1051
* 23:24 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
* 23:22 mutante: shutting down xenon
* 23:20 LeslieCarr: upgrading mw1050
* 23:11 LeslieCarr: upgrading 1040 to 1049
* 23:04 mutante: install package upgrades on neon
* 23:00 LeslieCarr: upgrading mw1030 - 1039
* 22:51 K4-713: synchronized payments to 7f70ccd580d64
* 22:47 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralAuth  'Peter's log line change to CentralAuth'
* 22:41 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php  'Enable translate on CentralNotice'
* 22:39 logmsgbot: mwalker synchronized docroot/bits/favicon/wikipedia.ico  'Bi resolution wiki favicon'
* 22:38 LeslieCarr: upgrading mw1120 to mw1129
* 22:37 mutante: removing spence from dsh group mediawiki-installation
* 22:33 logmsgbot: mwalker synchronized php-1.21wmf12/extensions/CentralNotice
* 22:26 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice  'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
* 22:22 LeslieCarr: upgrading mw1110 to mw1119
* 21:41 logmsgbot: kaldari synchronized php-1.22wmf1/extensions/Echo  'syncing Echo for bugfix'
* 21:35 ottomata: emery back up and upgraded to precise (thanks mutante!)
* 21:32 ^demon: fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
* 21:14 mutante: emery - revoking old puppet cert, resiging, initial puppet run
* 21:04 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php  'Remove jquery.clickTracking from AFT dependencies'
* 21:01 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
* 20:53 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/  'Update GettingStarted extension for E3 deployment'
* 20:34 logmsgbot: mflaschen synchronized wmf-config  'Config change for GettingStarted'
* 20:32 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 20:11 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:58 mutante: emery - upgrading from lucid to precise
* 19:46 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php  'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
* 19:41 Jeff_Green: running authdns-update to propagate frdata cname
* 19:31 AaronSchulz: Refreshed the job queue aggregator hash table
* 19:31 mutante: running puppet on brewster, testing lucid image on emery, rebooting again
* 19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Turn Echo back on for testwiki'
* 19:16 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php  'Add checkDelay to Echo email bundle jobs'
* 19:07 AaronSchulz: All jobs finished copying into redis
* 19:07 sbernardin: taking down ms-be9 for hw replacement
* 19:02 LeslieCarr: uninstalling nagios from spence
* 18:51 mutante: PXE booting emery into installer
* 18:48 ottomata: emery going down for upgrade to precise
* 18:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Turn Echo off on testwiki'
* 17:35 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Moved remaining jobs to redis'
* 17:24 sbernardin: taking down db11 for decommissioning
* 17:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
* 17:15 cmjohnson1: europium power disconnected during routine data center work
* 13:14 logmsgbot: reedy synchronized wmf-config/  'Disable ClickTracking :D'
* 13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
* 10:10 hashar: Jenkins is back up
* 09:23 hashar: restarting Jenkins to get the git verbose mode :(
* 08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone {{gerrit|58502}}. That should help with I/O starvation on gallium.
* 08:40 Nikerabbit: translation memory database rebuild finished
* 08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
* 08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/  'to master'
* 07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php  'Revert highlight.php recent modifications {{bug|47112}} {{gerrit|58664}}'
* 07:02 hashar: Put a live hack on fenari for {{bug|47112}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
* 01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them
 
== April 10 ==
* 23:31 LeslieCarr: deactivated XO ipv6 transit
* 22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
* 22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'moved refreshLinks to redis.'
* 22:20 ^demon|sick: restarting gerrit, was hung
* 22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
* 22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
* 21:59 notpeter: gracefulling apache on stafford
* 21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Moved htmlCacheUpdate jobs to redis'
* 21:19 ottomata: added python-statsd to apt repo for hashar
* 21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
* 21:07 ottomata: added python-voluptuous to apt repo (for hashar)
* 20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
* 20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Moved async upload jobs to redis'
* 19:13 notpeter: running puppetd --enable on all nodes via salt
* 19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
* 18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php  'Enabled use of redis for null test jobs'
* 18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
* 17:45 RobH: authdns-update to rename db1012 to stat1002
* 17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php  'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
* 17:24 LeslieCarr: restart unresponsive cp1037
* 17:14 RobH: authdns-update
* 15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
* 15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Nostalgia ext'
* 15:40 logmsgbot: demon synchronized wmf-config/extension-list  'Nostalgia ext'
* 15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
* 14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 14:19 logmsgbot: maxsem synchronized wmf-config  'Enable $wgMFVaryResources on test'
* 14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
* 12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
* 07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js  'Ia54dd738b3ce0995fa'
* 05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
* 05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
* 05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
* 04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
* 00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
* 00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'https://gerrit.wikimedia.org/r/#/c/58436/'
* 00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'https://gerrit.wikimedia.org/r/#/c/58436/'
 
== April 9 ==
* 23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
* 23:21 notpeter: running upgrades on db56 and rebooting
* 23:19 notpeter: running upgrades on db64 and rebooting
* 22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
* 22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
* 22:36 logmsgbot: reedy synchronized wmf-config/
* 22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
* 22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job.  No idea why but investigating.
* 19:56 Ryan_Lane: installed Nuke on wikitech
* 18:11 K4-713: updated payments anti-fraud rules
* 13:07 hashar: jenkins plugin upgrade is tracked in {{bug|47040}}. Luckily git plugin has been fixed just in time ;-D  Just need to migrate the config history.
* 12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
* 10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
* 08:54 hashar: Jenkins tests for mediawiki/core are broken ({{bug|47031}}) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change {{bug| 46723}} :/
* 08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
* 08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php  'Narayam and WebFonts'
* 05:42 Tim: on mw1149: enabled apache access log
* 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr  9 02:30:59 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr  9 02:18:53 UTC 2013
* 00:26 ^demon: jenkins is back. java gc took awhile.
* 00:19 ^demon: jenkins is hung, looking into it.
* 00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd
 
== April 8 ==
* 23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor  'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
* 23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor  'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
* 22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'Changing  from emery to vanadium'
* 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php  'Backported Poolcounter convenience class'
* 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php  'Backported Poolcounter convenience class'
* 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php  'Enable poolcounter for search'
* 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php  'Enable poolcounter for search'
* 21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php  'Deploying Poolcounter updates'
* 21:17 notpeter: temp stopping slave on db55
* 20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php  'Bumped file journal ttl to a year'
* 20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
* 20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs  'deployed c61688baef682b7d97068fefdcc399256786a387'
* 19:08 ^demon: gerrit back
* 19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
* 19:00 ottomata: removing some unused filters from emery's udp2log instance
* 18:34 Jeff_Green: deployed 1.22 on payments
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
* 17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'adding db1058 to s1 at full weight'
* 17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'adding db1058 to s1 at a low warmup rate'
* 11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
* 11:53 ^demon: bringing gerrit down for urgent update
* 10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
* 07:48 hashar: restarting Zuul for demo purposes :-)
* 04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
* 03:55 Tim: on all apaches: upgrading libpoppler
* 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr  8 02:25:07 UTC 2013
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr  8 02:18:18 UTC 2013
 
== April 7 ==
* 22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:31 logmsgbot: reedy synchronized wmf-config/  'Kill readerfeedback config, it aint coming back'
* 22:26 logmsgbot: reedy synchronized wmf-config/
* 22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
* 11:35 ^demon: restarting gerrit, stream-events seized up again.
* 07:26 hashar: gallium: restarted Zuul just to be sure.  Gerrit on manganese is stuck again and need to be restarted {{bug|46917}}
* 02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr  7 02:17:09 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr  7 02:11:21 UTC 2013
 
== April 6 ==
* 12:27 ^demon: also restarted zuul
* 12:19 ^demon: restarting gerrit again, stream-events halted like friday.
* 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr  6 02:22:48 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr  6 02:11:56 UTC 2013
* 01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
* 00:52 K4-713: adjusted anti-fraud rules on payments cluster
 
== April 5 ==
* 23:11 binasher: running iwlinks index migration on s4 (commons)
* 21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1021 to full weight, pulling db1023'
* 21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1021 at a low weight'
* 20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1021 for upgrade'
* 19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
* 18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php  's1: moving watchlist to db1049, pulling db1043, adding db1056'
* 18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
* 18:42 notpeter: restarting gerrit
* 18:22 logmsgbot: reedy synchronized php-1.21wmf12  'Killing active users in 1.21wmf12'
* 09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul.  It again receives events.
* 09:33 hashar: restarted Zuul
* 09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
* 08:58 hashar: for Jenkins / Gerrit issue see {{bug|46917}}
* 08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
* 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr  5 02:23:11 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr  5 02:12:34 UTC 2013
* 00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
* 00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
* 00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
* 00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
* 00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
* 00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
* 00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
* 00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list  'Remove E3Experiments and LastModified'
 
== April 4 ==
* 23:51 logmsgbot: demon synchronized wmf-config/extension-list  'Remove e3 since it was deleted'
* 23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
* 23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1043'
* 23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
* 23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1043'
* 23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1043'
* 23:25 binasher: shutting down mysql on db1043, preparing to reboot
* 23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1043'
* 23:17 mutante: wikinews.com now redirects to wikinews.org
* 23:11 mutante: DNS update - adding wikinews.com
* 23:08 logmsgbot: dzahn gracefulled all apaches
* 22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Re-enable Echo on testwiki'
* 22:49 mutante: running updateSpecialPages on cswikinews per request
* 22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Disable Echo on testwiki'
* 22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  'Bump Echo EventLogging schema for new notification types.'
* 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/  'GuidedTour 1.22wmf1 deployment for E3'
* 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/  'GettingStarted 1.22wmf1 deployment for E3'
* 22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/  'GuidedTour 1.21wmf12 deployment for E3'
* 22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/  'GettingStarted deployment for E3'
* 20:21 LeslieCarr: stopped cp3001 varnish
* 20:18 LeslieCarr: restarting varnish on cp3001
* 20:15 LeslieCarr: restarted udpmcast on hooft
* 20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
* 19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/57525/'
* 19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/57525/'
* 19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
* 19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/57528'
* 19:01 binasher: resuming iwlinks migrations on s3
* 17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master'
* 17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php  'Updating ArticleFeedbackv5 frwiki config'
* 16:16 paravoid: upgrading image & video scalers for libav
* 15:55 ottomata: now generating webstatscollector dumps on gadolinium.  snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
* 15:38 K4-713: adjusted anti-fraud rules on the payments cluster
* 13:26 ^demon: running jgit gc on all repos
* 05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
* 04:16 Ryan_Lane: opendj upgrade on virt1000 complete
* 03:56 Ryan_Lane: upgrading opendj on virt1000
* 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr  4 02:21:01 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr  4 02:13:31 UTC 2013
 
== April 3 ==
* 23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
* 22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
* 21:32 cmjohnson1: Running DNS update
* 20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
* 18:54 mutante: DNS update - also adding softwarewikipedia.org
* 18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
* 18:43 mutante: restarting pdns on ns1
* 18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
* 18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
* 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
* 18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
* 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
* 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
* 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
* 17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php  'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
* 17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php  'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
* 16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
* 16:34 cmjohnson1: authdns update
* 14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php  'setting weight on db1028 to 400'
* 13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
* 13:10 logmsgbot: reedy synchronized wmf-config/
* 12:20 mark: Rebooting cp3003 for RAID reconfiguration
* 10:43 mark: Deactivated AS6908 peering on cr2-knams
* 06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
* 03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr  3 02:31:44 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr  3 02:19:30 UTC 2013
* 01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
* 00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
* 00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit
 
== April 2 ==
* 23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php  '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
* 23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php  '(Ibc2633f1c) : 10000 => 5000.'
* 23:25 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving all search traffic back to eqiad'
* 23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php  'Fix for API list=recentchanges rccontinue'
* 23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php  'Fix for API list=recentchanges rccontinue'
* 23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php  'https://gerrit.wikimedia.org/r/#/c/57212/'
* 22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php  'https://gerrit.wikimedia.org/r/57208'
* 22:39 logmsgbot: maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/56696/'
* 22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
* 22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
* 22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
* 22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend  'touch'
* 22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
* 22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
* 21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
* 21:38 logmsgbot: reedy synchronized wmf-config/
* 21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'returning db1028'
* 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php  'https://gerrit.wikimedia.org/r/#/c/49071/'
* 20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/57078/'
* 20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'moving s1 watchlist to db1052, db1043 to full weight'
* 20:09 Ryan_Lane: added marc to the ops ldap group
* 20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php  'pulling db1001'
* 19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php  'adding db1028 back'
* 18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
* 18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
* 18:39 binasher: completed image img_media_mime migration on all projects
* 18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
* 17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
* 17:34 cmjohnson1: correction db1028
* 17:34 cmjohnson1: removing disk 0 from db1048 to replace
* 17:28 Jeff_Green: disable fundraising banner log rotation on locke
* 17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php  'adding db1001 back to production removing db1028'
* 17:04 cmjohnson1: setting weight to 100 on db1001
* 15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Message for ancient skin users'
* 15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
* 15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
* 15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr  2 15:12:45 UTC 2013
* 15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr  2 15:08:37 UTC 2013
* 14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php  '8th time is the charm'
* 14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'I hate l10nupdate'
* 14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php  'Notice for users of disabled skins'
* 14:24 paravoid: deploying squid config, diverting all of upload to swift
* 04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr  2 02:17:33 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr  2 02:10:24 UTC 2013
* 01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
* 00:44 logmsgbot: reedy synchronized wmf-config/
* 00:43 binasher: now running the image img_media_mime migration on commons (the big one)
* 00:41 mutante: restarting lucene on all pool4 servers (one by one)
* 00:36 mutante: creating search index for transitionteamwiki
* 00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
* 00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb  'Updating 1.22wmf1 interwiki cache'
* 00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb  'Updating 1.21wmf12 interwiki cache'
* 00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
 
== April 1 ==
* 23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
* 23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:40 mutante: creating swift containers for private wiki
* 23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb  'Updating interwiki cache'
* 23:37 mutante: dumpInterwiki.php to update interwiki cache
* 23:36 logmsgbot: py synchronized wmf-config/lucene-production.php  'moving all search traffic to pmtpa temporarily'
* 23:33 mutante: adding transitionteamwiki to private.dblist
* 23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
* 23:21 logmsgbot: dzahn s