Revision history of "Release Engineering/SAL"

Jump to navigation Jump to search

Diff selection: Mark the radio boxes of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | older 500) (20 | 50 | 100 | 250 | 500)
  • curprev 09:10, 20 May 2017imported>Stashbot 194,809 bytes +146 greg-g: executers are running again
  • curprev 19:05, 19 May 2017imported>Stashbot 194,663 bytes +739 mutante: fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab. remove again from instance level for phab-01
  • curprev 10:46, 15 May 2017imported>Stashbot 193,924 bytes +163 addshore: enabled beta-code-update-eqiad for some testing
  • curprev 20:31, 13 May 2017imported>Stashbot 193,761 bytes +158 bd808: Deleted stuck mediawiki-core-doxygen-publish job. Jenkins had it marked for a particular nodepool instance that was offline.
  • curprev 13:12, 12 May 2017imported>Stashbot 193,603 bytes +136 hashar: Trying to refresh Nodepool Jessie image. Should get HHVM pinned to 'experimental' component => 3.12.x
  • curprev 20:43, 11 May 2017imported>Stashbot 193,467 bytes +303 hashar: nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit. Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. T165074
  • curprev 20:28, 10 May 2017imported>Stashbot 193,164 bytes +1,666 bearND: Update mobileapps to 75b135e
  • curprev 12:44, 9 May 2017imported>Stashbot 191,498 bytes +248 hashar: deployment-ircd upgrading puppet 3.7.2 => 3.8.5
  • curprev 21:42, 8 May 2017imported>Stashbot 191,250 bytes +88 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/351131/
  • curprev 00:57, 8 May 2017imported>Stashbot 191,162 bytes +99 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/351130
  • curprev 01:16, 6 May 2017imported>Stashbot 191,063 bytes +634 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/352154
  • curprev 10:47, 4 May 2017imported>Stashbot 190,429 bytes +1,197 hashar: puppet ca destroy deployment-zookeeper01.eqiad.wmflabs
  • curprev 17:39, 3 May 2017imported>Stashbot 189,232 bytes +564 mdholloway: (this concerns integration-slave-jessie-android)
  • curprev 21:14, 2 May 2017imported>Stashbot 188,668 bytes +897 hashar: Manually cancelled a few mediawiki-core-jsduck-publish and mediawiki-core-doxygen-publish job in Jenkins build queue. They seems to deadlock Jenkins somehow :(
  • curprev 20:39, 1 May 2017imported>Stashbot 187,771 bytes +967 hashar: Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ )
  • curprev 21:57, 25 April 2017imported>Stashbot 186,804 bytes +171 halfak: deployed ores cc12103
  • curprev 20:17, 22 April 2017imported>Stashbot 186,633 bytes +102 hashar: Added FlorianSW to Github organization "wikimedia" (no team though)
  • curprev 12:25, 21 April 2017imported>Stashbot 186,531 bytes +356 hashar: T104048 zuul enqueue --trigger gerrit --pipeline postmerge --project AhoCorasick --change 345433,1
  • curprev 17:09, 20 April 2017imported>Stashbot 186,175 bytes +272 elukey: reverted hack on deployment-tin (apparently no effects on the jobrunner)
  • curprev 16:34, 19 April 2017imported>Stashbot 185,903 bytes +339 hashar: deleted nodepool alien ci-jessie-wikimedia-613597
  • curprev 14:29, 18 April 2017imported>Stashbot 185,564 bytes +411 hashar: unbreaking integration puppetmaster. Broke it when upgrading the puppet package :(
  • curprev 12:29, 14 April 2017imported>Stashbot 185,153 bytes +588 hashar: Delete integration-c1 instance (32GB RAM) on labvirt1004. It was used as a workaround for T161006
  • curprev 00:59, 14 April 2017imported>Stashbot 184,565 bytes +579 Amir1: three cherry-picks failed to merge, skipped them 93dad5bec8e937ef93bdd63046b0bbbf14ad9722 92c7d0b002a02ff46a00e79d6d89fe83d5f65c17 21d60a478ffb21160049679eea235bdb1010a489
  • curprev 15:14, 12 April 2017imported>Stashbot 183,986 bytes +876 hashar: rm -fR /mnt/home/jenkins-deploy/.android/build-cache/* # T162635
  • curprev 15:59, 11 April 2017imported>Stashbot 183,110 bytes +1,626 hashar: integration-config-tox-jessie job is broken due to the JJB upgrade
  • curprev 21:17, 10 April 2017imported>Stashbot 181,484 bytes +517 hashar: marked a nodepool node online manually. The instance was up but Jenkins failed to reach it due to some SEVERE: I/O error in channel
  • curprev 16:36, 6 April 2017imported>Stashbot 180,967 bytes +258 halfak: staging ores:554ea12
  • curprev 13:34, 5 April 2017imported>Stashbot 180,709 bytes +103 ema: testing possible fix for T162035 on deployment-ms-fe01
  • curprev 21:29, 4 April 2017imported>Stashbot 180,606 bytes +544 hashar: contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy due to T157818
  • curprev 20:43, 3 April 2017imported>Stashbot 180,062 bytes +534 bearND: Update mobileapps to fdd4e31
  • curprev 09:48, 1 April 2017imported>Stashbot 179,528 bytes +96 Sagan: puppet on deployment-tin looks like it is not running properly
  • curprev 23:51, 29 March 2017imported>Stashbot 179,432 bytes +447 Krinkle: Free up space on integration-slave-jessie-1001 by removing old /srv/jenkins-workspace and /srv/pbuilder dirs
  • curprev 19:53, 28 March 2017imported>Stashbot 178,985 bytes +624 hashar: Populating package manager cache of oojs-ui-npm-run-jenkins-node-6-jessie by manually triggering a build with ZUUL_PIPELINE=postmerge T155483
  • curprev 17:02, 27 March 2017imported>Stashbot 178,361 bytes +603 ebernhardson: cherry pick https://gerrit.wikimedia.org/r/344964 to puppetmaster to test upgrade to logstash 5.x
  • curprev 10:46, 25 March 2017imported>Stashbot 177,758 bytes +237 Amir1: deleting deployment-ores-redis (T160762)
  • curprev 21:34, 24 March 2017imported>Stashbot 177,521 bytes +95 Amir1: launching deployment-ores-redis-02 (T160762)
  • curprev 16:07, 23 March 2017imported>Stashbot 177,426 bytes +323 mobrovac: restbase deploying 752ca4b7
  • curprev 09:45, 22 March 2017imported>Stashbot 177,103 bytes +350 hashar: beta: purging all Linux kernel from Swift instances
  • curprev 16:47, 21 March 2017imported>Stashbot 176,753 bytes +644 halfak: halfak@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall (T160762)
  • curprev 20:51, 20 March 2017imported>Stashbot 176,109 bytes +356 andrewbogott: migrating deployment-urldownloader to labvirt1013
  • curprev 14:08, 17 March 2017imported>Stashbot 175,753 bytes +147 hashar: salt -v '*precise*' cmd.run 'puppet agent --disable "Pending shutdown on March 20th - T158652"'
  • curprev 21:48, 16 March 2017imported>Stashbot 175,606 bytes +229 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/343113/
  • curprev 20:29, 15 March 2017imported>Stashbot 175,377 bytes +2,087 bearND: Update mobileapps to bb8fcf2
  • curprev 19:23, 14 March 2017imported>Stashbot 173,290 bytes +288 thcipriani: removed alien nodepool instance via: openstack server delete ci-jessie-wikimedia-566503
  • curprev 13:19, 13 March 2017imported>Stashbot 173,002 bytes +551 hashar: Depooled Precise instances from Jenkins T158652 leaving the instances up for now.
  • curprev 12:29, 10 March 2017imported>Stashbot 172,451 bytes +110 kart_: Beta: T159800: Update DB index for T146450
  • curprev 16:20, 9 March 2017imported>Stashbot 172,341 bytes +190 gehel: upgrading elasticsearch on deployment-prep to v5.1.2
  • curprev 22:45, 8 March 2017imported>Stashbot 172,151 bytes +121 Reedy: https://gerrit.wikimedia.org/r/#/c/341916/ cherry picked onto deployment-puppetmaster02
  • curprev 22:39, 7 March 2017imported>Stashbot 172,030 bytes +393 hashar: upgrading jenkins02.ci-staging to jenkins 2.x
  • curprev 19:04, 6 March 2017imported>Stashbot 171,637 bytes +175 addshore: mwscript sql.php --wiki=aawiki "CREATE DATABASE cognate_wiktionary"
  • curprev 20:47, 2 March 2017imported>Stashbot 171,462 bytes +1,230 hashar: deployment-prep: restarted apache/puppet master. Maybe that will fix ssh_known_hosts being emptied from time to time T159332
  • curprev 19:09, 1 March 2017imported>Stashbot 170,232 bytes +640 addshore: "mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki he wiktionary hewiktionary he.wiktionary.beta.wmflabs.org" T158628
  • curprev 01:03, 1 March 2017imported>Stashbot 169,592 bytes +384 Reedy: beta-scap-eqiad giving Host key verification failed
  • curprev 21:36, 27 February 2017imported>Stashbot 169,208 bytes +74 bearND: Update mobileapps to c924126
  • curprev 03:50, 25 February 2017imported>Stashbot 169,134 bytes +124 MaxSem: deployment-prep Deleted January logs from deployment-fluorine02, was running out of space
  • curprev 13:56, 24 February 2017imported>Stashbot 169,010 bytes +454 hashar: Log refresh Nodepool instances to deploy slave script update to be able to merge mediawiki/composer.json into vendor/composer.json 6527f49..a7728a5 https://gerrit.wikimedia.org/r/#/c/339202/ T158674
  • curprev 18:35, 23 February 2017imported>Stashbot 168,556 bytes +284 greg-g: 18:29 < chasemp> !log labnodepool1001:~# service nodepool restart
  • curprev 20:59, 22 February 2017imported>Stashbot 168,272 bytes +257 hashar: Deleted jenkins job pplint-HEAD. Fully replaced by rake / puppet-syntax gem - T154894
  • curprev 14:53, 20 February 2017imported>Stashbot 168,015 bytes +227 hashar: integration: applying role::ci::slave::saucelabs to saucelabs-01
  • curprev 14:08, 17 February 2017imported>Stashbot 167,788 bytes +160 hashar: integration: deleting "repository" instance. No time to figure out how to ship Sonatype Nexus to it. T147635
  • curprev 18:34, 16 February 2017imported>Stashbot 167,628 bytes +539 greg-g: chase restarted nodepool, the daemon crashed
  • curprev 15:47, 15 February 2017imported>Stashbot 167,089 bytes +267 hashar: Zuul reducing gate-and-submit minimum amount of changes to process from the wrong 12 down to 2. In case of repeating failures it would end up running jobs for only two jobs which would prevent cancelling jobs for up to 11 changes!
  • curprev 14:38, 14 February 2017imported>Stashbot 166,822 bytes +184 hashar: Updating castor-save publish job to properly capture composer cache on Jessie ( it is in ~/.composer/cache for some reason) T156359
  • curprev 21:26, 13 February 2017imported>Stashbot 166,638 bytes +448 bearND: Update mobileapps to 3af473f
  • curprev 22:11, 10 February 2017imported>Stashbot 166,190 bytes +522 halfak: deployed ores:a15ec90
  • curprev 22:22, 9 February 2017imported>Stashbot 165,668 bytes +369 greg-g: manually kicked off a bunch of selenium tests after tgr and Reedy fixed T157636
  • curprev 22:26, 8 February 2017imported>Stashbot 165,299 bytes +265 mdholloway: mobileapps deployed 0efa7b8 in the beta cluster
  • curprev 17:49, 7 February 2017imported>Stashbot 165,034 bytes +168 halfak: deploying ores 7c80636
  • curprev 21:31, 6 February 2017imported>Stashbot 164,866 bytes +74 bearND: Update mobileapps to 034a391
  • curprev 21:37, 4 February 2017imported>Stashbot 164,792 bytes +118 halfak: deploying ores 7c80636
  • curprev 11:09, 3 February 2017imported>Stashbot 164,674 bytes +425 hashar: beta: removed old kernels from deployment-redis02 to free up disk space
  • curprev 21:56, 2 February 2017imported>Stashbot 164,249 bytes +680 hashar: integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-amd64.cow it was not properly provisioned causing build to fail (eg lack of /etc/hosts) Running puppet to reprocvision it (poke T156651)
  • curprev 00:52, 2 February 2017imported>Stashbot 163,569 bytes +881 tgr: added mhurd as member
  • curprev 22:12, 31 January 2017imported>Stashbot 162,688 bytes +328 thcipriani: started mysql on all integration precise instances via salt -- was stopped for some reason
  • curprev 14:25, 26 January 2017imported>Stashbot 162,360 bytes +307 hashar: Created Github repo for Gerrit replication https://github.com/wikimedia/mediawiki-libs-phpstorm-stubs T153252
  • curprev 11:04, 24 January 2017imported>Stashbot 162,053 bytes +171 hashar: Deleting integration-publisher (Precise) replaced by integration-publishing (Jessie). T156064 T143349
  • curprev 23:41, 23 January 2017imported>Stashbot 161,882 bytes +569 bearND: Update mobileapps to 66ef3c2
  • curprev 21:20, 21 January 2017imported>Stashbot 161,313 bytes +417 hashar: integration: updating slave scripts for https://gerrit.wikimedia.org/r/#/c/333389/
  • curprev 10:14, 20 January 2017imported>Stashbot 160,896 bytes +485 hashar: puppet fails on "integration" labs instances due to an attempt to unmount the non existing NFS /home. Filled T155820
  • curprev 22:11, 19 January 2017imported>Stashbot 160,411 bytes +607 Krenair: added bunch of others to the same group per request. we should figure out how to make this process sane somehow
  • curprev 10:49, 18 January 2017imported>Stashbot 159,804 bytes +306 hashar: Disconnected/connected Jenkins Gearman client. The beta cluster builds had a deadlock.
  • curprev 21:47, 17 January 2017imported>Stashbot 159,498 bytes +657 urandom: deployment-prep restarting Cassandra on deployment-restbase02
  • curprev 09:33, 16 January 2017imported>Stashbot 158,841 bytes +226 hashar: integration nuked the Zuul merger path for SelectTag mw extension ( on scandium /srv/ssd/zuul/git/mediawiki/extensions/SelectTag ) Failed to merge https://gerrit.wikimedia.org/r/#/c/331974/
  • curprev 00:33, 12 January 2017imported>Stashbot 158,615 bytes +297 legoktm: deploying https://gerrit.wikimedia.org/r/331796 and https://gerrit.wikimedia.org/r/331795
  • curprev 23:07, 10 January 2017imported>Stashbot 158,318 bytes +98 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/331099
  • curprev 05:20, 8 January 2017imported>Stashbot 158,220 bytes +174 Krenair: deployment-stream: live hacked /usr/lib/python2.7/dist-packages/socketio/handler.py a bit (added apostrophes) to try to make rcstream work
  • curprev 10:17, 7 January 2017imported>Stashbot 158,046 bytes +131 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki (T139110)
  • curprev 16:31, 6 January 2017imported>Stashbot 157,915 bytes +275 hashar: Nodepool Image ci-jessie-wikimedia-1483719758 in wmflabs-eqiad is ready
  • curprev 17:20, 5 January 2017imported>Stashbot 157,640 bytes +194 hashar: Dropping puppet source from https://doc.wikimedia.org/puppetsource/ . contint1001: sudo rm -fR /srv/org/wikimedia/doc/puppetsource (T143233)
  • curprev 21:30, 4 January 2017imported>Stashbot 157,446 bytes +434 mutante: deployment-cache-text-04 - running acme-setup command to debug .. Creating CSR /etc/acme/csr/beta_wmflabs_org.pem
  • curprev 11:22, 2 January 2017imported>Stashbot 157,012 bytes +421 hashar: Nodepool Image ci-jessie-wikimedia-1483355768 in wmflabs-eqiad is ready
  • curprev 05:00, 27 December 2016imported>Stashbot 156,591 bytes +107 Amir1: deploying 5230e7d in ores beta node (T154168)
  • curprev 12:09, 26 December 2016imported>Stashbot 156,484 bytes +122 hashar: beta: restarted varnish.service and varnish-frontend.service on deployment-cache-text04
  • curprev 09:02, 24 December 2016imported>Stashbot 156,362 bytes +99 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/329038
  • curprev 12:18, 23 December 2016imported>Stashbot 156,263 bytes +83 legoktm: deploying https://gerrit.wikimedia.org/r/328886
  • curprev 22:11, 22 December 2016imported>Stashbot 156,180 bytes +90 thcipriani: disable production l10nupdate for deployment freeze
  • curprev 05:57, 21 December 2016imported>Stashbot 156,090 bytes +210 Krinkle: Jenkins "Collapsing Console Sections" for PHPUnit was broken since "-d zend.enable_gc=0" was added to phpunit.php invocation. Updated pattern in Jenkins system configuration.
  • curprev 21:21, 19 December 2016imported>Stashbot 155,880 bytes +616 andrewbogott: and also python-functools32_3.2.3.2-3~bpo8+1_all.deb
  • curprev 22:34, 16 December 2016imported>Stashbot 155,264 bytes +1,159 legoktm: deploying https://gerrit.wikimedia.org/r/327202
  • curprev 21:00, 15 December 2016imported>Stashbot 154,105 bytes +471 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/324368
  • curprev 12:38, 14 December 2016imported>Stashbot 153,634 bytes +97 elukey: created deployment-copper on deployment-prep as temporary test
  • curprev 22:52, 13 December 2016imported>Stashbot 153,537 bytes +426 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/327119
  • curprev 04:01, 9 December 2016imported>Stashbot 153,111 bytes +148 legoktm: deploying https://gerrit.wikimedia.org/r/326070
  • curprev 23:35, 8 December 2016imported>Stashbot 152,963 bytes +1,642 legoktm: deploying https://gerrit.wikimedia.org/r/326048 https://gerrit.wikimedia.org/r/326050
  • curprev 15:04, 7 December 2016imported>Stashbot 151,321 bytes +398 hashar: Image ci-trusty-wikimedia-1481122712 in wmflabs-eqiad is ready T117418
  • curprev 21:20, 6 December 2016imported>Stashbot 150,923 bytes +490 hashar: Image ci-jessie-wikimedia-1481058839 in wmflabs-eqiad is ready T113342
  • curprev 20:43, 5 December 2016imported>Stashbot 150,433 bytes +672 hashar: Image ci-jessie-wikimedia-1480969940 in wmflabs-eqiad is ready (include trendingedits::packages which explicitly define the installation of librdkafka-dev' )
  • curprev 15:25, 4 December 2016imported>Stashbot 149,761 bytes +160 Krenair: Found a git-sync-upstream cron on deployment-mx for some reason... commented for now, but wtf was this doing on a MX server?
  • curprev 23:07, 3 December 2016imported>Stashbot 149,601 bytes +190 legoktm: deploying https://gerrit.wikimedia.org/r/325132
  • curprev 14:40, 2 December 2016imported>Stashbot 149,411 bytes +145 hashar: added Tobias Gritschacher to Gerrit "integration" group so he can +2 patches on integration/* repositories \O/
  • curprev 18:20, 1 December 2016imported>Stashbot 149,266 bytes +339 elukey: removing https://gerrit.wikimedia.org/r/#/c/305536 from the puppet master via rebase -i (no-op for beta)
  • curprev 17:22, 30 November 2016imported>Stashbot 148,927 bytes +513 gehel: restart of logstash on deployment-logstash2 - upgrade to Java 8 - T151325
  • curprev 15:23, 29 November 2016imported>Stashbot 148,414 bytes +305 hashar: Image ci-jessie-wikimedia-1480432368 in wmflabs-eqiad is ready
  • curprev 09:48, 28 November 2016imported>Stashbot 148,109 bytes +393 hashar: Image ci-trusty-wikimedia-1480326016 in wmflabs-eqiad is ready
  • curprev 16:15, 26 November 2016imported>Stashbot 147,716 bytes +617 Reedy: killed /srv/jenkins-workspace/workspace/mediawiki-core-*/src and /srv/jenkins-workspace/workspace/mwext-*/src from integration slaves to get rid of borked MW dirs
  • curprev 20:50, 24 November 2016imported>Stashbot 147,099 bytes +493 hashar: make contint1001 Jenkins slave to only builds jobs with a label matching the node https://integration.wikimedia.org/ci/computer/contint1001/configure T86659
  • curprev 15:04, 23 November 2016imported>Stashbot 146,606 bytes +229 Krenair: fixed puppet on deployment-cache-text04 by manually enabling experimental apt repo, see T150660
  • curprev 19:31, 22 November 2016imported>Stashbot 146,377 bytes +360 hashar: beta: rebased puppet master
  • curprev 12:46, 21 November 2016imported>Stashbot 146,017 bytes +145 hashar: beta: Cherry picked puppet fix for udp2log https://gerrit.wikimedia.org/r/#/c/322639/ T151169
  • curprev 00:10, 19 November 2016imported>Stashbot 145,872 bytes +237 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/322370
  • curprev 22:07, 17 November 2016imported>Stashbot 145,635 bytes +259 mutante: re-enabled puppet on contint1001 after live Apache fix
  • curprev 20:53, 16 November 2016imported>Stashbot 145,376 bytes +1,808 elukey: restored apache2 config on deployment-mediawiki06
  • curprev 16:03, 15 November 2016imported>Stashbot 143,568 bytes +120 hasharAway: adding thcipriani to the labs "git" project maintained by paladox
  • curprev 08:16, 14 November 2016imported>Stashbot 143,448 bytes +94 Amir1: cherry-picking 321096/3 in beta puppetmaster
  • curprev 14:02, 12 November 2016imported>Stashbot 143,354 bytes +86 Amir1: cherry-picked gerrit change 321096/2 in puppetmaster
  • curprev 23:48, 11 November 2016imported>Stashbot 143,268 bytes +376 bd808: Updated _template/logstash on deployment-logstash2 to include change from https://gerrit.wikimedia.org/r/#/c/320441/
  • curprev 09:33, 10 November 2016imported>Stashbot 142,892 bytes +185 hashar: Image ci-jessie-wikimedia-1478770026 in wmflabs-eqiad is ready
  • curprev 20:27, 9 November 2016imported>Stashbot 142,707 bytes +284 Krenair: removed default SSH access from production host 208.80.154.135, the old gallium IP
  • curprev 10:23, 8 November 2016imported>Stashbot 142,423 bytes +141 hashar: refreshing all jenkins jobs to clear out potential live hack I made but can't remember on which jobs I did
  • curprev 14:01, 7 November 2016imported>Stashbot 142,282 bytes +128 gilles: Pointing deployment-imagescaler01.eqiad.wmflabs' puppet to puppetmaster.thumbor.eqiad.wmflabs
  • curprev 13:20, 4 November 2016imported>Stashbot 142,154 bytes +880 hashar: gerrit: created mediawiki/extensions/PageViewInfo.git and renamed user group extension-WikimediaPageViewInfo to extension-PageViewInfo T148775
  • curprev 15:05, 3 November 2016imported>Stashbot 141,274 bytes +709 Amir1: deploy 0caa589 in ores to deployment-sca03
  • curprev 22:06, 2 November 2016imported>Stashbot 140,565 bytes +344 hashar: hello stashbot
  • curprev 22:22, 1 November 2016imported>Stashbot 140,221 bytes +498 Krenair: started mysql on -db03 to hopefully pull us out of read-only mode
  • curprev 20:57, 31 October 2016imported>Stashbot 139,723 bytes +132 Krenair: moving some nodes to deployment-puppetmaster02
  • curprev 20:51, 27 October 2016imported>Stashbot 139,591 bytes +989 hashar: reboot integration-puppetmaster01
  • curprev 19:22, 25 October 2016imported>Labslogbot 138,602 bytes +283 Python PyPi mirror has some issue. Impacts all CI jobs relying on tox https://status.python.org/ (hasharAway)
  • curprev 16:19, 24 October 2016imported>Labslogbot 138,319 bytes +150 upgrading deployment-puppetmaster to puppet 3.8.5 packages (andrewbogott)
  • curprev 09:42, 21 October 2016imported>Labslogbot 138,169 bytes +80 decommission of deployment-elastic08 - T147777 (gehel)
  • curprev 23:37, 20 October 2016imported>Labslogbot 138,089 bytes +164 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/317083 (Krinkle)
  • curprev 21:13, 14 October 2016imported>Labslogbot 137,925 bytes +569 Ran START SLAVE to restart replication after columns created directly on replica were deleted. (matt_flaschen)
  • curprev 21:21, 13 October 2016imported>Labslogbot 137,356 bytes +1,438 Deleted CI slaves integration-slave-jessie-1004 integration-slave-jessie-1005 integration-slave-trusty-1013 integration-slave-trusty-1014 integration-slave-trusty-1017 integration-slave-trusty-1018 (hashar)
  • curprev 13:37, 12 October 2016imported>Labslogbot 135,918 bytes +289 upgraded memcached on deployment-memc04 to 1.4.28-1.1+wmf1 as part of a perf experiment (T129963) - rollback: wipe https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep/host/deployment-memc04, apt-get remove memcached on deployment-memc04, puppet r...
  • curprev 21:35, 11 October 2016imported>Labslogbot 135,629 bytes +550 Force pushed Zuul patchqueue 5628f95...fc6a118 HEAD -> patch-queue/debian/precise-wikimedia (hasharAway)
  • curprev 15:45, 10 October 2016imported>Labslogbot 135,079 bytes +132 deployment-prep deployment-elastic0[5-8]: reduce the number of replicas to 1 max for all indices (dcausse)
  • curprev 20:10, 7 October 2016imported>Labslogbot 134,947 bytes +213 Created repository.integration.eqiad.wmflabs to play/Test Sonatype Nexus (hashar)
  • curprev 14:45, 6 October 2016imported>Labslogbot 134,734 bytes +273 deployment-mira disarmed/rearmed keyholder in an attempt to clear a Shinken alarm (hashar)
  • curprev 19:33, 5 October 2016imported>Labslogbot 134,461 bytes +145 removing mediawiki::conftool from deployment-mediawiki04, deployment-mediawiki06, deployment-mediawiki05 (andrewbogott)
  • curprev 19:43, 4 October 2016imported>Labslogbot 134,316 bytes +540 removed contint::slave_scripts and associated files from deployment-sca01 and deployment-sca02 (andrewbogott)
  • curprev 01:14, 4 October 2016imported>Labslogbot 133,776 bytes +890 New scap command line autocompletions are now installed on deployment-tin and deployment-mira refs T142880 (twentyafterfour)
  • curprev 09:41, 1 October 2016imported>Labslogbot 132,886 bytes +137 beta: shutdown deployment-db1 and deployment-db2 . Databases have been migrated to other hosts T138778 (hashar)
  • curprev 15:43, 29 September 2016imported>Labslogbot 132,749 bytes +641 logstash-beta: refreshed the field list via https://logstash-beta.wmflabs.org/app/kibana#/settings/indices/logstash (hashar)
  • curprev 23:56, 28 September 2016imported>Labslogbot 132,108 bytes +1,698 Deleted varnish cache files on deployment-cache-upload04 to free up space, disk full (MaxSem)
  • curprev 20:11, 27 September 2016imported>Labslogbot 130,410 bytes +195 Reloading Zuul to deploy 3c3289aa1a for T143938 and T146783 (hashar)
  • curprev 23:58, 26 September 2016imported>Labslogbot 130,215 bytes +564 Started udp2log-mw on deployment-fluorine02 for T146723 (bd808)
  • curprev 20:08, 24 September 2016imported>Labslogbot 129,651 bytes +400 deployment-tin is shutdown. Replaced by Jessie deployment-tin02 (hashar)
  • curprev 20:21, 23 September 2016imported>Labslogbot 129,251 bytes +686 integration: salt -v '*trusty*' cmd.run 'service mysql start' (hashar)
  • curprev 19:29, 22 September 2016imported>Labslogbot 128,565 bytes +241 switching Jenkins slaves workspace from /mnt/jenkins-workspace to /srv/jenkins-workspace (actually the same dir/inode on the filesystem) (hasharAway)
  • curprev 18:23, 21 September 2016imported>Labslogbot 128,324 bytes +2,266 shutting down integration-puppetmaster (yuvipanda)
  • curprev 21:49, 20 September 2016imported>Labslogbot 126,058 bytes +2,278 Deleting deployment-mira02 /srv was too small. Replaced by deployment-mira (hashar)
  • curprev 22:01, 19 September 2016imported>Labslogbot 123,780 bytes +1,285 shutdown integration-puppetmaster (legoktm)
  • curprev 07:11, 17 September 2016imported>Labslogbot 122,495 bytes +83 deploying https://gerrit.wikimedia.org/r/311024 (legoktm)
  • curprev 21:03, 16 September 2016imported>Labslogbot 122,412 bytes +1,025 deployment-tin did a git gc on /srv/deployment/ores That freed up disk space and cleared an alarm on co master mira02 (hashar)
  • curprev 22:29, 15 September 2016imported>Labslogbot 121,387 bytes +1,969 sudo salt '*precise*' cmd.run 'service mysql start', all mysql's are down (legoktm)
  • curprev 21:37, 14 September 2016imported>Labslogbot 119,418 bytes +537 integration: setting "ulimit -c 2097152" on all slaves due to Zend PHP segfaulting T142158 (hashar)
  • curprev 17:02, 13 September 2016imported>Labslogbot 118,881 bytes +1,525 re-enabling beta cluster jenkins jobs following maintenance window (marxarelli)
  • curprev 14:41, 12 September 2016imported>Labslogbot 117,356 bytes +588 applied base::firewall, beta::deployaccess, mediawiki::conftool, role::mediawiki::appserver to deployment-mediawiki04.deployment-prep.eqiad.wmflabs (Debian jessie instance) - T144006 (elukey)
  • curprev 18:45, 11 September 2016imported>Labslogbot 116,768 bytes +83 deploying https://gerrit.wikimedia.org/r/309829 (legoktm)
  • curprev 20:53, 9 September 2016imported>Labslogbot 116,685 bytes +399 testing scap 3.2.5-1 on beta cluster (thcipriani)
  • curprev 21:33, 8 September 2016imported>Labslogbot 116,286 bytes +1,462 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/309413 " Inject PHP_BIN=php5 for php53 jobs" (hashar)
  • curprev 20:44, 7 September 2016imported>Labslogbot 114,824 bytes +1,960 Re-enabled beta-code-update-eqiad . (matt_flaschen)
  • curprev 13:58, 6 September 2016imported>Labslogbot 112,864 bytes +798 Qunit jobs should be all fine again now. T144802 (hashar)
  • curprev 14:08, 5 September 2016imported>Labslogbot 112,066 bytes +130 Refreshing Nodepool base images for Trusty and Jessie. Managed to build new ones after T143769 (hashar)
  • curprev 20:36, 2 September 2016imported>Labslogbot 111,936 bytes +177 deploying https://gerrit.wikimedia.org/r/308227 (legoktm)
  • curprev 19:00, 1 September 2016imported>Labslogbot 111,759 bytes +670 T130861: Restarting Cassandra on deployment-restbase0[1-2] (urandom)
  • curprev 23:40, 31 August 2016imported>Labslogbot 111,089 bytes +986 forced puppet run on deployment-salt02. Had not run automatically for 8 hours (bd808)
  • curprev 23:31, 30 August 2016imported>Labslogbot 110,103 bytes +2,327 cherry-picking https://gerrit.wikimedia.org/r/#/c/307656/ fixed puppet on the elasticsearch machines! (yuvipanda)
  • curprev 07:56, 29 August 2016imported>Labslogbot 107,776 bytes +226 hard rebooting integration-slave-trusty-1012 via horizon and restarting puppet manually (hashar)
  • curprev 20:51, 27 August 2016imported>Labslogbot 107,550 bytes +412 integration: tweak sudo policy for jenkins-deploy running cowbuilder: env_keep+=DEB_BUILD_OPTIONS (hashar)
  • curprev 22:29, 26 August 2016imported>Labslogbot 107,138 bytes +551 deploying https://gerrit.wikimedia.org/r/307025 (legoktm)
  • curprev 00:28, 26 August 2016imported>Labslogbot 106,587 bytes +795 restarted puppetmaster service on deployment-puppetmaster (twentyafterfour)
  • curprev 15:14, 24 August 2016imported>Labslogbot 105,792 bytes +488 deploying ores d00171 (halfak)
  • curprev 18:21, 23 August 2016imported>Labslogbot 105,304 bytes +409 deploying https://gerrit.wikimedia.org/r/306257 (legoktm)
  • curprev 23:40, 22 August 2016imported>Labslogbot 104,895 bytes +72 updating slave_scripts on all slaves (legoktm)
  • curprev 22:03, 18 August 2016imported>Labslogbot 104,823 bytes +334 deployment-fluorine02: Hack 'datasets:x:10003:997::/home/datasets:/bin/bash' into /etc/passwd for T117028 (bd808)
  • curprev 22:28, 17 August 2016imported>Labslogbot 104,489 bytes +524 deploying https://gerrit.wikimedia.org/r/305408 (legoktm)
  • curprev 23:10, 16 August 2016imported>Labslogbot 103,965 bytes +350 max_servers at 6, seeing 6 allocated instances, still seeing 403 already used 10 of 10 instances :(( (thcipriani)
  • curprev 21:30, 15 August 2016imported>Labslogbot 103,615 bytes +710 update scap on beta to 3.2.3-1 bugfix release (thcipriani)
  • curprev 23:16, 13 August 2016imported>Labslogbot 102,905 bytes +61 cherry-picking 304678/1 into the puppetmaster (Amir1)
  • curprev 00:08, 13 August 2016imported>Labslogbot 102,844 bytes +311 deploying https://gerrit.wikimedia.org/r/304588 (legoktm)
  • curprev 23:56, 10 August 2016imported>Labslogbot 102,533 bytes +347 deploying https://gerrit.wikimedia.org/r/304149 (legoktm)
  • curprev 23:21, 9 August 2016imported>Labslogbot 102,186 bytes +412 ladsgroup@deployment-sca03:~$ sudo service celery-ores-worker restart (Amir1)
  • curprev 00:56, 9 August 2016imported>Labslogbot 101,774 bytes +341 deploying https://gerrit.wikimedia.org/r/303726 (legoktm)
  • curprev 12:01, 7 August 2016imported>Labslogbot 101,433 bytes +604 Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) (hashar)
  • curprev 12:31, 6 August 2016imported>Labslogbot 100,829 bytes +240 restarting uwsgi-ores and celery-ores-worker in deployment-sca03 (Amir1)
  • curprev 17:54, 5 August 2016imported>Labslogbot 100,589 bytes +251 Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/3 for testing (bd808)
  • curprev 00:41, 5 August 2016imported>Labslogbot 100,338 bytes +421 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/303113 (Krinkle)
  • curprev 20:28, 1 August 2016imported>Labslogbot 99,917 bytes +170 restarting deployment-ms-be01, not responding to ssh, mw-fe01 requests timing out (thcipriani)
  • curprev 23:27, 29 July 2016imported>Labslogbot 99,747 bytes +842 Rebooting deployment-logstash2; Console showed hung task timeouts (P3606) (bd808)
  • curprev 21:46, 28 July 2016imported>Labslogbot 98,905 bytes +580 xintegration: change sudo policy for jenkins-deploy to help on T141538 : env_keep+=WORKSPACE (hashar_)
  • curprev 16:13, 27 July 2016imported>Labslogbot 98,325 bytes +878 salt -v '*slave-trusty*' cmd.run 'service mysql start' ( was missing on integration-slave-trusty-1011.integration.eqiad.wmflabs ) (hashar)
  • curprev 20:13, 26 July 2016imported>Labslogbot 97,447 bytes +795 Zuul deployed https://gerrit.wikimedia.org/r/301093 which adds 'debian-glue' job on all of operations/debs/ repos (hashar)
  • curprev 01:08, 26 July 2016imported>Labslogbot 96,652 bytes +341 deployed ores a291da1 in sca03, ores-beta.wmflabs.org works as expected (Amir1)
  • curprev 20:06, 23 July 2016imported>Labslogbot 96,311 bytes +316 Cleanup jobrunner01 logs via -- sudo logrotate --force /etc/logrotate.d/mediawiki_jobrunner (bd808)
  • curprev 20:26, 22 July 2016imported>Labslogbot 95,995 bytes +1,322 T141114 upgraded jenkins-debian-glue from v0.13.0 to v0.17.0 on integration-slave-jessie-1001 and integration-slave-jessie-1002 (hashar)
  • curprev 16:12, 21 July 2016imported>Labslogbot 94,673 bytes +640 Updated our JJB fork cherry picking f74501e781f by madhuvishy. Was made to support the maven release plugin. Branch bump is 10f2bcd..6fcaf39 (hashar)
  • curprev 00:13, 21 July 2016imported>Labslogbot 94,033 bytes +1,279 Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/ to deployment-puppetmaster so wdqs nginx log parsing can be tested (bd808)
  • curprev 20:56, 18 July 2016imported>Labslogbot 92,754 bytes +385 Deleted deployment-fluorine:/srv/mw-log/archive/*-201605* freed 30 GB (thcipriani)
  • curprev 23:19, 16 July 2016imported>Labslogbot 92,369 bytes +52 testing morebots (paladox)
  • curprev 08:34, 15 July 2016imported>Labslogbot 92,317 bytes +132 Unpooling integration-slave-precise-1002 will use it as a zuul-server test instance temporarily (hashar)
  • curprev 18:54, 14 July 2016imported>Labslogbot 92,185 bytes +316 deployment-prep manually edited elasticsearch.yml on deployment-elastic05 and restarted to get it listening on eth0. Still looking into why puppet wrote out wrong config file (ebernhardson)
  • curprev 16:05, 13 July 2016imported>Labslogbot 91,869 bytes +512 Installing Cassandra 2.2.6-wmf1 on deployment-restbase0[1-2].deployment-prep.eqiad.wmflabs : T126629 (urandom)
  • curprev 20:29, 12 July 2016imported>Labslogbot 91,357 bytes +643 integration: force running unattended upgrade on all instances: salt --batch 4 -v '*' cmd.run 'unattended-upgrade' . That upgrades diamond and hhvm among others. imagemagick-common has a prompt though (hashar)
  • curprev 14:24, 11 July 2016imported>Labslogbot 90,714 bytes +579 Removing ZeroMQ config from the Jenkins jobs. It is now enabled globally. T139923 (hashar)
  • curprev 18:57, 9 July 2016imported>Labslogbot 90,135 bytes +223 deploying https://gerrit.wikimedia.org/r/297731 and https://gerrit.wikimedia.org/r/298142 (legoktm)
  • curprev 16:08, 8 July 2016imported>Labslogbot 89,912 bytes +817 scandium: git -C /srv/ssd/zuul/git/mediawiki/services/graphoid remote set-head origin --auto (hashar)
  • curprev 21:41, 7 July 2016imported>Labslogbot 89,095 bytes +283 Chowned php-master/vendor back to jenkins-deploy (MaxSem)
  • curprev 01:14, 7 July 2016imported>Labslogbot 88,812 bytes +1,085 Restarted logstash on deployment-logstash2 (bd808)
  • curprev 21:54, 5 July 2016imported>Labslogbot 87,727 bytes +201 CI has drained the gate-and-submit queue (hasharAway)
  • curprev 18:58, 4 July 2016imported>Labslogbot 87,526 bytes +170 Upgrading arcanist on permanent CI slaves since xhpast was broken T137770 (hashar)
  • curprev 13:10, 3 July 2016imported>Labslogbot 87,356 bytes +238 phabricator Update phab-01 and phab-05 (phab-02) and phab-03 to fix a security bug in phabricator (Did the update last night but forgot to log it) (paladox)
  • curprev 13:38, 2 July 2016imported>Labslogbot 87,118 bytes +74 reloading zuul for 15127b2..7e6a2e2 (jzerebecki)
  • curprev 10:31, 30 June 2016imported>Labslogbot 87,044 bytes +492 Deleting integration-slave-trusty-1015 . Can not bring up mysql T138074 and the ssh slave connection would not hold anyway. Must be broken somehow (hashar)
  • curprev 21:33, 28 June 2016imported>Labslogbot 86,552 bytes +96 deploying ores beec291 (halfak)
  • curprev 22:32, 27 June 2016imported>Labslogbot 86,456 bytes +680 deployment-prep deployed gerrit.wikimedia.org/r/296279 to puppetmaster to test kibana4 role (eberhardson)
  • curprev 15:04, 24 June 2016imported>Labslogbot 85,776 bytes +372 switch apps-android-wikimedia-* jobs to Jessie T138506 (hashar)
  • curprev 13:58, 23 June 2016imported>Labslogbot 85,404 bytes +391 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/295691 (Krinkle)
  • curprev 10:35, 17 June 2016imported>Labslogbot 85,013 bytes +181 offlined integration-slave-trusty-1015 T138074 (jzerebecki)
  • curprev 14:22, 14 June 2016imported>Labslogbot 84,832 bytes +275 T136971 on tin MediaWiki 1.28.0-wmf.6, from 1.28.0-wmf.6, successfully checked out. Applying security patches (hashar)
  • curprev 21:11, 13 June 2016imported>Labslogbot 84,557 bytes +898 https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1015/ put offline. Jenkins cant ssh / pool it for some reason (hashar)
  • curprev 15:49, 10 June 2016imported>Labslogbot 83,659 bytes +779 reloading zuul for 8c048fb..272d1ec (jzerebecki)
  • curprev 18:49, 9 June 2016imported>Labslogbot 82,880 bytes +372 restarting nutcracker on deployment-mediawiki02 (hashar)
  • curprev 02:56, 8 June 2016imported>Labslogbot 82,508 bytes +144 / on gallium is read-only (legoktm)
  • curprev 19:28, 7 June 2016imported>Labslogbot 82,364 bytes +449 Nodepool has troubles spawning instances probably due to on going (?) labs maintenance (hashar)
  • curprev 19:26, 6 June 2016imported>Labslogbot 81,915 bytes +308 Regenerating Nodepool snapshots for Trusty and Jessie (hasharAway)
  • curprev 00:09, 4 June 2016imported>Labslogbot 81,607 bytes +562 krinkle@integration-slave-trusty-1017:~$ sudo rm -rf /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/Babel (T86730) (Krinkle)
  • curprev 12:10, 2 June 2016imported>Labslogbot 81,045 bytes +129 Upgraded Zuul upstream code being 66c8e52..30a433b package is 2.1.0-151-g30a433b-wmf1precise1 (hashar)
  • curprev 17:49, 1 June 2016imported>Labslogbot 80,916 bytes +888 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/292186 (Krinkle)
  • curprev 20:24, 31 May 2016imported>Labslogbot 80,028 bytes +101 Reloading zuul to pick up I58f878f3fd19dfa21a46a52464575cb06aacbb22 (bd808)
  • curprev 18:39, 30 May 2016imported>Labslogbot 79,927 bytes +624 Upgraded our Jenkins Job Builder fork to 1.5.0 + a couple of cherry picks: cd63874...10f2bcd (hashar)
  • curprev 04:43, 28 May 2016imported>Labslogbot 79,303 bytes +95 depooling integration-slave-trusty-1015 to profile phpunit runs (ori)
  • curprev 19:29, 27 May 2016imported>Labslogbot 79,208 bytes +292 Refreshed Nodepool images (hasharAway)
  • curprev 22:23, 26 May 2016imported>Labslogbot 78,916 bytes +588 salt -v '*trusty*' cmd.run 'puppet agent --disable "Chromium needs to be v49. See T136188"' (hashar)
  • curprev 22:38, 25 May 2016imported>Labslogbot 78,328 bytes +1,025 running puppet agent manually on sca01 (Amir1)
  • curprev 07:03, 24 May 2016imported>Labslogbot 77,303 bytes +75 rebooting deployment-tin, can't log in (mobrovac)
  • curprev 19:35, 23 May 2016imported>Labslogbot 77,228 bytes +1,024 killed all mysqld process on Trusty CI slaves (hashar)
  • curprev 20:30, 20 May 2016imported>Labslogbot 76,204 bytes +151 Killing https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/43608/ which has been running for 5 hours (bd808)
  • curprev 16:47, 19 May 2016imported>Labslogbot 76,053 bytes +778 deployment-tin jenkins worker seems to be back online after [https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update some prodding] (thcipriani)
  • curprev 13:16, 18 May 2016imported>Labslogbot 75,275 bytes +413 deploying a05e830 to ores nodes (sca01 and ores-web) (Amir1)
  • curprev 14:39, 13 May 2016imported>Labslogbot 74,862 bytes +601 remove shadow l10nupdate user from deployment-tin and mira in beta (thcipriani)
  • curprev 22:53, 12 May 2016imported>Labslogbot 74,261 bytes +86 Started dead mysql on integration-slave-precise-1011 (bd808)
  • curprev 21:05, 11 May 2016imported>Labslogbot 74,175 bytes +1,452 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/288128 #T134946 (hashar)
  • curprev 19:17, 10 May 2016imported>Labslogbot 72,723 bytes +1,609 beta / CI purging old Linux kernels: salt -v '*' cmd.run 'dpkg -l|grep ^rc|awk "{ print \$2 }"|grep linux-image|xargs dpkg --purge' (hashar)
  • curprev 20:58, 9 May 2016imported>Labslogbot 71,114 bytes +890 Unbroke puppet on integration-raita.integration.eqiad.wmflabs . Puppet was blocked because role::ci::raita was no more. Fixed by rebasing https://gerrit.wikimedia.org/r/#/c/208024 T115330 (hashar)
  • curprev 19:45, 6 May 2016imported>Labslogbot 70,224 bytes +172 Restart cassandra-metrics-collector on deployment-restbase0[1-2] (urandom)
  • curprev 22:09, 5 May 2016imported>Labslogbot 70,052 bytes +139 Promoted Yurik and Jgirault to sysops on beta enwiki. Through shell because logging in is broken for me. (MaxSem)
  • curprev 21:28, 4 May 2016imported>Labslogbot 69,913 bytes +1,227 deployed puppet FQDN domain patch for OCG: https://gerrit.wikimedia.org/r/286068 and restarted ocg on deployment-pdf0[12] (cscott)
  • curprev 23:21, 3 May 2016imported>Labslogbot 68,686 bytes +1,748 Changed "Maximum Number of Retries" for ssh agent launch in jenkins for deployment-tin from "0" to "10" (bd808)
  • curprev 21:27, 2 May 2016imported>Labslogbot 66,938 bytes +1,284 updated OCG to version b775e612520f9cd4acaea42226bcf34df07439f7 (cscott)
  • curprev 18:31, 30 April 2016imported>Labslogbot 65,654 bytes +136 deploying d4f63a3 from github.com/wiki-ai/ores-wikimedia-config into targets in beta cluster via scap3 (Amir1)
  • curprev 16:37, 29 April 2016imported>Labslogbot 65,518 bytes +329 restarting zuul for 4e9d180..ebb191f (jzerebecki)
  • curprev 22:03, 28 April 2016imported>Labslogbot 65,189 bytes +1,833 deployment-restbase01 upgrade to 2.2.6 complete : T126629 (urandom)
  • curprev 23:57, 27 April 2016imported>Labslogbot 63,356 bytes +1,474 nodepool instances running again after an openstack rabbitmq restart by andrewbogott (thcipriani)
  • curprev 20:45, 26 April 2016imported>Labslogbot 61,882 bytes +875 Regenerating Nodepool Jessie snapshot to include composer and HHVM | T128092 (hashar)
  • curprev 00:46, 26 April 2016imported>Labslogbot 61,007 bytes +1,084 temporary keyholder fix in place in beta (thcipriani)
  • curprev 17:14, 24 April 2016imported>Labslogbot 59,923 bytes +65 reloading e06f1fe..672fc84 (jzerebecki)
  • curprev 18:13, 22 April 2016imported>Labslogbot 59,858 bytes +148 deploying https://gerrit.wikimedia.org/r/284841 (legoktm)
  • curprev 19:07, 21 April 2016imported>Labslogbot 59,710 bytes +234 scap version testing should be done, puppet should no longer be disabled on hosts (thcipriani)
  • curprev 22:28, 20 April 2016imported>Labslogbot 59,476 bytes +541 rolling back scap version in beta, legit failure :( (thcipriani)
  • curprev 01:01, 17 April 2016imported>Labslogbot 58,935 bytes +223 deploying https://gerrit.wikimedia.org/r/283837 (legoktm)
  • curprev 01:48, 13 April 2016imported>Labslogbot 58,712 bytes +83 deploying https://gerrit.wikimedia.org/r/282952 (legoktm)
  • curprev 19:47, 12 April 2016imported>Labslogbot 58,629 bytes +869 Cleaned up large hhbc cache file on deployment-medaiwiki03 via `sudo service hhvm stop; sudo rm /var/cache/hhvm/fcgi.hhbc.sq3; sudo service hhvm start` (bd808)
  • curprev 21:23, 11 April 2016imported>Labslogbot 57,760 bytes +458 deployed and reverted oath (csteipp)
  • curprev 22:46, 8 April 2016imported>Labslogbot 57,302 bytes +563 Created blobs1 table for all wiki DBs on Beta Cluster (matt_flaschen)
  • curprev 20:16, 7 April 2016imported>Labslogbot 56,739 bytes +217 deployment-mediawiki02.deployment-prep.eqiad.wmflabs , cleared up random left over stuff / big logs etc (hashar)
  • curprev 23:56, 5 April 2016imported>Labslogbot 56,522 bytes +561 Removed cherry-pick and rebased /var/lib/git/operations/puppet on integration-puppetmaster after merge of https://gerrit.wikimedia.org/r/#/c/281706/ (marxarelli)
  • curprev 17:30, 4 April 2016imported>Labslogbot 55,961 bytes +400 Phabricator going down in about 10 minutes to hopefully address the overheating issue: T131742 (greg-g)
  • curprev 07:02, 3 April 2016imported>Labslogbot 55,561 bytes +157 deploying https://gerrit.wikimedia.org/r/281079 (legoktm)
  • curprev 22:58, 2 April 2016imported>Labslogbot 55,404 bytes +425 added local hack to pupetmaster to make scap3 provider more verbose (Amir1)
  • curprev 22:16, 1 April 2016imported>Labslogbot 54,979 bytes +960 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/281046 (Krinkle)
  • curprev 00:16, 1 April 2016imported>Labslogbot 54,019 bytes +413 created oathauth_users table on centralauth db in beta (csteipp)
  • curprev 19:32, 30 March 2016imported>Labslogbot 53,606 bytes +709 deleted some nutcracker and hhvm log files on deployment-mediawiki01 to free space (twentyafterfour)
  • curprev 23:22, 29 March 2016imported>Labslogbot 52,897 bytes +365 running jenkins-jobs update config/ 'mwext-donationinterfacecore125-testextension-zend53' to deploy https://gerrit.wikimedia.org/r/#/c/280261/ (marxarelli)
  • curprev 22:38, 26 March 2016imported>Labslogbot 52,532 bytes +74 reloading zuul for 2d7e050..46923c8 (jzerebecki)
  • curprev 23:55, 25 March 2016imported>Labslogbot 52,458 bytes +1,083 deleting instances integration-slave-trusty-1002 and integration-slave-trusty-1005 (marxarelli)
  • curprev 16:39, 24 March 2016imported>Labslogbot 51,375 bytes +350 restarted rsync service on deployment-tin (thcipriani)
  • curprev 23:00, 23 March 2016imported>Labslogbot 51,025 bytes +200 rm-rf integration-slave-trusty-1013:/mnt/home/jenkins-deploy/tmpfs/jenkins-2/karma-54925082/ (bad permissions, caused Karma issues) (Krinkle)
  • curprev 17:40, 22 March 2016imported>Labslogbot 50,825 bytes +83 deploying https://gerrit.wikimedia.org/r/278926 (legoktm)
  • curprev 21:55, 21 March 2016imported>Labslogbot 50,742 bytes +1,183 zuul: almost all MediaWiki extensions migrated to run the npm job on Nodepool (with Node.js 4.3) T119143 . All tested. Will monitor the build results that ran overnight tomorrow (hashar)
  • curprev 13:04, 19 March 2016imported>Labslogbot 49,559 bytes +114 Jenkins: added ldap-labs-codfw.wikimedia.org as a fallback LDAP server T130446 (hashar)
  • curprev 17:16, 18 March 2016imported>Labslogbot 49,445 bytes +74 reloading zuul for e33494f..89a9659 (jzerebecki)
  • curprev 21:10, 17 March 2016imported>Labslogbot 49,371 bytes +318 updating scap on deployment-tin to test D133 (thcipriani)
  • curprev 14:28, 16 March 2016imported>Labslogbot 49,053 bytes +143 Updated jobs having the package manager cache system (castor) via https://gerrit.wikimedia.org/r/#/c/277774/ (hashar)
  • curprev 15:17, 15 March 2016imported>Labslogbot 48,910 bytes +542 added wikidata.beta.wmflabs.org in https://wikitech.wikimedia.org/wiki/Special:NovaAddress to deployment-cache-text04.deployment-prep.eqiad.wmflabs (jzerebecki)
  • curprev 22:18, 14 March 2016imported>Labslogbot 48,368 bytes +398 new jobs weren't processing in Zuul, lego fixed it and blamed Reedy (greg-g)
  • curprev 22:42, 11 March 2016imported>Labslogbot 47,970 bytes +235 deploying https://gerrit.wikimedia.org/r/276901 (legoktm)
  • curprev 20:22, 10 March 2016imported>Labslogbot 47,735 bytes +503 Nodepool Image ci-jessie-wikimedia-1457641052 in wmflabs-eqiad is ready (hashar)
  • curprev 17:56, 9 March 2016imported>Labslogbot 47,232 bytes +2,402 Cleaned up git clone state in deployment-tin.deployment-prep:/srv/mediawiki-staging/php-master and queued beta-code-update-eqiad to try again (T129371) (bd808)
  • curprev 21:03, 7 March 2016imported>Labslogbot 44,830 bytes +226 Nodepool upgraded to 0.1.1-wmf.4 , it no more waits 1 minute before deleted a used node | T118573 (hashar)
  • curprev 10:20, 6 March 2016imported>Labslogbot 44,604 bytes +83 deploying https://gerrit.wikimedia.org/r/274911 (legoktm)
  • curprev 19:31, 4 March 2016imported>Labslogbot 44,521 bytes +435 Nodepool Image ci-jessie-wikimedia-1457119603 in wmflabs-eqiad is ready - T128846 (hashar)
  • curprev 00:11, 4 March 2016imported>Labslogbot 44,086 bytes +947 salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" (Krinkle)
  • curprev 16:22, 2 March 2016imported>Labslogbot 43,139 bytes +297 reloading zuul for 9398fa1..943f17b (jzerebecki)
  • curprev 01:19, 2 March 2016imported>Labslogbot 42,842 bytes +1,454 force restarting zuul because the queue is very stuck https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart (thcipriani)
  • curprev 16:23, 29 February 2016imported>Labslogbot 41,388 bytes +447 salt -v '*slave*' cmd.run 'rm -fR /mnt/jenkins-workspace/workspace/mwext*jslint' T127362 (hashar)
  • curprev 01:30, 28 February 2016imported>Labslogbot 40,941 bytes +119 Rebooting integration-slave-precise-1012 – Might help T109704 (MySQL not running) (Krinkle)
  • curprev 15:14, 26 February 2016imported>Labslogbot 40,822 bytes +1,342 salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" T128191 (jzerebecki)
  • curprev 22:38, 25 February 2016imported>Labslogbot 39,480 bytes +890 beta: maybe deployment-jobunner01 is processing jobs a bit faster now. Seems like hhvm went wild (hashar)
  • curprev 00:31, 25 February 2016imported>Labslogbot 38,590 bytes +1,255 running puppet on beta to update scap to latest packaged version: sudo salt -b '10%' -G 'deployment_target:scap/scap' cmd.run 'puppet agent -t' (thcipriani)
  • curprev 22:45, 23 February 2016imported>Labslogbot 37,335 bytes +296 deployment-puppetmaster is in a weird rebase state (thcipriani)
  • curprev 22:06, 22 February 2016imported>Labslogbot 37,039 bytes +498 Restarted puppetmaster service on deployment-puppetmaster to "fix" error "invalid byte sequence in US-ASCII" (bd808)
  • curprev 20:19, 20 February 2016imported>Labslogbot 36,541 bytes +415 beta-code-update-eqiad job repeatedly stuck at "IRC notifier plugin" (Krinkle)
  • curprev 12:09, 19 February 2016imported>Labslogbot 36,126 bytes +289 killed https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ been running for 13 hours. Blocked because slave went offline due to labs reboots yesterday (hashar)
  • curprev 19:20, 18 February 2016imported>Labslogbot 35,837 bytes +492 deploying https://gerrit.wikimedia.org/r/271583 and https://gerrit.wikimedia.org/r/271581, both no-ops (legoktm)
  • curprev 07:36, 17 February 2016imported>Labslogbot 35,345 bytes +65 deploying https://gerrit.wikimedia.org/r/271201 (legoktm)
  • curprev 01:01, 17 February 2016imported>Labslogbot 35,280 bytes +1,635 attempting to turn off NFS on 52 instances on deployment-prep project (yuvipanda)
  • curprev 16:28, 15 February 2016imported>Labslogbot 33,645 bytes +1,296 reloading zuul for 2d16ad3..3bb0afa (jzerebecki)
  • curprev 23:54, 14 February 2016imported>Labslogbot 32,349 bytes +350 deploying https://gerrit.wikimedia.org/r/270656 (legoktm)
  • curprev 06:42, 13 February 2016imported>Labslogbot 31,999 bytes +638 restarted nutcracker on deployment-mediawiki01 (bd808)
  • curprev 23:54, 12 February 2016imported>Labslogbot 31,361 bytes +908 beta cluster broken since 20:30 UTC https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor havent looked (hashar)
  • curprev 22:53, 11 February 2016imported>Labslogbot 30,453 bytes +2,515 shutting down deployment-bastion (thcipriani)
  • curprev 23:54, 10 February 2016imported>Labslogbot 27,938 bytes +3,058 depooling Trusty slaves that only have 2GB of ram that is not enough. https://phabricator.wikimedia.org/T126545 (hashar_)
  • curprev 01:03, 10 February 2016imported>Labslogbot 24,880 bytes +3,895 deploying https://gerrit.wikimedia.org/r/269576 (legoktm)
  • curprev 00:39, 9 February 2016imported>Labslogbot 20,985 bytes +2,174 gallium edited /usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py and modified: replication_timeout = 300 -> replication_timeout = 10 (hashar)
  • curprev 00:57, 8 February 2016imported>Labslogbot 18,811 bytes +87 deploying https://gerrit.wikimedia.org/r/268031 (legoktm[NE])
  • curprev 18:34, 6 February 2016imported>Labslogbot 18,724 bytes +74 reloading zuul for bdb2ed4..46ccca9 (jzerebecki)
  • curprev 13:30, 5 February 2016imported>Labslogbot 18,650 bytes +739 beta cleaning out /data/project/logs/archive was from pre logstash area. We no more log this way since May 2015 apparently (hashar)
  • curprev 22:08, 4 February 2016imported>Labslogbot 17,911 bytes +201 reloading zuul for bed7be1..f57b7e2 (jzerebecki)
  • curprev 00:57, 4 February 2016imported>Labslogbot 17,710 bytes +2,091 Got deployment-bastion processing Jenkins jobs again via instructions left by my past self at https://phabricator.wikimedia.org/T72597#747925 (bd808)
  • curprev 18:32, 2 February 2016imported>Labslogbot 15,619 bytes +616 Reloading Zuul to deploy If1f3cb60f4ccb2c1bca112900dbada03a8588370 (marxarelli)
  • curprev 00:42, 2 February 2016imported>Labslogbot 15,003 bytes +1,473 due to T125474 (legoktm)
  • curprev 13:35, 31 January 2016imported>Labslogbot 13,530 bytes +453 Jenkins IRC bot started falling at Jan 30 01:04:00 2016 for whatever reason.... Should be fine now (hashar)
  • curprev 12:46, 30 January 2016imported>Labslogbot 13,077 bytes +110 integration-slave-jessie-1001 : fixed puppet.con server name and ran puppet (hashar)
  • curprev 18:43, 29 January 2016imported>Labslogbot 12,967 bytes +168 updated scap on beta (thcipriani)
  • curprev 23:22, 28 January 2016imported>Labslogbot 12,799 bytes +1,819 Updated portals on betalabs to master (MaxSem)
  • curprev 01:28, 28 January 2016imported>Labslogbot 10,980 bytes +1,252 deployment-prep recreating deployment-sca01 for T125003 (mobrovac)
  • curprev 16:51, 26 January 2016imported>Labslogbot 9,728 bytes +472 updated OCG to version 64050af0456a43344b32e3e93561a79207565eaf (cscott)
  • curprev 18:59, 25 January 2016imported>Labslogbot 9,256 bytes +525 started redis-server on deployment-redis01 by commenting out latency-monitor-threshold from the redis.conf (twentyafterfour)
  • curprev 06:45, 24 January 2016imported>Labslogbot 8,731 bytes +148 deploying https://gerrit.wikimedia.org/r/266039 (legoktm)
  • curprev 23:58, 22 January 2016imported>Labslogbot 8,583 bytes +1,274 removed skins from mwext-qunit workspace on trusty-1013 slave (legoktm)
  • curprev 21:41, 21 January 2016imported>Labslogbot 7,309 bytes +757 restored role::mail::mx on deployment-mx (hashar)
  • curprev 01:00, 21 January 2016imported>Labslogbot 6,552 bytes +1,287 jenkins-deploy@deployment-bastion:/srv/mediawiki-staging/php-master/extensions$ git pull && git submodule update --init --recursive (jzerebecki)
  • curprev 22:25, 19 January 2016imported>Labslogbot 5,265 bytes +695 deleting *zend* workspaces on precise slaves (legoktm)
  • curprev 18:02, 17 January 2016imported>Labslogbot 4,570 bytes +83 deploying https://gerrit.wikimedia.org/r/264605 (legoktm)
  • curprev 21:47, 16 January 2016imported>Labslogbot 4,487 bytes +991 deploying https://gerrit.wikimedia.org/r/264489 (legoktm)
  • curprev 22:22, 15 January 2016imported>Luke081515 3,496 bytes −211,345 time for archiving...
  • curprev 20:06, 14 January 2016imported>Labslogbot 214,841 bytes +213 deploying https://gerrit.wikimedia.org/r/264122 (legoktm)
  • curprev 21:06, 13 January 2016imported>Labslogbot 214,628 bytes +353 beta cluster code is up to date again. Got delayed by roughly 4 hours. (hashar)
  • curprev 23:30, 12 January 2016imported>Labslogbot 214,275 bytes +277 deploying https://gerrit.wikimedia.org/r/263757 https://gerrit.wikimedia.org/r/263756 (legoktm)
  • curprev 22:25, 11 January 2016imported>Labslogbot 213,998 bytes +501 Deleting old references on Zuul-merger for mediawiki/core : <tt>/usr/share/python/zuul/bin/python /home/hashar/zuul-clear-refs.py --until 15 /srv/ssd/zuul/git/mediawiki/core </tt> (hashar)
  • curprev 23:16, 7 January 2016imported>Labslogbot 213,497 bytes +290 deleted /mnt/jenkins-workspace/workspace/mediawiki-extensions-qunit/src/extensions/PdfHandler/.git/refs/heads/wmf/1.26wmf16.lock on slave 1013 (legoktm)
  • curprev 01:25, 7 January 2016imported>Labslogbot 213,207 bytes +835 reloading zuul for b0a5335..c16368a (jzerebecki)
  • curprev 16:54, 5 January 2016imported>Labslogbot 212,372 bytes +322 Removed elastic search from CI slaves https://phabricator.wikimedia.org/T89083 https://gerrit.wikimedia.org/r/#/c/259301/ (hashar_)
  • curprev 21:06, 4 January 2016imported>Labslogbot 212,050 bytes +192 gallium has puppet enabled again (hashar)
  • curprev 03:17, 2 January 2016imported>Labslogbot 211,858 bytes +76 purged varnishs on deployment-cache-text04 (yurik)
  • curprev 22:17, 1 January 2016imported>Labslogbot 211,782 bytes +121 No nodepool ci-jessie-* hosts seen in Jenkins interface and rake-jessie jobs backing up (bd808)
  • curprev 00:13, 30 December 2015imported>Labslogbot 211,661 bytes +779 rake-jessie jobs running again which will hopefully clear the large zuull backlog (bd808)
  • curprev 21:41, 26 December 2015imported>Labslogbot 210,882 bytes +618 integration: getting rid of $wgHTTPProxy https://gerrit.wikimedia.org/r/261096 (no more needed) (hashar)
  • curprev 23:37, 23 December 2015imported>Labslogbot 210,264 bytes +192 Reloading Zuul to deploy I39b9f292e95363addf8983eec5d08a0af527a163 (marxarelli)
  • curprev 23:31, 22 December 2015imported>Labslogbot 210,072 bytes +496 (when npm jobs run) - sudo rm -rf /mnt/home/jenkins-deploy/.npm at integration-slave-trusty-1015 (due to cache corruption) (Krinkle)
  • curprev 20:06, 21 December 2015imported>Labslogbot 209,576 bytes +541 Downgrading Jenkins plugin from 1.24 to 1.21 (hashar)
  • curprev 22:18, 19 December 2015imported>Labslogbot 209,035 bytes +412 sudo rm -rf /mnt/home/jenkins-deploy/tmpfs/jenkins* on integration-slave-precise-1014 (Krinkle)
  • curprev 01:23, 19 December 2015imported>Labslogbot 208,623 bytes +1,774 Added Reedy as a projectadmin in the integration project (bd808)
  • curprev 23:07, 17 December 2015imported>Labslogbot 206,849 bytes +1,799 sudo salt --show-timeout '*slave*' cmd.run 'rm -fR /mnt/home/jenkins-deploy/tmpfs/jenkins-?/*' (legoktm)
  • curprev 23:23, 16 December 2015imported>Labslogbot 205,050 bytes +877 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/259616 (Krinkle)
  • curprev 22:57, 15 December 2015imported>Labslogbot 204,173 bytes +881 On scandium created /srv/ssd/zuul/git/wikimedia/fundraising/crm repo manually. Namespace conflict with wikimedia/fundraising/crm/civicrm.git which prevented zuul-merger to clone the arm repo (hashar)
  • curprev 00:57, 15 December 2015imported>Labslogbot 203,292 bytes +613 marking integration-slave-trusty-1012 offline, strange zuul.cloner behavior. (thcipriani)
  • curprev 23:30, 13 December 2015imported>Labslogbot 202,679 bytes +171 Ran deployment-bastion:~bd808/cleanup-var-crap.sh and freed 846M on /var (bd808)
  • curprev 22:18, 11 December 2015imported>Labslogbot 202,508 bytes +909 Stopped zuul merger on gallium to have phabricator/extensions populated on scandium (namespacing issue). Restarted zuul-merger on gallium once done. (hashar)
  • curprev 01:12, 11 December 2015imported>Labslogbot 201,599 bytes +945 deploying https://gerrit.wikimedia.org/r/258395 (legoktm)
  • curprev 01:28, 10 December 2015imported>Labslogbot 200,654 bytes +1,070 deploying https://gerrit.wikimedia.org/r/258090 (legoktm)
  • curprev 20:31, 8 December 2015imported>Labslogbot 199,584 bytes +1,662 beta cluster instances switching to new ldap configuration (hashar)
  • curprev 16:16, 7 December 2015imported>Labslogbot 197,922 bytes +790 Nodepool no more listens for Jenkins events over ZeroMQ. No TCP connection established on port 8888 (hashar)
  • curprev 19:24, 4 December 2015imported>Labslogbot 197,132 bytes +210 bumped portals (MaxSem)
  • curprev 23:53, 3 December 2015imported>Labslogbot 196,922 bytes +509 Reloading Zuul to deploy If60f720995dfc7859e53cf33043b5a21b1a4b085 (marxarelli)
  • curprev 11:16, 2 December 2015imported>Labslogbot 196,413 bytes +556 configure wmf-insecte to join #wikimedia-android-ci ( https://gerrit.wikimedia.org/r/#/c/254905/3/jjb/mobile.yaml,unified ) (hashar)
  • curprev 16:06, 1 December 2015imported>Labslogbot 195,857 bytes +584 split mediawiki core parser tests under Zend to their own job https://gerrit.wikimedia.org/r/#/c/256006/ (hashar)
  • curprev 23:23, 29 November 2015imported>Labslogbot 195,273 bytes +436 updated cherry-pick of https://gerrit.wikimedia.org/r/#/c/255916/ to PS2 (bd808)
  • curprev 19:25, 27 November 2015imported>Labslogbot 194,837 bytes +235 Trebuchet returners not reporting to redis from sca0[12] (bd808)
  • curprev 22:35, 26 November 2015imported>Labslogbot 194,602 bytes +553 Upgraded Zuul on labs Precise hosts to zuul_2.1.0-60-g1cc37f7-wmf4precise1 ( https://phabricator.wikimedia.org/T119741 ) (hashar)
  • curprev 19:53, 25 November 2015imported>Labslogbot 194,049 bytes +173 ran mwscript sql.php --wiki=enwiki --wikidb=wikishared /srv/mediawiki-staging/php-master/extensions/Echo/db_patches/echo_unread_wikis.sql (legoktm)
  • curprev 21:16, 24 November 2015imported>Labslogbot 193,876 bytes +378 bumping our JJB fork to 4eb33b2. Pinning jenkins==0.4.8 with https://gerrit.wikimedia.org/r/255182 (hashar)
  • curprev 22:32, 23 November 2015imported>Labslogbot 193,498 bytes +1,069 Updating parsoidsvc-source-npm-* jobs https://gerrit.wikimedia.org/r/254946 (hashar)
  • curprev 21:15, 22 November 2015imported>Labslogbot 192,429 bytes +110 Zuul queue stuck as of 3 hours ago. Jenkins unresponsive or 503 over HTTP. (Krinkle)
  • curprev 21:59, 20 November 2015imported>Labslogbot 192,319 bytes +266 Removed shadow l10nupdate user from /etc/passwd & /etc/shadow on mira.deployment-prep.eqiad.wmflabs (bd808)
  • curprev 21:11, 13 November 2015imported>Labslogbot 192,053 bytes +992 zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/tools/releng --change 200240,1 (hashar)
  • curprev 22:03, 12 November 2015imported>Labslogbot 191,061 bytes +94 rm -rf mediawiki-extensions-qunit workspace on trusty-1012 (legoktm)
  • curprev 17:58, 11 November 2015imported>Labslogbot 190,967 bytes +56 reloading zuul for e81ee95..acbd851 (jzerebecki)
  • curprev 00:02, 11 November 2015imported>Labslogbot 190,911 bytes +500 deployment-puppetmaster:/var/lib/git/operations/puppet was left mid-rebase conflicts with portals apache config. Fixed conflicts added the file and continued the rebase. (thcipriani)
  • curprev 23:19, 9 November 2015imported>Labslogbot 190,411 bytes +630 put https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1015/ back on (hashar)
  • curprev 20:52, 7 November 2015imported>Labslogbot 189,781 bytes +74 reloading zuul for a2951c3..ccea029 (jzerebecki)
  • curprev 23:02, 6 November 2015imported>Labslogbot 189,707 bytes +699 reloading zuul for 583e87e..a2951c3 (jzerebecki)
  • curprev 22:50, 5 November 2015imported>Labslogbot 189,008 bytes +529 deploying https://gerrit.wikimedia.org/r/251418 (legoktm)
  • curprev 23:18, 4 November 2015imported>Labslogbot 188,479 bytes +820 Running `jenkins-jobs update config/ 'mw-tools-scap-tox-doc-publish'` to test Ib4753ad493115682d902cf15136199fd2083b8e5 (marxarelli)
  • curprev 22:48, 3 November 2015imported>Labslogbot 187,659 bytes +690 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/250847 (thcipriani)
  • curprev 20:48, 2 November 2015imported>Labslogbot 186,969 bytes +1,694 Upgraded Zuul on Trusty slaves to /root/zuul_2.1.0-60-g1cc37f7-wmf1trusty1_amd64.deb (hashar)
  • curprev 22:18, 1 November 2015imported>Labslogbot 185,275 bytes +83 deploying https://gerrit.wikimedia.org/r/250362 (legoktm)
  • curprev 21:32, 31 October 2015imported>Labslogbot 185,192 bytes +107 Ran `sudo rm /var/log/account/pacct.?` on deployment-bastion; freed ~425M (bd808)
  • curprev 18:20, 30 October 2015imported>Labslogbot 185,085 bytes +401 deployment-db2 replication recovered after slave stop/reset/set master position/start (marxarelli)
  • curprev 17:33, 29 October 2015imported>Labslogbot 184,684 bytes +621 deploying https://gerrit.wikimedia.org/r/249223 (legoktm)
  • curprev 00:12, 29 October 2015imported>Labslogbot 184,063 bytes +960 Manually fixed permissions on mw-config/portals, reinitialized submodule and synced (MaxSem)
  • curprev 20:42, 27 October 2015imported>Labslogbot 183,103 bytes +539 Nuking Nodepool instances using the previous snapshot ( ci-jessie-wikimedia-1445955240 ) (hashar)
  • curprev 23:04, 26 October 2015imported>Labslogbot 182,564 bytes +630 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/249015 (Krinkle)
  • curprev 05:58, 24 October 2015imported>Labslogbot 181,934 bytes +274 deployment-db2 data restored, replication working (marxarel_)
  • curprev 00:20, 24 October 2015imported>Labslogbot 181,660 bytes +896 deploying https://gerrit.wikimedia.org/r/248576 (legoktm)
  • curprev 10:36, 22 October 2015imported>Labslogbot 180,764 bytes +994 integration: cherry picked https://gerrit.wikimedia.org/r/#/c/244748/ "contint: install npm/grunt-cli with npm" , giving it a try one host a time (hashar)
  • curprev 14:51, 19 October 2015imported>Labslogbot 179,770 bytes +647 Adding CirrusSearch to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247280/ ) (hashar)
  • curprev 20:51, 16 October 2015imported>Labslogbot 179,123 bytes +530 Restarting Jenkins to remove potential dead locks before the week-end (hashar)
  • curprev 22:28, 15 October 2015imported>Labslogbot 178,593 bytes +450 deploying https://gerrit.wikimedia.org/r/246785 (legoktm)
  • curprev 18:02, 14 October 2015imported>Labslogbot 178,143 bytes +262 deploying https://gerrit.wikimedia.org/r/246276 (legoktm)
  • curprev 03:21, 13 October 2015imported>Labslogbot 177,881 bytes +109 Updated scap to 13c2af4 (Fix sync-dblist to go with dblist moves to folder) (bd808)
  • curprev 08:54, 12 October 2015imported>Labslogbot 177,772 bytes +408 zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243 (hashar)
  • curprev 00:38, 12 October 2015imported>Labslogbot 177,364 bytes +359 fixed puppet on deploymnet-restbase01 by running several manual apt-get and dpkg commands; had to downgrade zsh from 5.1.1-1 (unstable) to 5.0.7-5 (stable) (bd808)
  • curprev 21:57, 9 October 2015imported>Labslogbot 177,005 bytes +905 21:51 < ori> !log deployment-prep Accidentally clobbered /etc/init.d/mysql on deployment-db1, causing deployment-prep failures. Restored now (greg-g)
  • curprev 15:49, 7 October 2015imported>Labslogbot 176,100 bytes +1,030 Updated cherry-pick of https://gerrit.wikimedia.org/r/#/c/241984 on deployment-puppetmaster and forced puppet run on deployment-logstash2 (bd808)
  • curprev 00:16, 7 October 2015imported>Labslogbot 175,070 bytes +553 Restored normal mysql replication on deployment-beta (Krinkle)
  • curprev 22:20, 5 October 2015imported>Labslogbot 174,517 bytes +1,588 deployment-prep deleted deployment-bastion:/var/log/account files to free up some space on /var (Krenair)
  • curprev 21:39, 4 October 2015imported>Labslogbot 172,929 bytes +195 Changed MW private DB settings in beta to set $wgDBadmin* properly so the `sql` command works again (Krenair)
  • curprev 18:29, 3 October 2015imported>Labslogbot 172,734 bytes +56 reloading zuul for 4aa5d22..d609dad (jzerebecki)
  • curprev 01:05, 3 October 2015imported>Labslogbot 172,678 bytes +1,380 Clean up spurious remotes/origin/review/markahershberger/207722 branch in mediawiki/core (Krinkle)
  • curprev 16:30, 1 October 2015imported>Labslogbot 171,298 bytes +100 test again (valhallasw`cloud)
  • curprev 22:19, 30 September 2015imported>Labslogbot 171,198 bytes +551 Cherry-picked https://gerrit.wikimedia.org/r/#/c/241984 to deployment-puppetmaster (bd808)
  • curprev 22:36, 29 September 2015imported>Labslogbot 170,647 bytes +383 Added support for "Install MediaWiki" and "PHPUnit" to the Jenkins section detection (Krinkle)
  • curprev 23:37, 28 September 2015imported>Labslogbot 170,264 bytes +516 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/241960 (Krinkle)
  • curprev 07:22, 27 September 2015imported>Labslogbot 169,748 bytes +83 deploying https://gerrit.wikimedia.org/r/241500 (legoktm)
  • curprev 22:26, 26 September 2015imported>Labslogbot 169,665 bytes +604 deploying https://gerrit.wikimedia.org/r/241312 (legoktm)
  • curprev 17:18, 25 September 2015imported>Labslogbot 169,061 bytes +950 deploying https://gerrit.wikimedia.org/r/241086 (legoktm)
  • curprev 00:59, 25 September 2015imported>Labslogbot 168,111 bytes +1,002 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/240938 (Krinkle)
  • curprev 23:26, 23 September 2015imported>Labslogbot 167,109 bytes +2,054 Reloading Zuul to deploy Iaa33b2002e898d68fe78f236e225c6e4bac61679 (thcipriani)
  • curprev 17:05, 22 September 2015imported>Labslogbot 165,055 bytes +190 deploying https://gerrit.wikimedia.org/r/238985 (legoktm)
  • curprev 19:30, 21 September 2015imported>Labslogbot 164,865 bytes +281 lets get rid of PhantomJS https://phabricator.wikimedia.org/T113279 :D (hashar)
  • curprev 22:34, 18 September 2015imported>Labslogbot 164,584 bytes +178 we missed log entries on the wikitech log from the 15th, see https://tools.wmflabs.org/sal/releng?p=0&q=&d=2015-09-18 (greg-g)
  • curprev 18:09, 15 September 2015imported>Labslogbot 164,406 bytes +260 deploying https://gerrit.wikimedia.org/r/238506 (legoktm)
  • curprev 19:08, 14 September 2015imported>Labslogbot 164,146 bytes +388 reloading zuul for bd97ce4..4abd32e (jzerebecki)
  • curprev 19:48, 12 September 2015imported>Labslogbot 163,758 bytes +74 reloading zuul for c7f17b5..a813107 (jzerebecki)
  • curprev 21:44, 11 September 2015imported>Labslogbot 163,684 bytes +345 self note: test unifier should be named "Pinailleur" (hashar)
  • curprev 22:28, 10 September 2015imported>Labslogbot 163,339 bytes +402 cherry-picked https://gerrit.wikimedia.org/r/#/c/237523/ on deployment-puppetmaster (Krenair)
  • curprev 18:14, 9 September 2015imported>Labslogbot 162,937 bytes +705 restarted nutcracker on mediawiki02 (bd808)
  • curprev 21:11, 8 September 2015imported>Labslogbot 162,232 bytes +187 deploying https://gerrit.wikimedia.org/r/236943 (legoktm)
  • curprev 17:01, 7 September 2015imported>Labslogbot 162,045 bytes +275 reloading zuul for 1b53ca9..4d9d237 (jzerebecki)
  • curprev 22:07, 4 September 2015imported>Labslogbot 161,770 bytes +373 reloading zuul for 1219cc3..1b53ca9 (jzerebecki)
  • curprev 22:59, 3 September 2015imported>Labslogbot 161,397 bytes +578 Rebooting integration-dev.eqiad.wmflabs (marxarelli)
  • curprev 00:51, 3 September 2015imported>Labslogbot 160,819 bytes +612 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/235656 (Krinkle)
  • curprev 21:08, 1 September 2015imported>Labslogbot 160,207 bytes +151 marxarelli properly build a CI image using diskimage-builder \O/ (hashar)
  • curprev 01:12, 1 September 2015imported>Labslogbot 160,056 bytes +470 Re-restarting grrrit-wm rolled back to 2f5de55ff75c3c268decfda7442dcdd62df0a42d (James_F)
  • curprev 20:53, 30 August 2015imported>Labslogbot 159,586 bytes +229 beta-scap-eqiad failling due to some mwdeploy not being able to ssh to other hosts. Attempted to add the ssh key again following https://phabricator.wikimedia.org/T109007#1537572 which fixed it (hashar)
  • curprev 01:01, 29 August 2015imported>Labslogbot 159,357 bytes +1,565 Deleted local mwdeploy user on deployment-tmh01 that was causing scap failures (bd808)
  • curprev 00:56, 28 August 2015imported>Labslogbot 157,792 bytes +761 sudo keyholder arm on deployment-bastion fixed beta-scap-eqiad (thcipriani)
  • curprev 00:14, 27 August 2015imported>Labslogbot 157,031 bytes +1,624 Reloading Zuul to deploy Iaab45d659df4b817a0dd27a7ccde17d71f630aaa (marxarelli)
  • curprev 23:48, 25 August 2015imported>Labslogbot 155,407 bytes +1,482 stopping puppetmaster and disabling puppet runs on deployment-puppetmaster until we get a change to diagnose/rebuild (tomorrow) (thcipriani)
  • curprev 00:58, 25 August 2015imported>Labslogbot 153,925 bytes +1,607 Updated tin:/srv/deployment/integration/slave-scripts to b287e93 (Revert "Run mw-install-mysql.sh with statement tracing") and synced via trebuchet (bd808)
  • curprev 00:37, 23 August 2015imported>Labslogbot 152,318 bytes +83 deploying https://gerrit.wikimedia.org/r/233178 (legoktm)
  • curprev 00:08, 22 August 2015imported>Labslogbot 152,235 bytes +489 Reloading Zuul to deploy I86fcd065f7a77e293a8882fa3ad2c20fffe4b092 (marxarelli)
  • curprev 16:35, 19 August 2015imported>Labslogbot 151,746 bytes +591 Verified that elasticsearch and mysql are now running after the reinstall and a manual puppet run (marxarelli)
  • curprev 18:33, 18 August 2015imported>Labslogbot 151,155 bytes +583 (mysql wasn't started as puppet never got to that point) (jzerebecki)
  • curprev 22:18, 17 August 2015imported>Labslogbot 150,572 bytes +246 running schema change for gerrit:202344 on beta (legoktm)
  • curprev 00:38, 15 August 2015imported>Labslogbot 150,326 bytes +650 Reloading Zuul to deploy I9ef82b6d3ea7d83de8e4a67c9715ccf335c00b88 (marxarelli)
  • curprev 23:27, 13 August 2015imported>Labslogbot 149,676 bytes +545 deploying https://gerrit.wikimedia.org/r/230256 (legoktm)
  • curprev 22:16, 12 August 2015imported>Labslogbot 149,131 bytes +452 Cherry-picked https://gerrit.wikimedia.org/r/#/c/231179/ (Disable authentication for Kibana) (bd808)
  • curprev 21:54, 11 August 2015imported>Labslogbot 148,679 bytes +393 Cherry-picked https://gerrit.wikimedia.org/r/#/c/230922/ for testing (bd808)
  • curprev 01:27, 11 August 2015imported>Labslogbot 148,286 bytes +1,503 deploying https://gerrit.wikimedia.org/r/230619 (legoktm)
  • curprev 00:57, 8 August 2015imported>Labslogbot 146,783 bytes +396 deploying https://gerrit.wikimedia.org/r/230255 (legoktm)
  • curprev 22:53, 6 August 2015imported>Labslogbot 146,387 bytes +138 reloading zuul for e17f502..af1f5d1 (jzerebecki)
  • curprev 20:43, 5 August 2015imported>Labslogbot 146,249 bytes +237 marking integration-slave-precise-1012 back online, elasticsearch hung up on starting because /var/run/elasticsearch wasn't a directory :( (thcipriani)
  • curprev 00:12, 5 August 2015imported>Labslogbot 146,012 bytes +454 Reloading Zuul to deploy https://gerrit.wikimedia.org/r/229287 (Krinkle)
  • curprev 00:30, 4 August 2015imported>Labslogbot 145,558 bytes +105 Reloading Zuul to deploy I18fa56da9a27a4efeb061ceba773c8b50bc4a8f4 (marxarelli)
  • curprev 04:30, 2 August 2015imported>Labslogbot 145,453 bytes +325 deploying https://gerrit.wikimedia.org/r/228606 (legoktm)
  • curprev 01:28, 2 August 2015imported>Labslogbot 145,128 bytes +296 deploying https://gerrit.wikimedia.org/r/226753 (legoktm)
  • curprev 01:32, 31 July 2015imported>Labslogbot 144,832 bytes +71 reload zuul for 83a30e5..f2d2517 (jzerebecki)
  • curprev 23:50, 30 July 2015imported>Labslogbot 144,761 bytes +413 upgraded nutcracker to 0.4.1-1+wm2~precise1 on deployment-bastion (bd808)
  • curprev 23:55, 29 July 2015imported>Labslogbot 144,348 bytes +1,454 clearing disk space on integrations-slave-trusty-1012 with `find /mnt/jenkins-workspace/workspace -mindepth 1 -maxdepth 1 -type d -mtime +15 -exec rm -rf {} \;` (marxarelli)
  • curprev 23:43, 28 July 2015imported>Labslogbot 142,894 bytes +997 running `jenkins-jobs update config/ 'mwext-mw-selenium'` to deploy I7afa07e9f559bffeeebaf7454cc6b39a37e04063 (marxarelli)
  • curprev 00:46, 28 July 2015imported>Labslogbot 141,897 bytes +829 deploying https://gerrit.wikimedia.org/r/227383 (legoktm)
  • curprev 17:35, 24 July 2015imported>Labslogbot 141,068 bytes +1,770 updating integration slave scripts from integration-saltmaster to deploy I6906fadede546ce2205797da1c6b267aed586e17 (marxarelli)
  • curprev 23:03, 23 July 2015imported>Labslogbot 139,298 bytes +1,450 running `jenkins-jobs update config/ 'browsertests-*'` to deploy I2d0f83d0c6a406d46627578cb8db0706d1b8655d (marxarelli)
  • curprev 01:09, 23 July 2015imported>Labslogbot 137,848 bytes +909 beta-mediawiki-config-update-eqiad jobs stuck (legoktm)
  • curprev 22:54, 21 July 2015imported>Labslogbot 136,939 bytes +1,108 22:50 < chasemp> "then git reset --hard 9588d0a6844fc9cc68372f4bf3e1eda3cffc8138 in /etc/zuul/wikimedia" (greg-g)
  • curprev 16:55, 20 July 2015imported>Labslogbot 135,831 bytes +99 restarted puppetmaster on deployment-salt, was acting whacky (thcipriani)
  • curprev 21:45, 17 July 2015imported>Labslogbot 135,732 bytes +289 upgraded nodepool to 0.0.1-104-gddd6003-wmf4 . That fix graceful stop via SIGUSR1 and let me complete the systemd integration (hashar)
  • curprev 16:08, 16 July 2015imported>Labslogbot 135,443 bytes +378 kept nodepool stopped on labnodepool1001.eqiad.wmnet because it spams the cron log (hashar_)
  • curprev 20:53, 15 July 2015imported>Labslogbot 135,065 bytes +512 Added JanZerebecki as deployment-prep root (bd808)
  • curprev 00:45, 14 July 2015imported>Labslogbot 134,553 bytes +733 /srv/deployment/scap/scap on deployment-mediawiki02 had corrupt git cache info; moved to scap-corrupt and forced a re-sync (bd808)
  • curprev 04:35, 11 July 2015imported>Labslogbot 133,820 bytes +289 Updated /var/lib/git/labs/private to latest upstream (bd808)
  • curprev 16:12, 10 July 2015imported>Labslogbot 133,531 bytes +132 nodepool puppitization going on :-D (hashar)
  • curprev 22:16, 9 July 2015imported>Labslogbot 133,399 bytes +88 integration: pulled labs/private.git : dbef45d..d41010d (hashar)
  • curprev 23:17, 8 July 2015imported>Labslogbot 133,311 bytes +771 Kibana functional again. Imported some dashboards from prod instance. (bd808)
  • curprev 23:58, 7 July 2015imported>Labslogbot 132,540 bytes +867 updated scap to 303e72e (Increment deployment stats after sync-wikiversions) (bd808)
  • curprev 23:34, 6 July 2015imported>Labslogbot 131,673 bytes +806 Reloading Zuul to deploy I33ac72e7df498e58f0e25d8c59f167d13eae06cf (marxarelli)
  • curprev 14:07, 3 July 2015imported>Labslogbot 130,867 bytes +495 adding puppetmaster::certcleaner class to integration and beta puppetmaster (hashar)
  • curprev 01:18, 3 July 2015imported>Labslogbot 130,372 bytes +273 deploying https://gerrit.wikimedia.org/r/166074 (legoktm)
  • curprev 15:44, 1 July 2015imported>Labslogbot 130,099 bytes +1,313 Kunal awesome dashboard for repos https://www.mediawiki.org/wiki/User:Legoktm/ci (hashar)
  • curprev 10:07, 30 June 2015imported>Labslogbot 128,786 bytes +914 deployment-bastion sudo -u l10nupdate bash -c 'cd /srv/l10nupdate/mediawiki/extension && git submodule foreach git gc' (hashar)
  • curprev 23:19, 29 June 2015imported>Labslogbot 127,872 bytes +556 Moved logstash irc bot from logstash1 to logstash2 (bd808)
  • curprev 02:42, 27 June 2015imported>Labslogbot 127,316 bytes +571 deploying https://gerrit.wikimedia.org/r/221343 & https://gerrit.wikimedia.org/r/221344 (legoktm)
  • curprev 01:28, 27 June 2015imported>Labslogbot 126,745 bytes +719 deploying https://gerrit.wikimedia.org/r/221329 (legoktm)
  • curprev 23:38, 25 June 2015imported>Labslogbot 126,026 bytes +1,438 deploying https://gerrit.wikimedia.org/r/221001 (legoktm)
  • curprev 19:31, 24 June 2015imported>Labslogbot 124,588 bytes +1,064 rebooting deployment-cache-upload02 (hashar)
  • curprev 00:38, 24 June 2015imported>Labslogbot 123,524 bytes +944 reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/219513/ (marxarelli)
  • curprev 00:07, 23 June 2015imported>Labslogbot 122,580 bytes +366 deploying https://gerrit.wikimedia.org/r/220020 (legoktm)
  • curprev 14:58, 22 June 2015imported>Labslogbot 122,214 bytes +122,214 disabled sshd MAC/KEX hardening on beta (was https://gerrit.wikimedia.org/r/#/c/219828/ ) (hashar)
(newest | oldest) View ( | older 500) (20 | 50 | 100 | 250 | 500)