You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(Hauskatze: github: deleting https://github.com/wikimedia/labs-tools-signpost mirror | T212559)
imported>Nhatminh01
(archive 2018)
Line 1: Line 1:
== 2018-12-22 ==
* 13:30 Hauskatze: github: deleting https://github.com/wikimedia/labs-tools-signpost mirror {{!}} [[phab:T212559|T212559]]
== 2018-12-21 ==
* 11:17 hashar: new doc job is https://integration.wikimedia.org/ci/job/publish-to-doc1001/ and runs just after https://integration.wikimedia.org/ci/job/publish-on-contint1001/
* 11:17 hashar: updating all publish jobs to also publish to doc1001.eqiad.wmnet {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/480657/ {{!}} [[phab:T137890|T137890]]
== 2018-12-20 ==
* 18:53 thcipriani: unstick post-merge pipeline (beta-scap-eqiad stuck for 9 hours), [x] beta-scap-eqiad job clogging the works
* 05:10 legoktm: deployed https://gerrit.wikimedia.org/r/480908
* 05:04 legoktm: deployed https://gerrit.wikimedia.org/r/480904
== 2018-12-19 ==
* 23:27 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 18:05 Hauskatze: Created mediawiki/extensions/ExternalGuidance.git repos on Gerrit, set-up mirrors on Diffusion and GitHub; per mediawiki.org request.
* 15:20 hashar: Testing spicerack postmerge job for [[phab:T205894|T205894]]:  contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 480724,2
== 2018-12-18 ==
* 10:56 legoktm: rebuilding docker tox-* images
* 08:17 legoktm: rebuilding docker tox image https://gerrit.wikimedia.org/r/480428
== 2018-12-17 ==
* 22:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/480256
* 22:26 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/480165/
* 11:43 Amir1: ores:18d3657 is going beta
== 2018-12-14 ==
* 16:19 Krenair: [[phab:T204500|T204500]] Powering off the remaining trusty and main deployment instance, deployment-zotero01, which is no longer in use anyway
* 13:08 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf {{!}} tee WikibaseQualityConstraints-config.php # [[phab:T209957|T209957]]
* 06:23 kart_: Beta: Updated cxserver to {{Gerrit|de618f3}}
== 2018-12-13 ==
* 21:56 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 18:31 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/479501/
* 17:20 Amir1: ores:1a3de73 is going beta
* 09:32 hashar: Adding integration-slave-jessie-1001 back online
* 09:30 hashar: cleaned up disk on integration-slave-jessie-1001
* 01:05 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/479158
== 2018-12-12 ==
* 21:50 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-ConditionalShowSection {{!}} refs. [[phab:T211821|T211821]]
* 21:20 bearND: (beta): Update mobileapps to {{Gerrit|55981a8}}
* 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
* 19:37 greg-g: enabled 2fa on the wmfgerrit github user account, recovery codes in releng's pwstore
* 17:24 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf {{!}} tee WikibaseQualityConstraints-config.php # [[phab:T209957|T209957]]
== 2018-12-11 ==
* 20:03 hashar: building image docker-registry.wikimedia.org/releng/commit-message-validator:0.6.0 {{!}} [[phab:T210278|T210278]]
* 10:09 hashar: jenkins: removed some legacy/unused user token via https://integration.wikimedia.org/ci/administrativeMonitor/legacyApiToken/manage {{!}} [[phab:T206234|T206234]]
* 10:03 hashar: Deleting unused instance jenkinstest.integration.eqiad.wmflabs {{!}} [[phab:T208803|T208803]]
* 09:42 godog: enable profile::rsyslog::udp_localhost_compat
* 09:04 hashar: gerrit: added _joe_ to 'integration' project so he can +2 changes notably to integration/config ( https://gerrit.wikimedia.org/r/#/admin/groups/10,members )
== 2018-12-09 ==
* 23:38 Krenair: restarted apache on deployment-mediawiki-07 for [[phab:T211524|T211524]]
== 2018-12-05 ==
* 14:58 thcipriani: bring integration-slave-jessie-1004 back online (disks look fine)
* 02:57 kart_: Beta: Updated cxserver to {{Gerrit|c4240e6}}
== 2018-12-04 ==
* 06:04 legoktm: deploying https://gerrit.wikimedia.org/r/477115
* 04:08 legoktm: deploying https://gerrit.wikimedia.org/r/473724 https://gerrit.wikimedia.org/r/477416
== 2018-12-03 ==
* 22:05 Krenair: [[phab:T210214|T210214]] deleting deployment-cache-text04
* 15:01 Amir1: ores:03b9c98 is going beta
== 2018-11-30 ==
* 22:32 thcipriani: removing deployment-redis0{5,6} since I can't ssh in, and they don't seem to have any puppet roles applied
== 2018-11-29 ==
* 14:26 godog: switch mediawiki logging to use localhost syslog -> kafka -> logstash
* 13:54 hashar: Building docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.1.0 for https://gerrit.wikimedia.org/r/475496 {{!}} [[phab:T209849|T209849]]
* 08:35 hashar: Deleted instances castor02 and integration-publishing , replaced by new instances in the new WMCS region {{!}} [[phab:T208803|T208803]]
== 2018-11-28 ==
* 08:45 hashar: Switching Jenkins job cache to integration-castor03  with AN EMPTY CACHE  {{!}} [[phab:T208803|T208803]]
* 08:18 godog: test mediawiki kafka logging on deployment-mediawiki-07
== 2018-11-27 ==
* 15:55 Amir1: giving User:Fisch-WMDE and User:Thiemo Mättig (WMDE) admin rights on enwiki, dewiki and arwiki (WMDE staff, beta cluster)
* 13:59 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/475352
* 11:38 Amir1: Deploying ores:9b9ba06 to beta
== 2018-11-26 ==
* 22:22 Krenair: shutoff deployment-cache-text04, now replaced with deployment-cache-text05 in the new region - [[phab:T210214|T210214]]
* 18:02 Amir1: deploy ores:3cdaaa6 to beta
== 2018-11-23 ==
* 13:55 dcausse: restarted elasticsearch on all deployement-elastic0X nodes (search broken on the beta cluster)
* 11:09 hashar: Jenkins: removing plugins "Single Use Slave" and "Event Publisher (via ZMQ PUB SUB)". Were used for Nodepool {{!}} [[phab:T209361|T209361]]
* 10:29 hashar: Building releng/java8-wikidata-query-rdf:0.2.1 container for https://gerrit.wikimedia.org/r/#/c/integration/config/+/475428/ [[phab:T209776|T209776]]
== 2018-11-22 ==
* 10:34 hashar: Building container releng/java8-wikidata-query-rdf:0.2.0 to support Chrome ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/475298/ {{!}} [[phab:T209776|T209776]] )
== 2018-11-21 ==
* 21:44 Krenair: Changed beta.wmflabs.org and *.beta.wmflabs.org A records from 208.80.155.135 (deployment-cache-text04) to 185.15.56.36 (deployment-cache-text05)
* 20:49 hashar: gerrit: added WMDE-leszek to group "Gerrit Managers" https://gerrit.wikimedia.org/r/#/admin/groups/119,members {{!}} [[phab:T200311|T200311]]
* 03:25 bd808: Forced puppet run on deployment-mediawiki-0[79] to pick up new redis::shards settings [[phab:T210030|T210030]]
* 03:02 bd808: Repeated application of role::mediawiki::memcached on deployment-memc04, deployment-memc06, and deployment-memc07 for [[phab:T210030|T210030]]
* 01:22 bd808: Applied role::mediawiki::memcached on deployment-memc05.deployment-prep.eqiad.wmflabs to provision redis [[phab:T210030|T210030]]
== 2018-11-20 ==
* 23:45 mutante: deployment-deploy01 edited /srv/deployment/iegreview/iegreview/.git/DEPLOY_HEAD - - replaced deployment-tin with deployment-deploy1 to fix scap cloning / puppet
* 23:01 mutante: deployment-deploy01 edited /srv/deployment/scholarships/scholarships/.git - replaced deployment-tin with deployment-deploy1 to fix scap / cloning of scholarships app
* 22:42 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
* 20:23 Krenair: Changed deployment-prep mysql repl password in attempt to get replication working again, have stored it at deployment-puppetmaster03:/var/lib/git/labs/private/modules/secret/secrets/mysql/repl_password
* 18:18 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@7553087]
* 14:59 hashar: created integration-castor03.integration.eqiad.wmflabs intended as a replacement for castor02 {{!}} [[phab:T208803|T208803]]
* 14:34 hashar: beta-scap job fixed by deleting a file on deployment-mediawiki-07
* 14:27 hashar: deployment-mediawiki-07 :  sudo rm -fR /srv/mediawiki/.~tmp~/
* 13:55 andrewbogott: deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms"
* 13:08 twentyafterfour: removed local unix user mwdeploy from deployment-mediawiki-07 because it was shadowing the real mwdeploy user in ldap
* 13:01 twentyafterfour: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/luasandbox.so' - /usr/lib/php/20151012/luasandbox.so: cannot open shared object file: No such file or directory [[phab:T208101|T208101]]
* 12:11 twentyafterfour: scap failures on deployment-mediawiki-07 are related to uid/gid mismatch of the mwdeploy user, specifically the owner of that user's home dir is uid 603 but /etc/passwd{{!}}group have a different uid/gid for the same username. [[phab:T208101|T208101]]
* 11:47 hashar: Armed keyholder on deployment-deploy01 Got shutdown while being migrated a new cloud region # [[phab:T208101|T208101]]
* 11:00 hashar: deployment-deploy01 got migrated to a new region but the Jenkins configuration had not been updated. Adjusting IP address from 10.68.23.38 to 172.16.4.18 {{!}} [[phab:T208101|T208101]]
* 10:58 hashar: Clearing out deployment-deploy01 disk space. Went offline due to disk space consumption
* 10:04 Krenair: manually fixed deployment-mediawiki-09:/srv/mediawiki/wmf-config/db-labs.php to match deployment copy, not sure why it didn't deploy properly yet
== 2018-11-19 ==
* 22:40 Krenair: manually sorted certs for deployment-puppetdb02
* 22:23 Krenair: duplicated wiki(user{{!}}admin) mysql users on deployment-db0[34] - previous hosts 10.%, new hosts 172.16.%
* 18:32 Krenair: creating deployment-cache-text05 to replace deployment-cache-text04
* 15:53 Amir1: cherry-picking gerrit:474694/1 in beta puppetmaster
* 15:06 Amir1: ores:e957b24 is going beta
* 13:47 hashar: Shutdown integration-publishing , replaced by integration-publishing02 # [[phab:T208803|T208803]]
* 13:21 hashar: Created integration-publishing02 172.16.4.5 for WMCS region migration # [[phab:T208803|T208803]]
* 11:13 hashar: updating jobs  wikidata-query-rdf-maven-java8-docker wikidata-query-rdf-maven-java8-docker-site-publish for https://gerrit.wikimedia.org/r/#/c/integration/config/+/474660/
* 10:23 gehel: Updating docker-pkg files on contint1001 for wikdiata-query-rdf image
== 2018-11-17 ==
* 06:26 legoktm: deployed https://gerrit.wikimedia.org/r/474415
== 2018-11-16 ==
* 21:01 legoktm: deploying https://gerrit.wikimedia.org/r/473718
* 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/473671 https://gerrit.wikimedia.org/r/474330
* 20:36 legoktm: deployed https://gerrit.wikimedia.org/r/474306
* 17:37 hashar: Added integration-slave-docker-1046 and integration-slave-docker-1047
* 17:20 hashar: Deleted integration-slave-docker-1045 (wrong OS stretch, must be jessie)
* 17:19 hashar: Added two more Jenkins docker slaves: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1044/ and https://integration.wikimedia.org/ci/computer/integration-slave-docker-1045/
* 14:35 hashar: Building docker image releng/java8:0.4.2
* 13:33 hashar: building docker image releng/java8:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/472661/
== 2018-11-15 ==
* 21:09 hashar: Last Nodepool instance had id {{Gerrit|1099516}} (yeah more than a million)
* 17:15 Amir1: ores:51cdf6b is going beta
== 2018-11-14 ==
* 18:48 andrewbogott: moving deployment-mediawiki-07 to labvirt1008
* 18:31 andrewbogott: moving deployment-chromium01 to labvirt1009
* 18:06 andrewbogott: moving deployment-mx02 to labvirt1003
* 18:05 andrewbogott: migrating deployment-snapshot01 to labvirt1001
* 10:39 gtirloni: added gtirloni as projectadmin in deployment-prep project
== 2018-11-13 ==
* 22:39 thcipriani: reenabling and running puppet on deployment-deploy01
* 22:19 andrewbogott: moving deployment-urldownloader02 to labvirt1012
* 22:02 thcipriani: disable puppet on deployment-deploy01 temporarily while deployment-deploy02 is migrating to preserve dsh files
* 21:59 andrewbogott: moving deployment-deploy02 to another labvirt
* 21:55 andrewbogott: moving deployment-webperf12 to a new labvirt
* 21:50 andrewbogott: moving deployment-dumps-puppetmaster02 to a new labvirt
* 21:43 andrewbogott: moving deployment-elastic05 to a new labvirt to clear out labvirt1016
* 13:01 arturo: a puppet refactor for the aptly module may have caused some puppet issues. Should be solved now
* 09:31 addshore: manually brought integration-slave-docker-1021 back online
* 09:29 addshore: integration-slave-docker-1021:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '{print $3}')
* 09:26 addshore: integration-slave-docker-1021:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-5] months " {{!}} awk '{print $3}')
== 2018-11-12 ==
* 12:30 hashar: mass purging legacy Jenkins jobs from contint1001. They were still deployed but no more in JJB, that causes the logs / artifacts to be left behind. {{!}} [[phab:T209123|T209123]]
* 11:26 hashar: building container releng/operations-puppet:0.5.1 to bump mtail version ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/472960/ by godog )
== 2018-11-11 ==
* 23:47 addshore: manually set quibble-vendor-mysql-hhvm-docker timeout to be 60 (i should really get my jjb stuff working again)
== 2018-11-09 ==
* 13:43 Amir1: ores:0728805 is going beta
* 12:13 kart_: Update cxserver to {{Gerrit|01686f6}}
== 2018-11-08 ==
* 22:55 thcipriani: contint1001: cleanup with (essentially, but not exactly) https://phabricator.wikimedia.org/P7781
* 22:21 thcipriani: contint1001:sudo find /srv/jenkins/builds -maxdepth 2 -mindepth 2 -type d -regex '.*/[1-9][0-9]+' -mtime +500 -exec rm -r "{}" \;
* 21:50 thcipriani: contint1001:rm apps-android-wikipedia-test builds > 30 days old
* 15:42 Krenair: dropped https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/434055/ cherry-pick and rebased to fix puppet updates
* 15:37 addshore: for https://gerrit.wikimedia.org/r/472476
* 15:36 addshore: addshore@deployment-deploy01:~$ sudo puppet agent -tv
* 14:30 hashar: updated search-highlighter-maven-java8-docker-site-publish to decouple install and site goals {{!}} [[phab:T208496|T208496]]
== 2018-11-07 ==
* 23:00 thcipriani: repooling integration-slave-docker-1017 after cleaning up docker images
* 19:09 Amir1: ores:25dfa4f is going to beta cluster
* 09:06 hashar: building container releng/operations-puppet:0.5.0  for python3 {{!}} [[phab:T208873|T208873]]
== 2018-11-06 ==
* 19:56 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/472044
* 15:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/blubber/+/471277/
== 2018-11-04 ==
* 17:53 legoktm: deploying https://gerrit.wikimedia.org/r/471251
* 17:50 legoktm: deployed https://gerrit.wikimedia.org/r/471283
== 2018-11-02 ==
* 19:54 Amir1: stopping redis-server on deployment-ores01, this would make ores go down in beta cluster. Testing for [[phab:T181632|T181632]]
* 14:04 Krenair: made onimisionipe a projectadmin per request in -cloud
* 02:30 legoktm: deploying https://gerrit.wikimedia.org/r/471198
* 00:38 legoktm: deploying https://gerrit.wikimedia.org/r/471186 https://gerrit.wikimedia.org/r/471181
== 2018-11-01 ==
* 17:12 legoktm: deploying https://gerrit.wikimedia.org/r/471047
* 15:55 legoktm: deploying https://gerrit.wikimedia.org/r/471033 https://gerrit.wikimedia.org/r/471034
* 15:41 legoktm: deploying https://gerrit.wikimedia.org/r/471032
* 15:29 legoktm: deploying https://gerrit.wikimedia.org/r/471029
* 15:11 legoktm: building docker for https://gerrit.wikimedia.org/r/471025
* 14:48 andrewbogott: moving deployment-redis05  to labvirt1012
* 14:47 Krenair: shut off deployment-redis05 for migration to new physical host
* 14:45 legoktm: building mediawiki-tarball docker image https://gerrit.wikimedia.org/r/471019
== 2018-10-31 ==
* 21:16 Krenair: remove horizon hiera config for deployment-redis0[56] to unbreak puppet and remove old redis0[12] instance IPs [[phab:T208040|T208040]]
* 19:55 andrewbogott: moving deployment-elastic06 to labvirt1012
* 19:40 andrewbogott: moving deployment-cpjobqueue to labvirt1012 to help clear out labvirt1017
* 19:11 andrewbogott: moving deployment-kafka-jumbo-1 to labvirt1012 to help clear out labvirt1017
* 18:54 andrewbogott: moving deployment-kafka-main-2 to labvirt1012 to help clear out labvirt1017
* 17:21 Amir1: ores:70ba14b is going to beta
* 13:23 godog: enable statsd reporting for swift
== 2018-10-30 ==
* 16:51 hashar: building docker container releng/operations-puppet:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/470574/
* 14:48 addshore: beta [[phab:T208307|T208307]]: wikiadmin@10.68.23.30(wikidatawiki)> delete from wb_changes_dispatch where chd_site = 'specieswiki'
* 11:15 hashar: contint1001: cleaned up old Docker images
* 10:13 hashar: Building Docker container releng/operations-puppet:0.4.0 for https://gerrit.wikimedia.org/r/470419 by _joe_
== 2018-10-29 ==
* 22:56 legoktm: deployed https://gerrit.wikimedia.org/r/470453 https://gerrit.wikimedia.org/r/469954 https://gerrit.wikimedia.org/r/470241 https://gerrit.wikimedia.org/r/468900 https://gerrit.wikimedia.org/r/469181
* 22:54 James_F: Manually triggered a beta-update-databases-eqiad run to see if our fix for [[phab:T208043|T208043]] has worked.
* 10:04 hashar: cleaned disk on integration-slave-jessie-1002
* 04:07 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 00:44 legoktm: also deploying https://gerrit.wikimedia.org/r/470059
* 00:44 legoktm: deploying https://gerrit.wikimedia.org/r/469948
== 2018-10-26 ==
* 09:04 hashar: jenkins: update IRC config at https://integration.wikimedia.org/ci/configure to replace #wikimedia-ri-ci by #wikimedia-infrastructure ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/469034/ )
== 2018-10-25 ==
* 17:21 bearND: (beta): Update mobileapps to {{Gerrit|58cbdff}}
* 07:50 hashar: enabling puppet again on deployment-deploy01 . Was disabled by _joe_ for apache-fast-test hacking
== 2018-10-24 ==
* 20:44 hasharDinner: Rebuilding CI containers for Quibble 0.0.28
* 20:27 hasharDinner: tagged Quibble 0.0.28 at {{Gerrit|1ac8fe353a8b7bbf5b31e6094ef06ee459c553a3}}
* 08:57 hashar: gerrit: added Lars "liw" Wirzenius to the Administrators group {{!}} [[phab:T207830|T207830]]
* 07:27 Krenair: [[phab:T207825|T207825]] reapplied role::jobqueue_redis::master to deployment-redis prefix
* 07:01 Krenair: [[phab:T207825|T207825]] replacing deployment-redis3-changeprop with deployment-redis3-changeprop02 (jessie m1.small)
* 06:59 Krenair: [[phab:T207825|T207825]] moved role::jobqueue_redis::master role from deployment-redis prefix to deployment-redis0[56]
* 05:22 kart_: Beta: Updated cxserver to {{Gerrit|9ad60d9}}
== 2018-10-23 ==
* 23:59 legoktm: deploying https://gerrit.wikimedia.org/r/468171
== 2018-10-22 ==
* 13:01 kart_: Beta: Update cxserver to {{Gerrit|7f996f3}}
* 09:17 _joe_: deployment-prep: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/468583/, disable puppet on deployment-mediawiki-09
* 00:27 Krenair: Added gtirloni as a member per [[phab:T207474|T207474]] - I imagine he'll want to get in to look at shinken-related things
== 2018-10-21 ==
* 21:27 Krenair: rm'd scap l10n stuff on deployment-deploy01 as usual
* 12:25 legoktm: deployed https://gerrit.wikimedia.org/r/468816
== 2018-10-19 ==
* 11:36 hashar: nuking some caches on castor02.integration.eqiad.wmflabs to free up disk space
* 07:08 legoktm: also deploying https://gerrit.wikimedia.org/r/c/integration/config/+/468022
* 07:07 legoktm: deploying https://gerrit.wikimedia.org/r/467329 https://gerrit.wikimedia.org/r/468417 https://gerrit.wikimedia.org/r/468393 https://gerrit.wikimedia.org/r/467724
== 2018-10-18 ==
* 13:27 Amir1: deploying ores:d724d20
== 2018-10-16 ==
* 19:09 twentyafterfour: Branching 1.32.0-wmf.26
* 19:08 legoktm: integration-slave-docker-1017 back online
* 19:07 legoktm: root@integration-slave-docker-1017:/# docker rmi $(docker images -q) to clean up disk space
* 04:22 legoktm: deploying https://gerrit.wikimedia.org/r/467563
* 04:14 legoktm: deploying https://gerrit.wikimedia.org/r/467562
* 04:04 legoktm: deploying https://gerrit.wikimedia.org/r/467560
* 03:06 Krinkle: Rebooted deployment-deploy01 Jenkins slave (stuck for 8 hours)
== 2018-10-15 ==
* 22:28 legoktm: deployed https://gerrit.wikimedia.org/r/467536
* 22:20 legoktm: fixed disk space issue with integration-slave-jessie-1004 and brought it back online
* 22:16 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 22:09 legoktm: triggering jobs on contint1001 via gear_client, kill if necessary
* 22:03 legoktm: building Quibble PHP 7.3 images (https://gerrit.wikimedia.org/r/467259)
* 20:31 bearND: Beta cluster: Update mobileapps to {{Gerrit|c2a4ef9}}
* 19:53 legoktm: rebuilding npm* docker images for https://gerrit.wikimedia.org/r/467451
* 18:13 legoktm: deploying https://gerrit.wikimedia.org/r/467434
* 17:56 legoktm: building php73(-compile)? images https://gerrit.wikimedia.org/r/465204
* 15:14 thcipriani: bring integration-slave-jessie-1001 back online
* 13:30 hashar: Cleaned up disk space on integration-slave-jessie-1001
* 05:10 kart_: Update cxserver to {{Gerrit|b51f363}}
* 04:00 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n*
== 2018-10-13 ==
* 04:45 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/467023
* 04:34 legoktm: deploying https://gerrit.wikimedia.org/r/467022
* 03:20 Krinkle: Fixed console section pattern for "setup quibble mw-update"
* 03:19 Krinkle: Fixed console section pattern for "setup quibble npm-ls"
* 03:04 legoktm: deploying https://gerrit.wikimedia.org/r/467016
* 02:15 legoktm: deploying https://gerrit.wikimedia.org/r/467012
* 01:55 legoktm: building quibble-stretch-php72 image https://gerrit.wikimedia.org/r/466991
* 01:25 legoktm: rolling out https://gerrit.wikimedia.org/r/c/integration/config/+/466950
== 2018-10-12 ==
* 21:13 legoktm: deployed https://gerrit.wikimedia.org/r/465049
* 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=simplewiki --quick
* 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=ruwiki --quick
* 20:14 mutante: contint1001 - gzip merge-debug.log.2018-*  debug.log.2018-* etc.. all in /var/log/zuul/  there seems to be no compression (but max age is 1 month)
* 20:11 mutante: contint1001 - gzip zuul.log.2018-* in /var/log/ to prevent running out of disk
* 20:08 mutante: contint1001 - apt-get clean - frees 1GB to bring it _just_ under the Icinga warning threshold :p
* 19:23 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/465606
* 17:18 marxarelli: bringing integration-slave-docker-1021 back online
* 17:16 marxarelli: killing 7 long-running containers (> 1 hour) on docker integration nodes ([[phab:T198517|T198517]])
* 16:42 marxarelli: killing 9 long-running containers (> 1 hour) on docker integration nodes ([[phab:T198517|T198517]])
* 16:31 marxarelli: taking integration-slave-docker-1021 offline (host is unresponsive)
* 16:25 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 321 affected jobs ([[phab:T198517|T198517]])
* 16:05 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 1 job mediawiki-quibble-vendor-mysql-php70-docker ([[phab:T198517|T198517]])
* 15:30 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 320 affected jobs
* 15:26 marxarelli: killing 51 long-running containers (> 1 hour) on docker integration nodes ([[phab:T198517|T198517]])
* 13:22 Reedy: running foreachwiki update.pph
* 11:48 Amir1: ores:ba4ea40 is going beta
* 05:14 legoktm: deployed https://gerrit.wikimedia.org/r/466821
* 04:40 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/466812
* 03:52 legoktm: deployed https://gerrit.wikimedia.org/r/466811
* 03:46 legoktm: tagged 0.0.27 of quibble ({{Gerrit|ec8d280de69cda798d95183a4e3f480b45bc9eba}})
== 2018-10-11 ==
* 23:50 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 (full deployment delayed until around 1600 tomorrow)
* 23:45 marxarelli: killed 6 long-running containers (> 3 hours) on docker integration nodes ([[phab:T198517|T198517]])
* 23:34 marxarelli: redeploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
* 23:14 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job
* 23:08 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
* 19:28 thcipriani: integration-slave-jessie-1003 back online
* 19:28 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
* 18:18 Amir1: ores:1a5630f is going beta
* 18:17 marxarelli: killed 42 long-running (> 3 hours) containers on docker integration nodes ([[phab:T198517|T198517]])
* 18:16 marxarelli: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to all 320 affected jobs
* 17:47 marxarelli: killing long-running containers (> 3 hours) on docker integration nodes ([[phab:T198517|T198517]])
* 17:37 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to 8 wmf-quibble-* jobs
* 02:03 Krinkle: Fix Jenkins' section pattern for 'setup-docker-quibble-castor-load'
* 02:03 Krinkle: Fix Jenkins' collapsible sections for "npm-install" and "mw-install" under quibble to match current patterns
== 2018-10-10 ==
* 19:38 awight: restarted ORES celery workers on ores2003 (~17:00), ores200* (17:05)
* 17:15 twentyafterfour: cowboy-coding on deployment-deploy01 to solve scap fatal error dcheck failing to catch fatals
== 2018-10-09 ==
* 19:00 Krinkle: Re-enable beta-scap-eqiad job
* 18:19 Krinkle: Messing with scap in beta to test [[phab:T121597|T121597]] / D1114
* 15:32 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
== 2018-10-08 ==
* 03:31 kart_: Update(d) cxserver to {{Gerrit|47a864b}}
* 03:02 legoktm: onlined integration-slave-jessie-1002
* 02:20 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-10-06 ==
* 13:59 Reedy: cleared some large folders out of /tmp on deployment-deploy01
== 2018-10-05 ==
* 19:47 marxarelli: bringing integration-slave-docker-1040 back online
* 19:05 marxarelli: taking integration-docker-slave-1040 offline for docker daemon restart
* 19:04 marxarelli: bringing integration-slave-docker-1038/1041/1043 back online
* 19:02 marxarelli: taking integration-docker-slave-1038/1041/1043 offline for docker daemon restart
* 19:01 marxarelli: bringing integration-slave-docker-1033/1037 back online
* 18:58 marxarelli: taking integration-docker-slave-1033/1037 offline for docker daemon restart
* 18:56 marxarelli: bringing integration-slave-docker-1034 back online
* 18:56 marxarelli: integration-puppetmaster01:/var/lib/git/operations/puppet is up-to-date again after manually updating submodules and subsequent automated git-sync-upstream
* 18:33 marxarelli: taking integration-slave-docker-1034 offline for docker daemon restart
* 11:18 Krenair: rm -rf /tmp/scap_l10n_* on deployment-deploy01
* 06:00 legoktm: deployed https://gerrit.wikimedia.org/r/464757
* 04:35 legoktm: deploying https://gerrit.wikimedia.org/r/464747
* 01:17 legoktm: triggering php71 quibble jobs manually via contint1001
== 2018-10-04 ==
* 22:23 thcipriani: updated jenkins jobs: https://phabricator.wikimedia.org/P7634
* 22:19 thcipriani: updating 103 jenkins jobs that use ci-src-setup-simple
* 21:45 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/464719/
* 21:13 marxarelli: bringing integration-slave-jessie-1003 back online
* 21:12 marxarelli: deleted /mnt/home/jenkins-deploy/{.m2,.gradle} on integration-slave-jessie-1003
* 21:10 marxarelli: deleting cache directories in /mnt/home/jenkins-deploy on integration-slave-jessie-1003 to free up disk space
* 20:31 marxarelli: terminating long-running (> 3 hours) CI docker containers ([[phab:T198517|T198517]])
== 2018-10-03 ==
* 18:45 marxarelli: deploying {{Gerrit|I0fcd95427a6468213feb0be2ab7ab52f0d719f90}} for all 313 affected jobs
* 18:16 marxarelli: deploying {{Gerrit|I0fcd95427a6468213feb0be2ab7ab52f0d719f90}} for 12 mediawiki-quibble-vendor-* jobs
* 17:46 marxarelli: removing old workspaces on integration-slave-docker-1034 (`rm -rf /srv/jenkins-workspace/workspace/*`) and bringing back online
* 16:52 thcipriani: reloading zuul to deploy Change gate-and-submit-l10n to low precedence
* 15:46 marxarelli: bringing integration-slave-docker-1040/1043 nodes back online after killing long running docker job and freeing up /var/lib/docker space ([[phab:T206134|T206134]])
* 15:40 marxarelli: killing long running docker jobs on integration-slave-docker-1040/1043 which are filling up /var/lib/docker with log output
* 15:12 thcipriani: reloading zuul to deploy l10 pipeline
* 14:53 thcipriani: integration-slave-docker-1038 removed workspaces, brought back online integration-slave-docker-104{0,3} need more investigation since docker is full(?)
* 07:07 mdholloway: deployment-maps04 deployed [kartotherian/deploy@27062b4]: Specify WDQS endpoint in the service config ([[phab:T205607|T205607]])
* 06:22 _joe_: cherry-picking 455154 (php-fpm installation) on deployment-prep
== 2018-10-02 ==
* 21:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/464048/
* 18:22 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 18:15 legoktm: deploying https://gerrit.wikimedia.org/r/464000
* 16:45 thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*
* 16:20 thcipriani: investigating integration-slave-docker-1037
* 08:56 godog: bounce logstash
* 07:59 legoktm: deployed https://gerrit.wikimedia.org/r/463905
* 07:38 legoktm: building quibble-stretch-php71 docker image
* 05:32 legoktm: rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883
* 03:26 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886
== 2018-10-01 ==
* 20:09 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space
* 17:13 marxarelli: bringing integration-slave-docker-1041 back online following source directory clean up ([[phab:T205902|T205902]])
* 16:52 marxarelli: removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041
* 07:10 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest
* 05:24 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to {{Gerrit|a1fa41b}}
== 2018-09-29 ==
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463615
* 01:22 Krinkle: Marking integration-slave-docker-1041 as offline (again). Why/How did it come back?
* 00:24 Krinkle: Marking integration-slave-docker-1041 as offline. Various odd build failures, including https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/6913/console
== 2018-09-28 ==
* 15:31 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki  --prefix ([[phab:T201009|T201009]])
* 14:50 thcipriani: investigating integration-slave-docker-1041
* 07:35 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@bf09080]: Update mobileapps to {{Gerrit|7878ffc}}
== 2018-09-27 ==
* 17:53 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@a0054ba]: Update mobileapps to {{Gerrit|0d6c2b7}}
* 07:38 mdholloway: deployment-maps04 updated tilerator and kartotherian node modules ([[phab:T195513|T195513]], [[phab:T200594|T200594]])
== 2018-09-26 ==
* 15:13 thcipriani: integration-slave-docker-1034:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online -- https://phabricator.wikimedia.org/P7592
* 15:05 thcipriani: integration-slave-docker-1033:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 14:47 thcipriani: investigating integration-slave-docker-103{3,4}
* 11:57 Amir1: [[gerrit:462927]] (ores) is going to beta
* 08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
* 08:14 hashar: Restarting CI Jenkins on contint1001
== 2018-09-25 ==
* 23:01 marxarelli: configured new jenkins node integration-slave-docker-1043 with 6 executors
* 23:01 marxarelli: replaced integration-slave-docker-1042 with new integration-slave-docker-1043 instance
* 22:39 marxarelli: launching new integration-slave-docker-1042 bigram instance
* 22:33 marxarelli: deleting remaining m1.medium instances used as m4executors ([[phab:T205362|T205362]])
* 22:15 marxarelli: taking remaining m1.medium m4executor jenkins nodes offline ([[phab:T205362|T205362]])
* 18:16 marxarelli: reconfiguring bigram jenkins nodes to use 6 executors. 7 were configured by mistake ([[phab:T205362|T205362]])
* 18:00 marxarelli: configuring new integration-slave-docker-1041 jenkins node with 7 executors ([[phab:T205362|T205362]])
* 17:42 marxarelli: configuring new jenkins node integration-slave-docker-1040 with 7 executors ([[phab:T205362|T205362]])
* 17:38 marxarelli: launching integration-slave-docker-1041 bigram instance ([[phab:T205362|T205362]])
* 17:30 marxarelli: the puppet parameter for docker_lvm_volume specified in horizon was not applied correctly on the first puppet run for some reason. tearing down integration-slave-docker-1039...
* 17:25 marxarelli: launching integration-slave-docker-1040 bigram instance ([[phab:T205362|T205362]])
* 17:24 marxarelli: deleting instances integration-slave-docker-1007/1008 ([[phab:T205362|T205362]])
* 17:13 marxarelli: launching new integration-slave-docker-1039 bigram instance
* 17:12 marxarelli: taking integration-slave-docker-1007/1008 offline for replacement ([[phab:T205362|T205362]])
* 17:09 marxarelli: deleting integration-slave-docker-1030/1031 instances ([[phab:T205362|T205362]])
* 17:05 marxarelli: taking integration-slave-docker-1030/1031 offline for replacement
* 16:47 marxarelli: increasing executors to 7 for jenkins nodes integration-slave-docker-1033/1034
* 16:46 marxarelli: new instance creation delayed due to quota
* 16:45 marxarelli: launching new integration-slave-docker-1039/1040 bigram instances
* 01:21 legoktm: deployed https://gerrit.wikimedia.org/r/450508
* 00:36 legoktm: deploying https://gerrit.wikimedia.org/r/462609
* 00:22 legoktm: deploying https://gerrit.wikimedia.org/r/453447
== 2018-09-24 ==
* 20:21 bearND: (beta): Update mobileapps to {{Gerrit|badb463}}
* 10:55 hashar: gerrit: granting labs/tools/* project owners the ability to submit changes {{!}} https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/
* 09:51 hashar: deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo  and created a new. Its size of 69632 bytes might cause slow writes?? {{!}} [[phab:T204762|T204762]]
* 09:24 hashar: Live hacked scap code on deployment-deploy01 for [[phab:T204762|T204762]] and reverted hack changes
* 08:32 hashar: deployment-deploy01 rm -fR /tmp/scap_l10n_*
* 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/462341
* 03:45 kart_: Update cxserver to {{Gerrit|d913793}}
== 2018-09-23 ==
* 14:03 Krenair: rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining
* 01:05 andrewbogott: rebooted deployment-maps03; OOM and also [[phab:T205195|T205195]]
== 2018-09-22 ==
* 20:51 Hauskatze: github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for [[phab:T183891|T183891]]
* 20:05 Hauskatze: github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer {{!}} [[phab:T183891|T183891]]; moving to the next one
* 18:21 Krenair: went to do the same with deployment-maps03 and accidentally broke SSH access to the server
* 18:21 Krenair: removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced [[phab:T153468|T153468]], this just explains why puppet hadn't purged stuff)
* 18:01 Krenair: rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
* 18:00 Krenair: rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
== 2018-09-21 ==
* 17:26 marxarelli: adding jenkins node integration-slave-docker-1038 with 7 executors
* 16:47 marxarelli: added new jenkins node integration-slave-docker-1037 with 7 executors
* 15:49 marxarelli: replacing integration-slave-docker-1036 with new bigram instance
* 15:48 marxarelli: taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations
* 15:17 marxarelli: integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations
* 15:15 marxarelli: taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend
* 15:13 marxarelli: launching integration-slave-docker-1037 bigram instance
* 13:03 Amir1: ores:7b987a7 is going beta
* 05:32 legoktm: deployed https://gerrit.wikimedia.org/r/461510
== 2018-09-20 ==
* 23:48 marxarelli: adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors
* 23:23 marxarelli: launching integration-slave-docker-1035/1036 bigram instances
* 23:20 marxarelli: taking integration-slave-docker-1004/1005 offline for replacement ([[phab:T202160|T202160]])
* 16:52 Amir1: deploy ores:ee2d28b
* 11:21 hashar: Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. {{!}} [[phab:T203181|T203181]] and [[phab:T204919|T204919]]
* 11:17 hashar: deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes)
* 11:07 hashar: deployment-deploy01 is out of disk space (again)
== 2018-09-19 ==
* 21:18 Hauskatze: github: deleted https://github.com/wikimedia/mediawiki-services-ocg-collection {{!}} [[phab:T183891|T183891]]
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|a224e99}}
* 18:56 Amir1: ores:76fe25a goes to beta ([[phab:T204862|T204862]])
== 2018-09-18 ==
* 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
* 09:46 hashar: updating mwselenium-quibble-docker to Quibble 0.0.26
* 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
* 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins  # untouched since 2016
* 07:31 hashar: cleaning disk on deployment-sca01
* 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors
== 2018-09-17 ==
* 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
* 22:00 Reedy: update.php in screen is done
* 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
* 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
* 20:17 bearND: (beta): Update mobileapps to {{Gerrit|d56e4cf}}
* 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy {{Gerrit|I5df2b72fa411adf05027c8f6745284e7e1dedf7a}}
* 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
* 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest ([[phab:T109776|T109776]])
* 13:24 hashar: Building docker images for Quibble 0.0.25 {{!}} https://gerrit.wikimedia.org/r/460536
* 12:58 hashar: mwselenium-quibble-docker to Chromium 69 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ {{!}} [[phab:T204214|T204214]]
* 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
* 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
* 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
* 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775
== 2018-09-16 ==
* 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
* 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism {{!}} [[phab:T200856|T200856]]
* 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> {{!}} [[phab:T204372|T204372]]
== 2018-09-14 ==
* 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
* 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
* 14:16 hashar: deployment-deploy01:  sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
* 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
* 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
* 13:48 hashar: Tagged Quibble 0.0.25 {{Gerrit|01663f5d9bcc9e77473732a9fd230271bbd43db4}}
* 11:49 hashar: Cleaned /tmp on deployment-deploy01
* 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ {{!}} [[phab:T204340|T204340]]
* 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
* 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473
== 2018-09-13 ==
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
* 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
* 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor  mediawiki/vendor fails composer --ansi validate --no-check-publish
* 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 {{!}} [[phab:T203902|T203902]]
* 08:42 hashar: aborted maintenance-disconnect-full-disks job {{!}} [[phab:T204077|T204077]]
* 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
* 05:13 legoktm: onlined integration-slave-docker-1021
* 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
* 05:08 legoktm: deployed
* 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184
== 2018-09-12 ==
* 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
* 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
* 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
* 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
* 20:51 marxarelli: deleting integration-slave-docker-1002 instance
* 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
* 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
* 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
* 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
* 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
* 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
* 18:21 marxarelli: deleting integration-slave-docker-1027 instance
* 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
* 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
* 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
* 17:59 marxarelli: deleting integration-slave-docker-1001 instance
* 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
* 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
* 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
* 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
* 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
* 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
* 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker  for Chromium 69 and Firefox 60 - [[phab:T203902|T203902]]
* 00:14 marxarelli: deleting instance integration-slave-docker-1029
== 2018-09-11 ==
* 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
* 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
* 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
* 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
* 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
* 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*'  # [[phab:T188742|T188742]]
== 2018-09-10 ==
* 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
* 08:34 Amir1: ores {{Gerrit|ae96071}} is going to beta
== 2018-09-09 ==
* 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
* 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
* 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265
== 2018-09-08 ==
* 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
* 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
* 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
* 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
* 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
* 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882
== 2018-09-07 ==
* 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
* 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
* 18:30 legoktm: started gear_client on contint1001
* 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
* 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
* 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4.  8 CPUS can not sustain 5 concurrent Quibble builds {{!}} [[phab:T201972|T201972]]
== 2018-09-06 ==
* 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
* 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
* 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill {{Gerrit|eae9ba3a1459}} -> stopped a container that had been running for 5 hours
* 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
* 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026
== 2018-09-05 ==
* 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
* 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
* 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
* 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095
== 2018-09-04 ==
* 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for [[phab:T203431|T203431]]
* 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
* 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808
== 2018-09-03 ==
* 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
* 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
* 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
* 07:54 hashar: deployment-deploy01 scap sync-l10n master
* 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
* 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
* 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
* 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
* 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278
== 2018-09-02 ==
* 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv {{!}} attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
* 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ {{!}} more permissions fixes for beta-scap-eqiad
* 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
* 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap {{!}} fixing beta-scap-update-equiad failures
== 2018-09-01 ==
* 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
* 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
* 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
* 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
* 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
* 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
* 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
== 2018-08-31 ==
* 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
* 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
* 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
* 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
* 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
* 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
* 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528
== 2018-08-30 ==
* 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
* 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for [[phab:T199417|T199417]]
* 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for [[phab:T199422|T199422]]
* 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
* 19:38 thcipriani: bring integration-slave-docker-1025 back online
* 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per [[phab:T197918|T197918]]
* 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
* 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
* 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
* 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
* 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
* 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073
== 2018-08-29 ==
* 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ ([[phab:T203093|T203093]])
* 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189  ([[phab:T203110|T203110]])
* 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
* 12:35 hashar: fixed keyholder on deployment-deploy01
* 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002
== 2018-08-28 ==
* 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ {{!}} [[phab:T203009|T203009]]
* 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 {{!}} [[phab:T203009|T203009]]
* 17:30 legoktm: reverted back to previous quibble version ([[phab:T203009|T203009]])
* 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 {{!}} https://gerrit.wikimedia.org/r/455861
* 16:13 hashar: building docker containers for quibble 0.0.22
* 16:06 hashar: tagged quibble 0.0.22 {{Gerrit|5093492821693918da4fecdcb15a8ca831ce560f}}
== 2018-08-27 ==
* 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
* 18:49 Amir1: ores:0e8cc73 is going to beta cluster
* 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ {{!}} [[phab:T179963|T179963]]
* 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian {{!}} [[phab:T174338|T174338]]
== 2018-08-25 ==
* 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
* 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
* 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107
== 2018-08-24 ==
* 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
* 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
* 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO
== 2018-08-23 ==
* 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
* 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
* 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
* 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
* 11:06 _joe_: reverting my cherry-pick
* 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
* 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge
== 2018-08-22 ==
* 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
* 20:05 bearND: (beta): Update mobileapps to {{Gerrit|141ff20}} ([[phab:T202105|T202105]] [[phab:T202237|T202237]])
* 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
* 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport [[phab:T202220|T202220]]
* 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
* 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
* 05:37 TimStarling: on deployment-db03 installing pt-heartbeat
== 2018-08-21 ==
* 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
* 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms ([[phab:T202260|T202260]])
* 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used)  rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* {{!}} [[phab:T202341|T202341]]
* 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
* 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
* 01:13 legoktm: triggering seccheck jobs manually on contint1001
== 2018-08-20 ==
* 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
* 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
* 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
* 20:16 mateusbs17: (beta): Update mobileapps to {{Gerrit|95e976d}} ([[phab:T202105|T202105]])
* 19:53 marxarelli: Deleting instance integration-slave-docker-1016
* 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
* 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
* 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
* 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
* 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced {{!}} [[phab:T192561|T192561]]
* 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
* 01:42 legoktm: running jobs manually on contint1001 with gear_client
* 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
* 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
* 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884
== 2018-08-18 ==
* 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04
== 2018-08-17 ==
* 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
* 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
* 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
* 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
* 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
* 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
* 08:46 addshore: reloaded zuul
* 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618
== 2018-08-16 ==
* 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to {{Gerrit|a808c9d}} ([[phab:T201979|T201979]])
== 2018-08-15 ==
* 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for [[phab:T201782|T201782]].
* 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins
== 2018-08-14 ==
* 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
* 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
* 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
* 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
* 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2
== 2018-08-13 ==
* {{safesubst:SAL entry|1=23:05 Krenair: deactivated and cleaned puppet node entries for deployment-{tin,mira} T192561}}
* 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
* 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
* 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......
== 2018-08-12 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
* 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
* 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
* 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
* 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
* 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179
== 2018-08-10 ==
* 17:58 marxarelli: Reloading zuul to deploy {{Gerrit|I32007b56d05c80aadefa686c505e1da23c343dc1}}
* 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins
== 2018-08-09 ==
* 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
* 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
* 17:13 bearND: (beta): Update mobileapps to {{Gerrit|616ffef}}
* 01:42 awight: [[phab:T201518|T201518]]: ORES, fawiki wp10, misc updates
== 2018-08-08 ==
* 20:39 James_F: Updated mobileapps to {{Gerrit|162ebcf}} in Beta Cluster
* 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
* 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055  / [[phab:T199850|T199850]]
== 2018-08-07 ==
* 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / [[phab:T201107|T201107]])
* 19:52 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/450644
* 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005
== 2018-08-06 ==
* 16:11 addshore: reload zuul for  https://gerrit.wikimedia.org/r/450604
* 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
* 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514
== 2018-08-04 ==
* 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296
== 2018-08-02 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
* 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
* 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it ([[phab:T201077|T201077]])
* 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
* 14:07 gehel: re-imaging depoyment-elastic* to stretch
* 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py
== 2018-08-01 ==
* 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
* 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones {{!}} [[phab:T197469|T197469]]
* 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 ([[phab:T199801|T199801]])
* 10:08 dcausse: deployment-prep [[phab:T199270|T199270]]: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01
== 2018-07-31 ==
* 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
* 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
* 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor .  We use 'es6' branch instead.  (made for dcausse )
* 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399
== 2018-07-30 ==
* 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
* 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:03 Krinkle: Delete and recreate deployment-webperf13 ([[phab:T195312|T195312]] / [[phab:T180761|T180761]])
* 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
* 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value ([[phab:T200634|T200634]])
* 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/  {{!}} [[phab:T195918|T195918]]
* 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, [[phab:T195709|T195709]]
* 17:04 hashar: github: granted @thcipriani owner rights
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087
== 2018-07-29 ==
* 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002
== 2018-07-28 ==
* 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
* 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # [[phab:T197698|T197698]]
== 2018-07-27 ==
* 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
* 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container {{!}} [[phab:T195918|T195918]]
* 14:43 hashar: Building tox-certcentral container for vgutierrez !
* 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
* 12:05 hashar: Building releng/quibble images 0.0.21
* 11:40 hashar: Tagged quibble 0.0.21  {{Gerrit|e681dce021e}}
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ [[phab:T200508|T200508]]
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
* 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337
== 2018-07-26 ==
* 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
* 15:03 zeljkof: Reloading Zuul to deploy {{Gerrit|208a505b5d2d864e57924325ca7665cf1231c8dc}}
* 04:00 kart_: Updated cxserver to {{Gerrit|e98c81b}}
== 2018-07-24 ==
* 05:22 kart_: Updated cxserver to {{Gerrit|d3c9d15}}
== 2018-07-23 ==
* 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to {{Gerrit|254cef5}}
* 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
* 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, [[phab:T195312|T195312]]
* 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 [[phab:T195312|T195312]]
* 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / [[phab:T195312|T195312]]
* 15:34 Krinkle: Creating deployment-webperf13 - [[phab:T195312|T195312]]
* 15:34 Krinkle: Deleting deployment-webperf12 - [[phab:T195312|T195312]]
* 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
* 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB.  The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable
== 2018-07-19 ==
* 21:47 hashar: CI:  Force npm loglevel to 'info'  via a Zuul parameter {{!}} [[phab:T198348|T198348]]
* 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
* 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/#/c/446863 {{!}} [[phab:T198348|T198348]]
* 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/442989 {{!}} [[phab:T198348|T198348]]
== 2018-07-17 ==
* 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - [[phab:T195312|T195312]], [[phab:T180761|T180761]].
== 2018-07-16 ==
* 21:10 awight: ran namespaceDupes.php on beta enwiki
* 19:56 bearND: (beta): Update mobileapps to {{Gerrit|bed7b29}}
== 2018-07-14 ==
* 21:20 Krenair: redirected security_audit traffic (see [[phab:T72181|T72181]]) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in [[phab:T192996|T192996]])
* 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds -  [[phab:T179963|T179963]]
== 2018-07-13 ==
* 17:31 Reedy: Reloading Zuul to deploy stuff
== 2018-07-12 ==
* 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
* 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05
== 2018-07-11 ==
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|b5e152d}}
* 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
* 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch
== 2018-07-10 ==
* 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002  ( apt-get clean , apt-get autoremove --purge ,  nuked /srv/jenkins-workspace/workspace/*lint
* 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator {{!}} [[phab:T199103|T199103]]
* 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884  as well)
* 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in [[phab:T72792|T72792]])
* 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 {{!}} [[phab:T198171|T198171]]
== 2018-07-09 ==
* 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs ([[phab:T142152|T142152]])
* 22:09 Krinkle: Deleting instance deployment-apertium02 - [[phab:T142152|T142152]]
* 19:10 Krenair: deleted deployment-mediawiki06 [[phab:T192996|T192996]]
* 17:35 zeljkof: Reloading Zuul to deploy {{Gerrit|7967dad02843d3692abaf3acfd548047f192c9f6}}
* 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation {{!}} [[phab:T198952|T198952]]
* 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport {{!}} [[phab:T198957|T198957]]
* 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat {{!}} [[phab:T198955|T198955]]
* 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon {{!}} [[phab:T198864|T198864]]
== 2018-07-08 ==
* 20:17 Krinkle: Shutting off deployment-apertium02 ([[phab:T142152|T142152]])
* 18:31 Krenair: deleted deployment-mx [[phab:T184244|T184244]]
* 16:55 Krenair: deleted deployment-redis01 [[phab:T179371|T179371]]
* 16:54 Krenair: deleted deployment-redis02 [[phab:T179371|T179371]]
* 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/
== 2018-07-07 ==
* 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git
== 2018-07-06 ==
* 03:53 kart_: Updated cxserver to {{Gerrit|bfc9c84}}
== 2018-07-05 ==
* 06:27 kart_: Updated cxserver to {{Gerrit|f8c71a1}}
== 2018-07-04 ==
* 20:15 Reedy: beta unbroke beta code autodeploy [[phab:T72597|T72597]]
* 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ [[phab:T192996|T192996]]
* 05:31 Krinkle: beta-scap-eqiad  failing due to mediawiki06 still being a scap target  (despite being shutdown per [[phab:T192996|T192996]])
== 2018-07-03 ==
* 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
* 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
* 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev {{!}} [[phab:T198076|T198076]]
* 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 ([[phab:T195312|T195312]])
* 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod  ([[phab:T195312|T195312]], [[phab:T194390|T194390]])
* 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - [[phab:T192996|T192996]]
== 2018-07-02 ==
* 20:28 hashar: building docker image releng/tox:0.1.1  , that adds libssl-dev {{!}} [[phab:T198076|T198076]]
* 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub {{!}} [[phab:T198322|T198322]]
* 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023
== 2018-06-30 ==
* 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space
== 2018-06-28 ==
* 21:17 hasharAway: castor02: nuking cache of npm/node jobs via  rm -fR /srv/jenkins-workspace/caches/*/*/*node*  (note: other jobs might still have a npm cache) {{!}} [[phab:T198348|T198348]]
* 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm  and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm {{!}} [[phab:T198348|T198348]]
* 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker {{!}} [[phab:T198348|T198348]]
* 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.
== 2018-06-27 ==
* 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 {{!}} [[phab:T196346|T196346]] [[phab:T198336|T198336]]
* 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to {{Gerrit|d7221ba}}
== 2018-06-26 ==
* 22:17 Krenair: arming keyholder on deployment-deploy01
* 20:16 Krenair: done the same on -sca02 and -apertium02
* 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
* 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015  (converted to m1.medium instances)
* 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
* 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
* 13:18 hashar: cleaned containers on integration-slave-docker-1006
== 2018-06-25 ==
* 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to {{Gerrit|8c76d52}}
* 17:45 Krenair: fixed puppet repo rebasing
* 16:20 hashar: deployment-prep:  git gc  on a few repositories under /srv/mediawiki-staging/php-master
* 12:19 kart_: "Beta: Update cxserver to ece5e7a"
* 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice {{!}} [[phab:T196618|T196618]]
== 2018-06-23 ==
* 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up
== 2018-06-22 ==
* 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'    # some instances are causing disk warnings
* 14:46 zeljkof: Reloading Zuul to deploy {{Gerrit|94c38c16c14c59bf6ff2de6c9b693b9a903da1c2}}
* 12:34 hashar: Building docker images for Quibble 0.0.19 {{!}} [[phab:T197687|T197687]]
* 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium {{!}} [[phab:T197687|T197687]]
== 2018-06-21 ==
* 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor {{!}} [[phab:T197698|T197698]]
* 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2  https://gerrit.wikimedia.org/r/c/integration/config/+/441349/
== 2018-06-20 ==
* 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
* 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck {{!}} [[phab:T197806|T197806]]
* 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image {{!}} [[phab:T197806|T197806]]
== 2018-06-19 ==
* 15:01 hashar: CI docker slaves:  docker container prune --force ; docker image prune --force
* 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php
== 2018-06-18 ==
* 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:07 Amir1: change_tag_def is back now
* 16:05 Amir1: making enwiki in beta clsuter readonly
== 2018-06-15 ==
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|529965bb8c8cfa2bb7539c7c79c9dd55de6c11ac}}
* 08:30 hashar: github: deleted lot of archived/read-only extensions and skins {{!}} [[phab:T180864|T180864]]
* 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
* 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
* 07:51 hashar: cleaning disk space on integration-slave-jessie-1001
== 2018-06-14 ==
* 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
* 14:26 zeljkof: Reloading Zuul to deploy {{Gerrit|04b1c864f3b9d49e9b3882a1fc4c8fbf272c1bd7}}
* 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
* 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
* 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01{{!}} [[phab:T197207|T197207]]
== 2018-06-13 ==
* 19:47 Krenair: powered off deployment-mx [[phab:T184244|T184244]]
* 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera {{!}} TT186096
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027
== 2018-06-12 ==
* 17:40 halfak: deploying ores {{Gerrit|36037b6}}
* 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829
== 2018-06-11 ==
* 21:47 halfak: deploying ores {{Gerrit|6ee8775}}
* 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
* 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean {{!}} [[phab:T196430|T196430]]
* 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-06-10 ==
* 17:03 Krenair: also fixed scap install on -zotero01
* 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
* 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'
== 2018-06-09 ==
* 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from [[phab:T182927|T182927]] to test
* 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
* 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
* 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances [[phab:T179371|T179371]]
== 2018-06-08 ==
* 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
* 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn {{!}} [[phab:T196433|T196433]]
* 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration {{!}} [[phab:T196442|T196442]]
* 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces
== 2018-06-07 ==
* 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
* 17:24 Reedy: unstuck jenkins deployments to beta
* 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
* 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
* 14:16 _joe_: killed "puppet master" process in depoloyment-prep
* 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex {{!}} [[phab:T193844|T193844]]
* 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
* 10:41 hashar: Building Docker images for Quibble 0.0.18
* 09:15 addshore: added wmde-leszek to deployment-prep
== 2018-06-06 ==
* 21:40 bearND: BC: Update mobileapps to {{Gerrit|5ea008c}}
* 20:36 Amir1: deploying {{Gerrit|65ce165}} on ores
* 20:25 Amir1: deploying {{Gerrit|65e979f}} on ores
* 19:59 bearND: Update mobileapps to {{Gerrit|3bf9be5}} on BC
* 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) {{!}} [[phab:T196108|T196108]]
* 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore  (archived repo) {{!}} [[phab:T196513|T196513]]
* 18:58 thcipriani: @deployment-tin: sudo keyholder arm
* 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow {{!}} [[phab:T186969|T186969]]
* 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ {{!}} [[phab:T196509|T196509]]
* 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS  {{!}} [[phab:T196509|T196509]]
== 2018-06-05 ==
* 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to {{Gerrit|66727b7}}
* 12:07 kart_: Update cxserver to {{Gerrit|7fb7671}}
== 2018-06-04 ==
* 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to {{Gerrit|f579f0d}}
* 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force [[phab:T196401|T196401]]
* 15:44 awight: ORES: Fix [[phab:T194322|T194322]]
== 2018-06-02 ==
* 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
* 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
* 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
* 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky
== 2018-06-01 ==
* 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
* 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01)  - [[phab:T195314|T195314]]
* 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, [[phab:T195314|T195314]]
* 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
* 20:51 legoktm: deleting old versions of docker images
* 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
* 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
* 20:26 mutante: contint1001 - apt-get clean  got a little bit more disk space
* 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
* 08:45 hashar: Bumping Quibble jobs to 0.0.16
* 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716
== 2018-05-31 ==
* 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
* 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ {{!}} [[phab:T187622|T187622]]
* 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
* 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - [[phab:T195314|T195314]]
* 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs [[phab:T196062|T196062]]
* 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
* 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
* 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
* 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
* 07:16 legoktm: repooled integration-slave-docker-1010
* 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
* 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - [[phab:T195314|T195314]]
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
* 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, [[phab:T195314|T195314]]
* 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722
== 2018-05-30 ==
* 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
* 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
* 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
* 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
* 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large ([[phab:T192561|T192561]])
== 2018-05-29 ==
* 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
* 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
* 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - [[phab:T193734|T193734]]
* 12:42 zeljkof: Reloading Zuul to deploy {{Gerrit|bcd36d9a2cd2b2ebb1ec77bdf0fcdcc612aeae8d}}
* 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
* 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
* 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
* 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
* 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
* 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
* 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
* 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
* 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
* 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
* 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938
== 2018-05-28 ==
* 11:09 hashar: Building Quibble Docker images 0.0.13 {{!}} [[phab:T195634|T195634]]
* 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type {{!}} [[phab:T195634|T195634]]
* 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems
== 2018-05-27 ==
* 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
* 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673
== 2018-05-26 ==
* 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
* 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours
== 2018-05-25 ==
* 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
* 21:41 hashar: cleaned up disk space on deployment-tin  . Reenabled jenkins slave.
== 2018-05-24 ==
* 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
* 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
* 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
* 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
* 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
* 15:56 zeljkof: Reloading Zuul to deploy {{Gerrit|2e02d97a03e9aaf6a01e287f381934b8a7b03c9a}}
* 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
* 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
* 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846
== 2018-05-23 ==
* 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
* 20:07 bearND: Beta Cluster: Update mobileapps to {{Gerrit|29ebe0f}}
* 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
* 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
* 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363
== 2018-05-22 ==
* 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for [[phab:T194926|T194926]], finished without errors on any of the hosts)
* 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done [[phab:T194926|T194926]]
* 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
* 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref [[phab:T195312|T195312]]
== 2018-05-21 ==
* 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.
== 2018-05-20 ==
* 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
* 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-05-19 ==
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|9f3a185cb7d020757830e546e30e5d5e4ed67978}}
* 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
* 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
* 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
* 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845
== 2018-05-18 ==
* 18:31 zeljkof: Reloading Zuul to deploy {{Gerrit|f8e9625622aab96f0718f5fc9655117c8ebc3f55}}
* 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs [[phab:T194900|T194900]] [[phab:T191050|T191050]]
== 2018-05-17 ==
* 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image
== 2018-05-16 ==
* 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs [[phab:T177896|T177896]]
* 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
* 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308
== 2018-05-15 ==
* 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
* 23:22 legoktm: live hacking zuul config on contint1001
* 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020
== 2018-05-14 ==
* 19:41 bearND: BC: Update mobileapps to {{Gerrit|39c16e4}}
* 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
* 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
* 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0
== 2018-05-11 ==
* 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
* 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default
== 2018-05-10 ==
* 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
* 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
* 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
* 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
* 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
* 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
* 19:36 legoktm: kicked postmerge queue in jenkins
* 19:10 twentyafterfour: reset jenkins slave on deployment-tin
* 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' {{!}} [[phab:T194376|T194376]]
* 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'
== 2018-05-09 ==
* 22:53 awight: ORES: wheels fixups
* 21:31 awight: Bump ORES wheels
* 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
* 21:04 awight: ORES: drafttopic in beta
* 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space
== 2018-05-08 ==
* 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
* 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 ([[phab:T190717|T190717]])
== 2018-05-07 ==
* 20:19 bearND: Update mobileapps to {{Gerrit|c1f4de6}} in BC
* 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
* 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
* 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
* 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495
== 2018-05-06 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
* 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290
== 2018-05-05 ==
* afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
* 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
* 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
* 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055
== 2018-05-04 ==
* 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
* 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock
== 2018-05-03 ==
* 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 {{!}} [[phab:T193767|T193767]]
* 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159
== 2018-05-02 ==
* 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
* 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
* 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for [[phab:T193611|T193611]]
* 10:43 hashar: Building quibble docker iamges 0.0.11-2  [[phab:T118371|T118371]] [[phab:T193222|T193222]]
* 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
* 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
* 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291
== 2018-05-01 ==
* 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'
== 2018-04-30 ==
* 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble/  (thanks zeljkof )
* 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740
== 2018-04-27 ==
* 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
* 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins.  Triggered with a custom gear_client.py script from contint1001 . PID 13134
== 2018-04-26 ==
* 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
* 19:00 hashar: Building releng/quibble 0.0.11 docker images
* 17:41 mobrovac: stopped cpjobqueue and purging logs
* 16:51 thcipriani: mark integration-slave-jessie-1001 online
* 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline
== 2018-04-25 ==
* 21:27 hasharAway: building releng/quibble docker images @0.0.10
* 21:13 hasharAway: tagged quibble 0.0.10 {{Gerrit|a4e88ae}}
* 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
* 18:55 awight: ORES: Revscoring 2.2.2
* 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769
== 2018-04-24 ==
* 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859
== 2018-04-23 ==
* 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
* 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
* 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to {{Gerrit|b011b2a}}
* 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
* 07:27 hashar: Upgrading Blue Ocean on CI Jenkins  1.4.2 -> 1.5.0
== 2018-04-20 ==
* 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
* 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
* 08:38 hashar: Cleaned integration-slave-docker-1005  disk (deleting workspace and all docker images)
* 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
* 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
* 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
* 00:46 awight: roll back ORES beta to master
* 00:08 awight: Push ORES git-lfs to look at stuff
== 2018-04-19 ==
* 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
* 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
* 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 {{!}} [[phab:T192576|T192576]]
* 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
* 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
* 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
* 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
* 09:41 hashar: building releng/quibble-stretch:0.0.8-2
* 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done [[phab:T192476|T192476]]
== 2018-04-18 ==
* 22:53 eddiegp: started nutcracker on deployment-jobrunner03
* 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref [[phab:T178457|T178457]]
* 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
* 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
* 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
* 20:29 Reedy: reloaded zuul for 427469
* 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to {{Gerrit|fb161d7}}
* 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - [[phab:T192471|T192471]]
* 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
* 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
* 13:18 addshore: added WMDE-Fisch to deployment-prep
* 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
* 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/  [[phab:T192432|T192432]]
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
* 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
* 04:26 legoktm: try once more to refresh nodepool snapshot
* 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
== 2018-04-17 ==
* 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
* 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
* 20:28 hasharAway: building releng/quibble.*:0.0.8 images
* 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
* 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009  (m4executor label)
* 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only  use of that role in WMCS
* 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only  use of that role in WMCS
* 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)
== 2018-04-16 ==
* 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
* 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
* 22:25 legoktm: kick stuch beta-update jenkins jobs
* 20:35 mdholloway: restarted restbase and mobileapps services for testing ([[phab:T192287|T192287]])
* 18:58 awight: Update ORES editquality; [[phab:T185903|T185903]]
* 13:13 _joe_: also shutting down deployment-jobrunner02 [[phab:T192071|T192071]]
* 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
* 12:48 _joe_: turning off deployment-mediawiki{04,05} [[phab:T192071|T192071]], will be deleted by EOW
* 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837
== 2018-04-13 ==
* 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
* 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
* 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
* 08:48 eddiegp: debugging [[phab:T189493|T189493]] on beta
* 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state [[phab:T192071|T192071]]
* 07:22 legoktm: restarting jenkins
* 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
* 07:01 hashar: rebuilding quibble containers to use 0.0.7
* 00:53 awight: ORES: Test large file in LFS
== 2018-04-12 ==
* 23:01 awight: Try gerrit-based submodules for ORES, [[phab:T180627|T180627]]
* 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
* 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
* 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep ([[phab:T192071|T192071]])
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
* 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
* 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver [[phab:T192071|T192071]]
* 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/
== 2018-04-11 ==
* 21:04 awight: ORES experiment with git-lfs, [[phab:T180627|T180627]]
* 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446
== 2018-04-10 ==
* 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module ([[phab:T184854|T184854]])
* 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package {{!}} [[phab:T191771|T191771]]
== 2018-04-09 ==
* 19:50 awight: Redundant virtualenv for ORES
* 18:06 awight: Restore to ORES master branch
* 17:17 awight: Test git-lfs in ORES
* 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix [[phab:T191786|T191786]] (success: Notice: Applied catalog in 27.11 seconds)
== 2018-04-06 ==
* 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for [[phab:T173887|T173887]]
* 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 ([[phab:T190717|T190717]])
* 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep ([[phab:T191578|T191578]])
== 2018-04-05 ==
* 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
* 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt {{!}} mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
* 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak [[phab:T173887|T173887]]
* 17:08 bearND: Update mobileapps to {{Gerrit|dbc0687}}
* 17:05 hashar: docker: building releng/quibble:0.0.5
* 17:00 bearND: Update mobileapps to {{Gerrit|2d5ab5b}} on beta
* 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
* 07:52 moritzm: removed  unused/defunct  deployment-videoscaler01 from deployment-prep ([[phab:T191293|T191293]])
* 07:52 moritzm: removed  unused/defunct  deployment-tmh01 from deployment-prep ([[phab:T191293|T191293]])
* 07:47 hashar: deployment-prep: made Muehlenhoff an admin
== 2018-04-04 ==
* 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
* 21:24 awight: Roll back beta ORES
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to {{Gerrit|2d5ab5b}}
* 20:12 awight: Try dsh scap config for ORES
* 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
* 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
* 15:30 addshore: reload zuul for  https://gerrit.wikimedia.org/r/423652
* 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
* 11:58 hashar: Building releng/quibble-stretch:0.0.4
* 11:58 hashar: Building releng/quibble:0.0.4
* 07:11 hashar: deployment-prep: adding EddieGP as a member
== 2018-04-03 ==
* 09:49 hashar: building releng/quibble:0.0.3
* 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
== 2018-04-02 ==
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to {{Gerrit|58a0a88}}
== 2018-03-31 ==
* 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
== 2018-03-30 ==
* 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis ([[phab:T189694|T189694]])
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} [[phab:T191077|T191077]]
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} TT191077
== 2018-03-29 ==
* 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript  updateCollation.php --wiki=fawiki ([[phab:T190965|T190965]])
* 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
* 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
* 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1  https://gerrit.wikimedia.org/r/#/c/422959/
* 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
* 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
* 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
* 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host ([[phab:T191029|T191029]])
* 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug [[phab:T184234|T184234]]" due to git sync failures
== 2018-03-28 ==
* 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
* 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
* 20:45 bearND: Update mobileapps to {{Gerrit|a5833a0}} on BC
* 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
* 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
* 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)
== 2018-03-27 ==
* 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 {{!}} https://gerrit.wikimedia.org/r/#/c/422168/
* 08:30 kart_: Update cxserver to {{Gerrit|9e8ebda}}
* 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs
== 2018-03-26 ==
* 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to {{Gerrit|534f95d}}
* 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 {{!}} [[phab:T190393|T190393]]
* 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)
== 2018-03-23 ==
* 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin
== 2018-03-22 ==
* 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)
== 2018-03-21 ==
* 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
* 23:22 bd808: Raised security group quota from 20 to 40
* 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
* 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to {{Gerrit|e6b50a0}}
* 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
* 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
* 17:52 hashar: deployment-prep:  scap sync-file  docroot/wwwportal/portal '[[phab:T173887|T173887]]'
* 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins
== 2018-03-20 ==
* 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
* 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to {{Gerrit|634a15f}}
* 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
* 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
* 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
* 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
== 2018-03-19 ==
* 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
* 16:57 awight: ORES beta service is restored.
* 16:45 awight: Put ORES-beta back onto master branch
* 16:43 awight: ORES-beta has been down since January.
* 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
* 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
* 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
* 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1  {{!}} https://gerrit.wikimedia.org/r/#/c/420311/ {{!}} [[phab:T190032|T190032]]
* 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
* 07:43 kart_: Add cx_version to cx_translations table ([[phab:T190009|T190009]])
== 2018-03-18 ==
* 22:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/420230
* 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344
== 2018-03-16 ==
* 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
* 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
* 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
* 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/*  # some pack of 6GB belonged to root
* 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
* 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock [[phab:T189859|T189859]]
* 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance
== 2018-03-15 ==
* 17:15 bearND: Update mobileapps to {{Gerrit|c5e1522}}
* 16:43 Amir1: created deployment-ores01 to use stretch
* 16:36 Amir1: deleting deployment-ores01 to reimage
* 15:47 awight: ORES with git-lfs, scap config
* 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
* 00:21 awight: ORES with git-lfs
== 2018-03-14 ==
* 23:30 awight: new ORES submodule, pre-git-lfs
* 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to {{Gerrit|9f4a80c}}
* 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
* 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
* 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
* 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
* 12:16 kart_: update cxserver to {{Gerrit|c355d0c}}
* 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
* 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
* 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch  to Nodepool (they use too much disk space) {{!}} [[phab:T189365|T189365]]
== 2018-03-13 ==
* 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:08 awight: Enable Extension:JADE, [[phab:T176333|T176333]]
* 11:26 kart_: Update cxserver to {{Gerrit|bd2ccfc}}
* 10:39 kart_: Update cxserver to {{Gerrit|fd2c4be}}
* 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 {{!}} [[phab:T188112|T188112]]
* 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ {{!}} [[phab:T188112|T188112]]
* 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3  (HACK) {{!}} [[phab:T188112|T188112]]
* 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 {{!}} [[phab:T188112|T188112]]
== 2018-03-12 ==
* 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to {{Gerrit|5c90db7}}
* 20:09 bd808: Forced puppet run on deployment-logstash2 for {{gerrit{{!}}418986}}
* 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
* 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
* 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
* 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ {{!}} [[phab:T155840|T155840]]
* 12:24 hashar: Jenkins: uninstalled the "cvs" plugin
== 2018-03-10 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - [[phab:T189365|T189365]]
== 2018-03-09 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0  https://gerrit.wikimedia.org/r/#/c/417960/
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container {{!}} [[phab:T188686|T188686]]
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343
== 2018-03-08 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 .  Upgrade packages and reboot.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 18:36 bearND: Update mobileapps to {{Gerrit|afb0167}}
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc  # 1.4GBytes
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 10:27 hashar: Deploy docker images for /deploy repositories {{!}} https://gerrit.wikimedia.org/r/#/c/417217/
== 2018-03-07 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" {{!}} [[phab:T188686|T188686]]
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 02:28 legoktm: manually triggering jenkins jobs
== 2018-03-06 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets  parent permission group to scoring/ores
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic  scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from  scoring/ores itself inheriting from scoring/
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to {{Gerrit|afbe9af}}
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h
== 2018-03-05 ==
* 16:38 Reedy: deleted the stack traces too
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-03-04 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 02:12 Krenair: Regenerated captcha images for [[phab:T164047|T164047]]
== 2018-03-02 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs
== 2018-03-01 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 02:11 legoktm: manually queued jenkins jobs
== 2018-02-28 ==
* 16:31 legoktm: manually queued jenkins jobs
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 07:29 legoktm: mass queuing jenkins jobs again
== 2018-02-27 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 -  https://gerrit.wikimedia.org/r/#/c/415102/
* 10:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d9ed9d4dded7d646fc9c4b54155613eef99752a9}}
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.
== 2018-02-26 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx ([[phab:T188288|T188288]])
== 2018-02-24 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
== 2018-02-23 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 11:50 hashar: deployment-mediawiki04 :  apt-get clean && apt-get autoremove --purge
* 11:50 hashar: deployment-mediawiki04 :  rm /var/cache/hhvm/*.sq3  and restarting hhvm
* 11:48 hashar: deployment-mediawiki04 is out of disk space on /  causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only {{!}} [[phab:T187995|T187995]]
== 2018-02-22 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/413374/
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to {{Gerrit|a1339a9}}
== 2018-02-21 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0  and reloading Zuul {{!}} https://gerrit.wikimedia.org/r/413227
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 16:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d90d617e0794725280f9f2660effabea74533cbe}}
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis
== 2018-02-20 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
== 2018-02-19 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions
== 2018-02-18 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
== 2018-02-16 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely {{!}} https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-02-15 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/  (no jobs touched)
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only {{!}} [[phab:T187428|T187428]]
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255
== 2018-02-14 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to {{Gerrit|f23519f}}
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB
== 2018-02-13 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 10:51 moritzm: shutdown deployment-videoscaler01 ([[phab:T187063|T187063]])
* 10:49 moritzm: shutdown deployment-tmh01 ([[phab:T187063|T187063]])
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/  # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089
== 2018-02-12 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for [[phab:T186870|T186870]]
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. [[phab:T54919|T54919]]
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - [[phab:T186870|T186870]])
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - [[phab:T186870|T186870]])
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - [[phab:T186870|T186870]]
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/
== 2018-02-10 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - [[phab:T186870|T186870]])
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
== 2018-02-09 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 [[phab:T186856|T186856]]
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. {{!}} [[phab:T179378|T179378]] [[phab:T186856|T186856]]
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for {{Gerrit|I73a3142271d76f9640005752b1a336123b921b9f}}
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - [[phab:T186856|T186856]]
* 00:59 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
* 00:58 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
== 2018-02-08 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test {{!}} [[phab:T179552|T179552]]
== 2018-02-07 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230
== 2018-02-06 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix  [[phab:T186381|T186381]]
* 21:14 legoktm: restarted zuul due to patch being stuck ([[phab:T186381|T186381]])
* 19:25 hashar: Restarted Zuul due to [[phab:T186381|T186381]]
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480
== 2018-02-05 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 16:09 mdholloway: mobileapps deployment to BC failed with error ([[phab:T186532|T186532]])
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 {{!}} [[phab:T186494|T186494]]
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 {{!}} [[phab:T186494|T186494]]
== 2018-02-04 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
== 2018-02-03 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins
== 2018-02-01 ==
* 16:14 Amir1: deleting deployment-sca03 ([[phab:T184501|T184501]])
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-31 ==
* 22:01 mdholloway: updated mobileapps to {{Gerrit|3d717fa}} on beta cluster
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-30 ==
* 20:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/404778
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
== 2018-01-29 ==
* 23:24 awight: Experiment with versioned ORES venv, [[phab:T181071|T181071]]
== 2018-01-24 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
== 2018-01-23 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from  integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase {{!}} https://wikitech.wikimedia.org/wiki/Keyholder
== 2018-01-22 ==
* afk: restarting jenkins
== 2018-01-20 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
== 2018-01-19 ==
* 17:23 zeljkof: Reloading Zuul to deploy {{Gerrit|25cab6fdb53f1e5d284192159c43f0ce28f6e604}}
* 17:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7d6b4eecc6d7854de7db6714bcfd006d860f3e28}}
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
== 2018-01-18 ==
* 18:29 bearND: (beta): Update mobileapps to {{Gerrit|2690899}}
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state
== 2018-01-17 ==
* 16:24 zeljkof: Reloading Zuul to deploy {{Gerrit|5f757310f499a6a2cdf036dde3d258046377186f}}
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for [[phab:T185024|T185024]]  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
== 2018-01-16 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images{{!}}grep wmfreleng{{!}}awk '{print \$3}'{{!}}sort{{!}}uniq{{!}}xargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images {{!}} https://gerrit.wikimedia.org/r/#/c/404422/
== 2018-01-15 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy {{Gerrit|ff0a02decf1c75b2cef4f438c622d548c45fd270}}
* 17:14 zeljkof: Reloading Zuul to deploy {{Gerrit|fff6431628437cd9b3ccd5bafe5b58480b2a3714}}
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for [[phab:T180841|T180841]]
* 13:37 kart_: Ran update.php on metawiki betacluster ([[phab:T180841|T180841]])
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git {{!}} [[phab:T174338|T174338]]
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo
== 2018-01-12 ==
* 14:11 zeljkof: Reloading Zuul to deploy {{Gerrit|3816837e6d70ee7eb276c9b00034a5d2012a8afd}}
== 2018-01-11 ==
* 17:50 tgr: added Groovier1 to project members for [[phab:T158909|T158909]]
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403654
* 13:45 hashar: Migrate  composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403647
== 2018-01-10 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers {{!}} https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826
== 2018-01-09 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' {{!}} [[phab:T158243|T158243]] [[phab:T162191|T162191]]
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git {{!}} [[phab:T183569|T183569]]
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki ([[phab:T184276|T184276]])
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores ([[phab:T184282|T184282]])
* 08:48 Amir1: stopping ores services in deployment-sca03 ([[phab:T184282|T184282]])
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 ([[phab:T184282|T184282]])
== 2018-01-08 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
== 2018-01-05 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
== 2018-01-03 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3  the version in apt.wm.o
* 19:09 hashar: apt-get upgrade on deployment-tin  . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)
== 2018-01-01 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for [[phab:T183862|T183862]]
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki ([[phab:T183862|T183862]])
* 19:56 Amir1: restarting ores services in deployment-sca03 ([[phab:T183862|T183862]])
{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 00:57, 1 January 2019

Archives