You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(hashar: Upgraded Jenkins to 2.73.1 earlier today)
imported>Stashbot
(James_F: Zuul: [wikimedia-cz/web-*] Migrate tests from php73+ to php74+)
 
Line 1: Line 1:
== 2017-10-09 ==
== 2022-09-23 ==
* 13:26 hashar: Upgraded Jenkins to 2.73.1 earlier today
* 18:09 James_F: Zuul: [wikimedia-cz/web-*] Migrate tests from php73+ to php74+
* 08:53 hashar: hard restart integration-slave-docker-1001 via horizon. It is deadlocked somehow. - [[phab:T177749|T177749]]
* 18:06 James_F: Zuul: [labs/tools/guc] Migrate tests from php73+ to php74+
* 18:04 James_F: Zuul: [labs/tools/coverme] Migrate tests from php73+ to php74+
* 15:55 James_F: Docker: Building and publishing php74 versions of composer-security-check, mediawiki-phan, mediawiki-phan-testrun, and phpmetrics
* 13:26 James_F: Zuul: Run php 7.4 phan for extensions and skins


== 2017-10-06 ==
== 2022-09-22 ==
* 13:22 hashar: Jenkins: adding Maven-3.0.5 to the tool configuration https://integration.wikimedia.org/ci/configureTools/
* 20:40 zabe: shutoff deployment-db07 # [[phab:T318126|T318126]]
* 11:58 hashar: Jenkins: installed Warnings plugin
* 20:36 zabe: take deployment-prep out of read-only # [[phab:T318126|T318126]]
* 11:54 hashar: Jenkins: removing the Violations plugin. It is not used.
* 20:32 zabe: failover deployment-prep master from deployment-db07 to deployment-db09 # [[phab:T318126|T318126]]
* 09:22 hashar: integration: purged bunch of old containers: sudo cumin 'name:slave-docker' 'yes {{!}} docker container prune'
* 20:25 zabe: set deployment-prep as read-only # [[phab:T318126|T318126]]
* 16:26 dancy: Upgrading scap to latest code revision in beta cluster
* 10:38 zabe: deployment-db10: start replication # [[phab:T318126|T318126]]


== 2017-10-05 ==
== 2022-09-21 ==
* 19:15 hasharAway: rebooting integration-slave-docker-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 23:34 zabe: shutoff deployment-db08 # [[phab:T318126|T318126]]
* 19:11 hasharAway: rebooting integration-slave-jessie-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 23:00 jeena: restarting zuul to try and fix CI issues
* 13:16 hashar: Image snapshot-ci-jessie-1507208677 in wmflabs-eqiad is ready
* 20:46 zabe: clone deployment-db10 from deployment-db08 # [[phab:T318126|T318126]]
* 11:47 hashar: Refreshing Nodepool Jessie snapshot to get java 8 by default - [[phab:T162828|T162828]]
* 18:49 TheresNoTime: cherry-picked [[gerrit:833839]] to deployment-puppetmaster04, testing [[phab:T317417|T317417]]
* 10:56 hashar: integration: unbreak the puppet master. Was stuck do a cherry pick that needed a rebase
* 18:19 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db10 # [[phab:T318126|T318126]]
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/382361
* 17:55 zabe: create volume db10 and attach to deployment-db10 # [[phab:T318126|T318126]]
* 04:16 legoktm: deploying https://gerrit.wikimedia.org/r/382354
* 17:54 zabe: create deployment-db10 as g3.cores8.ram16.disk20 # [[phab:T318126|T318126]]
* 14:21 zabe: deployment-db09: restart mariadb # [[phab:T318126|T318126]]
* 13:55 TheresNoTime: modified deployment-prep "prometheus" security group - port 80, [[phab:T315699|T315699]]
* 13:18 James_F: Jenkins: Dropped 16 more old node jobs left on the server.
* 13:11 James_F: Jenkins: Dropped four old node10 jobs left on the server (oojs-core-node10-browser-docker, ooui-special-node10-plus-php80-composer-docker, wikipeg-special-node10-plus-php72-composer-docker, wikipeg-special-node10-plus-php80-composer-docker)
* 13:05 James_F: Jenkins: Dropped scap-pipeline-stretch and trigger-scap-pipeline-stretch following {{Gerrit|26c74a1}}
* 12:36 hashar: Reloaded Zuul for Remove Stretch from mediawiki/tools/scap - https://gerrit.wikimedia.org/r/833705
* 09:46 andrewbogott: removed some stray whitespace in /var/lib/git/operations/puppet that was preventing rebase on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs


== 2017-10-04 ==
== 2022-09-20 ==
* 13:19 andrewbogott: migrating 'deployment-kafka-jumbo-1' to labvirt1017
* 22:00 zabe: deployment-db09: start replication # [[phab:T318126|T318126]]
* 20:06 zabe: deployment-db09: import dump into mariadb # [[phab:T318126|T318126]]
* 20:04 zabe: rsynced dump from deployment-db08 to deployment-db09 # [[phab:T318126|T318126]]
* 08:08 hashar: Upgrading CI and releases Jenkins plugins notably to update the git client [[phab:T315897|T315897]]
* 02:06 zabe: created backup of all databases on deployment-db08 # [[phab:T318126|T318126]]


== 2017-10-03 ==
== 2022-09-19 ==
* 22:38 thcipriani: git stash /srv/mediawiki-staging/php-master/extensions/Echo to fix beta-code-update-eqiad
* 23:58 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db09 # [[phab:T318126|T318126]]
* 14:26 hashar: Created https://github.com/wikimedia/analytics-wikistats2 - [[phab:T177288|T177288]]
* 23:57 zabe: create volume db09 and attach to deployment-db09 # [[phab:T318126|T318126]]
* 14:23 hashar: Gerrit: created analytics/wikistats2.git for fdans - [[phab:T177288|T177288]]
* 23:57 zabe: create deployment-db09 as g3.cores8.ram16.disk20 # [[phab:T318126|T318126]]
* 07:15 legoktm: deploying https://gerrit.wikimedia.org/r/381937
* 20:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/833066
* 05:33 legoktm: deploying https://gerrit.wikimedia.org/r/381812 https://gerrit.wikimedia.org/r/381378
* 16:54 James_F: Zuul: [operations/mediawiki-config] Switch to PHP 7.4 jobs
* 16:24 James_F: Zuul: [mediawiki/core] Add php80 and php81 to `check php` command
* 15:36 James_F: Zuul: [mediawiki/core] run phan on PHP 7.4 for [[phab:T316518|T316518]]
* 13:50 James_F: Zuul: [mediawiki/core] Add a non-vendor php81 job for main branch for [[phab:T316078|T316078]]
* 12:06 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T316128|T316128]]
* 11:57 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T316128|T316128]]


== 2017-10-02 ==
== 2022-09-16 ==
* 15:52 zeljkof: Reloading Zuul to deploy {{Gerrit|895f33da2a8daeb1fa2634bcd4eddfbbbcd78819}}
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster
* 14:57 addshore: docker push npm image from https://gerrit.wikimedia.org/r/#/c/381384/2
* 14:47 zeljkof: Reloading Zuul to deploy {{Gerrit|9ac7821ae384612ad8249d03e47f264004d1199a}}
* 14:22 zeljkof: Reloading Zuul to deploy {{Gerrit|00890a511274d4b215034979db91708fbf6b0d0f}}
* 12:32 addshore: docker push composer, mediawiki-phan & mediawiki-phpcs latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:23 addshore: docker push ci-jessie, php & php-mediawiki latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:16 addshore: marking integration-slave-docker-1002 as offline [[phab:T177039|T177039]]
* 11:08 zeljkof: Reloading Zuul to deploy {{Gerrit|c1d7b5f1f7db4968cc2acc85539c298865a3b5ff}}


== 2017-09-30 ==
== 2022-09-15 ==
* 14:28 zeljkof: Reloading Zuul to deploy {{Gerrit|c08a3ad67b5bc1cbdb900d6b4138034b4be83c0f}}
* 19:56 thcipriani: Updating development images on contint primary
* 17:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/832374
* 11:03 TheresNoTime: soft reboot deployment-parsoid12, unresponsive


== 2017-09-29 ==
== 2022-09-13 ==
* 19:45 hashar: Deleting integration-slave-jessie-php55
* 22:14 zabe: delete deployment-urldownloader02
* 17:34 zeljkof: Reloading Zuul to deploy {{Gerrit|0e26c8697438a88fbdd62884dcb40e664ba98fa2}}
* 16:42 zeljkof: Reloading Zuul to deploy {{Gerrit|09445b837a03a2ce906fe848aec8350f59ab5898}}
* 15:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7f66813dc0842dadfdb74c9257582aed26f35d60}}
* 14:15 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.logininput.org ( testing w/o delete [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 14:10 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.loginpartner.org --delete ( testing [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 13:00 hashar: github: created https://github.com/wikimedia/integration-quibble for gerrit replication
* 12:53 hashar: gerrit: marked labs/tools/grrrit archived
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'weeks ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'months ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:40 addshore: marking integration-slave-docker-1001 as online - [[phab:T177039|T177039]]
* 09:33 addshore: rebooting integration-slave-docker-1001
* 09:10 addshore: wm-ci-docker-push mediawiki-phpcs:v2017.09.29.09.08 & latest https://gerrit.wikimedia.org/r/381413
* 05:59 legoktm: marking integration-slave-docker-1001 as offline - [[phab:T177039|T177039]]
* 00:19 mutante: releases1001 - created user for "no_justification", dropped pass in home dir
* 00:12 mutante: jenkins now configured and running at https://releases.wikimedia.org/ci/ ([[phab:T164030|T164030]]) - but needs additional admin users and puppet is still disabled for temp hack fix


== 2017-09-28 ==
== 2022-09-12 ==
* 16:58 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381242
* 22:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/19
* 15:37 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.28.15.28
* 20:38 RhinosF1: (for lack of a better place) added Cyberpower678 to acl*userdisable. has enough clue, fairly active and trusted.
* 11:35 hashar: docker push wmfreleng/tox:v2017.09.28.11.34 Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 08:14 James_F: Zuul: [mediawiki/extensions/Realnames] Enable quibble composer jobs
* 11:34 hashar: docker push wmfreleng/ci-jessie:v2017.09.28.11.33 . Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 09:20 moritzm: upgraded mediawiki04-mediawiki06 in deployment-prep to HHVM 3.18.5
* 08:39 hashar: Deleted integration-saltmaster and deployment-salt02 . Replaced by integration-cumin and deployment-cumin - [[phab:T176314|T176314]]
* 08:32 hashar: Migrated Hiera config from https://wikitech.wikimedia.org/wiki/Hiera:Integration to Horizon
* 08:31 hashar: Removing salt configuration from integration and deployment-prep projects. Replaced by cumin. - [[phab:T176314|T176314]]


== 2017-09-27 ==
== 2022-09-09 ==
* 21:25 hashar: salt is being replaced by cumin instances being deployment-cumin and integration-cumin . Check this out: https://wikitech.wikimedia.org/wiki/Cumin !
* 17:46 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T311126|T311126]]
* 20:12 hashar: Deleted aptly.integration.eqiad.wmflabs and the https://integration-aptly.wmflabs.org/repo/ webproxy. They were for php5.5 packages on jessie, now available on apt.wm.o - [[phab:T174972|T174972]]
* 17:25 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T311126|T311126]]
* 19:39 hashar: deployment-prep: purging "ferm" on hosts that no more have it applied via puppet. There were some old iptables rules left around blocking access
* 17:08 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T316409|T316409]]
* 16:38 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381024
* 16:36 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T316409|T316409]]
* 12:38 addshore: Reloading Zuul to deploy - Add lintr-docker-non-voting [integration/config] - https://gerrit.wikimedia.org/r/380746
* 10:49 hashar: devtools: fixed fqdn of instances puppetmaster-1001 and gerrit-prod-1001 by manually editing `/etc/hosts` # [[phab:T317404|T317404]]
* 12:27 hasharAway: docker push wmfreleng/tox:v2017.09.27.12.26  https://gerrit.wikimedia.org/r/#/c/380926/
* 11:01 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.27.10.53 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.27.10.51 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/composer:v2017.09.27.10.49 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.27.10.45 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/tox:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 10:08 addshore: docker push docker.io/wmfreleng/ci-jessie:v2017.09.27.09.59 & latest (from https://gerrit.wikimedia.org/r/#/c/378033/5)


== 2017-09-26 ==
== 2022-09-08 ==
* 16:15 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.26.16.09 & latest (from PS11 of https://gerrit.wikimedia.org/r/379479)
* 20:50 dduvall: running `./fab deploy_docker` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/830919
* 16:01 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.26.15.45 & latest (From PS10 of https://gerrit.wikimedia.org/r/379479)
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster
* 13:58 addshore: added hashar to https://hub.docker.com/u/wmfreleng
* 13:07 moritzm: adding  deployment-videoscaler01 to deployment-prep (stretch-based video scaler)
* 12:45 addshore: Reloading Zuul to deploy - Low prio queue for libraryupdater [integration/config] - https://gerrit.wikimedia.org/r/380307
* 12:28 addshore: fab docker_pull_image:wmfreleng/lintr:v2017.09.26.12.04
* 12:26 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/379818
* 12:06 addshore: docker push docker.io/wmfreleng/ & latest (PS6 of https://gerrit.wikimedia.org/r/378831 )


== 2017-09-25 ==
== 2022-09-07 ==
* 17:07 mutante: Greg is now a contint-admin
* 15:05 TheresNoTime: making hack changes to beta to test [[phab:T317195|T317195]] resolution
* 12:36 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker rmi wmfreleng/operations-puppet:0.0.1 wmfreleng/operations-puppet:0.1.0'
* 12:30 addshore: Reloading Zuul to deploy Refactor 'operations-puppet-tests-docker' into macros for easy reuse [integration/config] - https://gerrit.wikimedia.org/r/379959
* 09:12 moritzm: added deployment-mediawiki07 to deployment-prep (stretch-based app server, WIP)


== 2017-09-24 ==
== 2022-09-06 ==
* 10:49 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run "sudo docker images --no-trunc --format '{{.ID}} {{.CreatedSince}}' {{!}} grep ' months' {{!}} awk '{ print $1 }' {{!}} xargs --no-run-if-empty docke r rmi"
* 15:42 bd808: Promoted user 'StrikerBot' to admin on gitlab.wikimedia.org so that Striker can use the account to attach Developer accounts to gitlab via API.
* 10:37 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.24.10.33 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2 (actually this time))
* 02:05 James_F: Running REL1_39 branch commands for [[phab:T313920|T313920]]
* 09:58 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.22.18.51 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2)
* 00:20 Krinkle: Prune various old mediawiki/core wmf branches for Gerrit usability, ref [[phab:T303828|T303828]]


== 2017-09-22 ==
== 2022-09-02 ==
* 21:55 tabbycat: Granted Greg G. 'staff' global rights on the beta cluster per request
* 15:59 zabe: added vwalters as member of the deployment-prep project [[phab:T316943|T316943]]
* 20:37 hashar: Image snapshot-ci-jessie-1506112074 in wmflabs-eqiad is ready
* 13:40 Krinkle: " ENOENT: no such file or directory, lstat " failing quibble jobs on integration-agent-docker-1024
* 20:28 hashar: updating nodepool image for jessie [2/x]
* 20:03 hasharAway: Updating nodepool image for jessie
* 17:22 addshore: docker push docker.io/wmfreleng/tox:v2017.09.22.17.16 & latest # (From current master)
* 15:24 hashar: Restarted Jenkins (out of memory)
* 10:06 hashar: deployement-salt02 migrated hiera config from wikitech to horizon. Removed the class role::deployment::salt_masters
* 08:44 hashar: Upgraded docker on integration-slave-docker-1001 and integration-slave-docker-1002 - [[phab:T176267|T176267]]
* 07:13 greg-g: some jsduck jobs are running now, serially, for the backlogged queue. Unsure of starved jobs (integration-config-qa, pywikibot-beta-cluster, etc)
* 07:04 greg-g: deleting stuck mediawiki-core-jsduck-publish jobs in Jenkins UI
* 06:57 greg-g: pinged an opsen, hopefully they'll restart zuul shortly
* 06:45 greg-g: Zuul is stuck, no jobs are processing


== 2017-09-21 ==
== 2022-09-01 ==
* 10:23 elukey: removed {{Gerrit|6fdf6ee653}} from deployment-prep's puppet master cherry picks (seemed an old version of https://gerrit.wikimedia.org/r/#/c/357985)
* 09:36 zabe: shutoff deployment-urldownloader02
* 07:43 hashar: Updating Jenkins jobs for Quibble 1.4.5 > 1.4.6  +  php 7.4 update {{!}} [[phab:T305525|T305525]] {{!}} [[phab:T314586|T314586]] {{!}} [[phab:T316601|T316601]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/828611


== 2017-09-20 ==
== 2022-08-31 ==
* 15:46 addshore: reloading zuul for https://gerrit.wikimedia.org/r/#/c/379250/
* 23:41 zabe: deleted shutoff deployment-restbase03
* 13:59 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.20.13.49 & latest # built from master
* 16:39 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension # [[phab:T308738|T308738]]
* 13:59 addshore: docker push docker.io/wmfreleng/composer:v2017.09.20.13.44 & latest # built from master
* 16:37 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T312870|T312870]]
* 13:59 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.20.13.44 & latest # built from master
* 16:21 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T312870|T312870]]
* 13:59 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.20.13.43 & latest # built from master
* 16:18 hashar: Tag Quibble 1.4.6 @ {{Gerrit|8828487d0}} # [[phab:T305525|T305525]] [[phab:T314586|T314586]]
* 13:59 addshore: docker push docker.io/wmfreleng/php:v2017.09.20.13.40 & latest # built from master
* 15:27 James_F: Docker: Building and publishing quibble-fresnel image based on php74 for [[phab:T316525|T316525]]
* 13:07 tabbycat: deployment-prep Ran cleanupSpam.php on deploymentwiki. Further testing with regards to ongoing development and updating of the script.
* 11:53 addshore: Reloading Zuul (Testing)


== 2017-09-19 ==
== 2022-08-30 ==
* 17:26 legoktm: removed rights from User:Sau226 on beta cluster due to block of account used for browser tests
* 16:00 James_F: Zuul: [labs/tools/heritage] Switch postmerge job to tox-py37-coverage-publish for [[phab:T316627|T316627]]
* 09:13 tabbycat: Re-run previous script and it worked this time, see https://deployment.wikimedia.beta.wmflabs.org/wiki/Template_talk:Rotate/en
* 09:32 hashar: doc: on doc1002: `sudo -u doc-uploader rm -fR /srv/doc/mw-tools-scap/` That got moved to `/srv/doc` and a redirect has been set. # [[phab:T315541|T315541]]
* 09:11 tabbycat: Ran mwscript cleanupSpam.php on the beta cluster, but it didn't worked (looks it is not fetching the domains properly)


== 2017-09-18 ==
== 2022-08-29 ==
* 20:12 addshore: deleted unused images that were *months old* on docker slaves
* 21:25 inflatador: ES6->7 upgrade in beta-cluster [[phab:T315604|T315604]]
* 19:01 addshore: addshore@contint1001:~$ sudo service zuul reload
* 13:39 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/826985
* 18:45 thcipriani: reloading zuul to test https://gerrit.wikimedia.org/r/#/c/378665/2
* 12:34 James_F: Zuul: [mediawiki/core] Don't run phan on PHP 7.4, it doesn't pass; for [[phab:T316518|T316518]]
* 16:53 elukey: removed https://gerrit.wikimedia.org/r/#/c/377753/ from the git cherry-picks in operations/puppet on puppetmaster02
* 12:08 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-re-try] for [[phab:T293924|T293924]]
* 11:28 James_F: Zuul: [mediawiki/core] Make PHP 8.1 voting on REL1_38 and REL1_39 for [[phab:T316080|T316080]]
* 09:00 hashar: Updated Jenkins job mediawiki-quibble-composer-mysql-php80-docker to capture core dumps using https://gerrit.wikimedia.org/r/c/integration/config/+/496392  # [[phab:T315167|T315167]]
* 00:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/827015


== 2017-09-17 ==
== 2022-08-25 ==
* 18:59 addshore: Reloading Zuul to deploy archiving of 2 extensions
* 15:53 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/826593
* 07:37 James_F: Zuul: [mediawiki/extensions/UnlinkedWikibase] Add quibble for [[phab:T316183|T316183]]


== 2017-09-14 ==
== 2022-08-23 ==
* 19:37 tgr: updated PrivateSettings.php for [[phab:T175868|T175868]]
* 17:40 hashar: Stopping Gerrit
* 10:38 elukey: cherry-pick https://gerrit.wikimedia.org/r/#/c/377753/7 on deployment-prep's puppetmaster02 to test it on the new kafka jumbo instances
* 11:54 hashar: Manually applied a `docker-pkg` fix on contint2001 to prevent it from downloading unrelated images [[phab:T310458|T310458]]
* 10:35 hashar: CI puppet master:  added class geoip::data::package and parameters:  puppetmaster::geoip::fetch_private: false  puppetmaster::geoip::use_proxy: false    - [[phab:T175864|T175864]]


== 2017-09-13 ==
== 2022-08-22 ==
* 10:13 addshore: docker push docker.io/wmfreleng/operations-puppet:v2017.09.13.09.23 (#d693f74c9b3404220a2ad2934f526d4f4455914b)
* 07:17 taavi: trying to disconnect jenkins from gearman and then re-connect to see if it helps with [[phab:T315818|T315818]]
* 09:25 hashar: Deleting integration-slave-trusty-1003 and integration-slave-trusty-1001  - [[phab:T175696|T175696]]
* 07:12 taavi: restart zuul-merger on contint2001 [[phab:T315818|T315818]]
* 09:14 hashar: nodepool: openstack image delete image-ci-trusty  - [[phab:T175696|T175696]]
* 07:49 hashar: Jenkins: removing the Ubuntu JDK from https://integration.wikimedia.org/ci/configureTools/
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - [[phab:T161882|T161882]]
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - T 161882


== 2017-09-12 ==
== 2022-08-21 ==
* 20:35 hashar: pooling integration-slave-jessie-1003 and integration-slave-jessie-1004
* 13:07 Reedy: looks live various CI jobs (coverage etc) have been stuck for about 8.5 hours
* 19:40 hashar: hacked integration-slave-jessie hosts to ship them php5.5
* 13:00 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/824862
* 18:49 hasharAway: nodepool: deleted image image-ci-trusty_old_20170804  Keeping image-ci-trusty just in case
* 14:57 hashar: Deleted all left over jenkins jobs having ci-trusty-wikimedia label. - [[phab:T161882|T161882]]
* 14:46 hashar: provisionning integration-slave-jessie-1003 and integration-slave-jessie-1004 to move php55lint to them. NOT READY YET - [[phab:T161882|T161882]]
* 14:05 hashar: Deleting integration-slave-trusty-1004  - [[phab:T161882|T161882]]
* 13:09 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-815477
* 11:09 hashar: Image snapshot-ci-jessie-1505213295 in wmflabs-eqiad is ready
* 10:48 hashar: nodepool: force updating jessie image to grab php5.5-luasandbox - [[phab:T161882|T161882]] [[phab:T174972|T174972]]


== 2017-09-11 ==
== 2022-08-19 ==
* 23:27 thcipriani: restarting jenkins
* 23:04 TheresNoTime: resized deployment-mwlog01's /srv volume, restarted
* 22:38 legoktm: deploying https://gerrit.wikimedia.org/r/377361
* 22:57 TheresNoTime: shutting down deployment-mwlog01 for [[phab:T315707|T315707]]
* 12:47 hashar: Nodepool: refreshing jessie snapshot to get php5.5-luasandbox installed
* 15:50 James_F: Docker: Build stalled out for 30 minutes; terminated and re-started.
* 15:15 dancy: Upgrading scap to latest code revision in beta cluster
* 15:11 James_F: Docker: Building and publishing images with PHP 8.0.22 for [[phab:T315167|T315167]]


== 2017-09-10 ==
== 2022-08-18 ==
* 01:44 bd808: nodepool running steadily again, but has been heavily throttled to hopefully prevent another weekend thundering herd of doom failure for the OpenStack backend
* 17:01 hashar: Restarted zuul-merger on contint1001 # [[phab:T315586|T315586]]
* 16:42 hashar: Reloaded Zuul for {{Gerrit|Ie83b19699a8526bf67f5610a0aa89dcedc0e3979}}
* 13:14 awight: [beta] Deploying new kartotherian version


== 2017-09-09 ==
== 2022-08-17 ==
* 22:15 bd808: `sudo journalctl -u nodepool --since today --no-pager` shows many LaunchStatusException failures.
* 14:18 zabe: fix merge conflicts in deployment-prep private repo # [[phab:T315394|T315394]]
* 10:27 hashar: Built image docker-registry.discovery.wmnet/releng/commit-message-validator:1.0.0  # [[phab:T315159|T315159]]


== 2017-09-07 ==
== 2022-08-16 ==
* 13:02 hashar: nodepool: Image snapshot-ci-jessie-1504788047 in wmflabs-eqiad is ready {{!}} [[phab:T174972|T174972]]
* 20:51 RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350
* 11:58 hashar: nodepool: updating snapshot-ci-jessie to add php5.5-redis {{!}} [[phab:T161882|T161882]] [[phab:T174972|T174972]]
* 20:30 hashar: Repooled integration-agent-docker-1028 , it was mysteriously unreachable [[phab:T315372|T315372]]
* 11:10 addshore: Reloading Zuul to deploy "Add gate-submit jobs for analytics/wmde/* repos"
* 19:18 Krinkle: mediawiki/extensions/EventLogging$ git remote-wildcard-br-d 'wmf/1.35*' 'wmf/1.36*' 'wmf/1.37*' 'wmf/1.38*'
* 02:44 legoktm: deploying https://gerrit.wikimedia.org/r/376460
* 19:17 Krinkle: mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref [[phab:T303828|T303828]]
* 19:16 TheresNoTime: manually running `/usr/local/bin/wmf-beta-update-databases.py` on `deployment-deploy03`
* 17:16 TheresNoTime: soft-rebooting deployment-mediawiki12


== 2017-09-06 ==
== 2022-08-12 ==
* 21:32 bearND: Update mobileapps to {{Gerrit|2cb6281}} ([[phab:T168848|T168848]] [[phab:T169277|T169277]] [[phab:T169274|T169274]] [[phab:T162179|T162179]] [[phab:T164033|T164033]] [[phab:T167921|T167921]] [[phab:T174698|T174698]] [[phab:T168848|T168848]] [[phab:T174808|T174808]])
* 17:47 dancy: Restarting zuul
* 17:42 dancy: Restarting Jenkins in an attempt to get CI jobs running again
* 00:54 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service . Certificate was close to expiry


== 2017-09-05 ==
== 2022-08-11 ==
* 23:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/376034
* 21:11 mutante: restarted phd service on phab2001
* 19:34 gilles: deployed PrivateSettings.php change to add Thumbor username to Swift configuration
* 19:12 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16
* 12:26 jnuche: Reenabled CI beta sync jobs after cluster incident
* 11:48 jnuche: Temporarily disabled CI beta sync jobs until issue in cluster is resolved
* 10:25 zabe: take deployment-prep out of read-only mode


== 2017-09-04 ==
== 2022-08-10 ==
* 15:59 zeljkof: Reloading Zuul to deploy {{Gerrit|ca1c6ec05122e6f50a5a52fecbbd2bd8e882ad87}}
* 11:36 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/822052
* 12:21 hashar: Image snapshot-ci-jessie-1504527142 in wmflabs-eqiad is ready
* 11:37 hashar: nodepool: refreshing jessie snapshot
* 10:03 addshore: Reloading Zuul to deploy mwext-php70-phan-jessie-docker experimental job
* 00:42 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/libs$ sudo -u zuul rm -rf XMPReader


== 2017-09-02 ==
== 2022-08-09 ==
* 08:32 legoktm: rm -rf /var/logs/kafka on deployment-kafka01 to free up disk space
* 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
* 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
* 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
* 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for [[phab:T293924|T293924]]
* 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for [[phab:T293851|T293851]]


== 2017-08-31 ==
== 2022-08-08 ==
* 23:32 Krenair: fixed deployment-imagescaler0[12] puppet by installing a package and file manually, some puppetisation still needed - https://phabricator.wikimedia.org/T174746
* 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
* 23:04 Krenair: that also did deployment-cache-(upload{{!}}text)04
* 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for [[phab:T309872|T309872]]
* 22:50 Krenair: fixed deployment-ms-be0[34] puppet by removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/371582/1 - details in a comment there
* 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for [[phab:T314630|T314630]]
* 15:49 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/374970/
* 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166
* 14:52 zeljkof: Reloading Zuul to deploy {{Gerrit|dd504839b3a48f431199785421751a5e3e38af57}}
* 10:06 zeljkof: Reloading Zuul to deploy {{Gerrit|e91b30ff6df435a6a3c55fe978be6bbf1f1cbde4}}
* 06:25 legoktm: deploying https://gerrit.wikimedia.org/r/374937


== 2017-08-30 ==
== 2022-08-05 ==
* 12:49 hashar: gerrit: marked wikimedia/communications/WMBlog as read-only - [[phab:T172372|T172372]]
* 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for [[phab:T296967|T296967]]
* 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
* 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for [[phab:T314630|T314630]]
* 14:48 James_F: Zuul: Add WelpThatWorked to allow list
* 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin


== 2017-08-29 ==
== 2022-08-04 ==
* 15:39 hashar: Created integration-slave-jessie-php55 to try out a php5.5 package on Jessie - [[phab:T161882|T161882]]
* 15:21 dancy: Deleting beta-mediawiki-config-update-eqiad job
* 15:06 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-793795
* 15:16 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405
* 08:45 hashar: Restarting Jenkins for openjdk update
* 10:01 TheresNoTime: clearing out stuck beta deployment jobs [[phab:T314378|T314378]] [[phab:T72597|T72597]]
* 08:11 hashar: refreshing all Jenkins jobs with a newer version of JJB


== 2017-08-28 ==
== 2022-08-03 ==
* 14:54 hashar: integration: rebase integration puppet master. Got a conflict due to r -> r_lang renaming ( https://gerrit.wikimedia.org/r/#/c/363337/ )
* 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
* 08:52 hashar: gerrit: added ldap/ciadmin  to the 'integration' group.  [[phab:T169557|T169557]] [[phab:T173233|T173233]]
* 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
* 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
* 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
* 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for [[phab:T309872|T309872]]
* 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
* 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too


== 2017-08-25 ==
== 2022-08-02 ==
* 15:11 zeljkof: Reloading Zuul to deploy {{Gerrit|b6704e249f0d9e83de5835908eaddff40f104e2f}}
* 19:26 James_F: Zuul: [design/codex] Switch coverage job back to -direct
* 15:23 dancy: Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028)
* 15:22 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028
* 07:55 TheresNoTime: cleared stuck beta deployment jobs [[phab:T72597|T72597]]


== 2017-08-24 ==
== 2022-08-01 ==
* 21:55 mdholloway: disk was full on integration-slave-jessie-android; deleted ~8gb of old screenshots from /tmp to clear some space
* 23:16 James_F: Zuul: [design/codex] Switch to node16
* 16:54 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/372780/7
* 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
* 15:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/371138/
* 22:53 TheresNoTime: remove stuck beta deployment jobs
* 11:56 zeljkof: Reloading Zuul to deploy {{Gerrit|c20a7402467efb669a30dc06ec70c41fc6853193}}
* 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
* 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
* 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
* 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
* 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
* 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for [[phab:T314306|T314306]]
* 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
* 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
* 09:14 TheresNoTime: clearing stuck beta CI jobs


== 2017-08-22 ==
== 2022-07-29 ==
* 06:49 legoktm: deploying https://gerrit.wikimedia.org/r/370695
* 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for [[phab:T313919|T313919]]
* 06:44 legoktm: deploying https://gerrit.wikimedia.org/r/372229
* 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes


== 2017-08-21 ==
== 2022-07-28 ==
* 18:18 mutante: addshore is now a contint-admin
* 17:54 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/818189/


== 2017-08-18 ==
== 2022-07-27 ==
* 20:21 legoktm: deploying https://gerrit.wikimedia.org/r/372222
* 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch [[phab:T300463|T300463]]
* 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for [[phab:T274965|T274965]]
* 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
* 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
* 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref [[phab:T143162|T143162]], [[phab:T193565|T193565]]


== 2017-08-16 ==
== 2022-07-25 ==
* 23:46 legoktm: reloading Zuul to deploy https://gerrit.wikimedia.org/r/372208 https://gerrit.wikimedia.org/r/371653 https://gerrit.wikimedia.org/r/371757 https://gerrit.wikimedia.org/r/371640
* 22:16 dduvall: re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 21:25 dduvall: disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769


== 2017-08-15 ==
== 2022-07-23 ==
* 17:57 bearND: Update mobileapps to {{Gerrit|33b80dd}} ([[phab:T172829|T172829]] [[phab:T152441|T152441]] [[phab:T172021|T172021]] [[phab:T103362|T103362]])
* 17:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/816251


== 2017-08-14 ==
== 2022-07-21 ==
* 09:46 TabbyCat: maurelio@deployment-tin:/srv/mediawiki/dblists$ expanddblist flow-computed > /home/maurelio/flow-test.dblist (to test expandblist for a patch I am working on)
* 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster


== 2017-08-11 ==
== 2022-07-20 ==
* 20:25 addshore: added mediawiki::maintenance::wikidata to deployment-tin
* 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
* 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline


== 2017-08-07 ==
== 2022-07-19 ==
* 15:11 thcipriani: restarting jenkins for plugin update
* 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
* 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # [[phab:T310905|T310905]]


== 2017-08-06 ==
== 2022-07-18 ==
* 13:28 TabbyCat: Ran mwscript extensions/WikimediaMaintenance/dumpInterwiki.php deploymentwiki on the beta cluster
* 19:43 dancy: Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster
* 13:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # [[phab:T311752|T311752]]
* 10:40 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745
* 09:58 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included
* 09:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/814728


== 2017-08-04 ==
== 2022-07-17 ==
* 15:21 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/370222/2
* 13:00 taavi: reloading zuul for https://gerrit.wikimedia.org/r/814356


== 2017-08-03 ==
== 2022-07-16 ==
* 12:02 hashar: Added  integration-slave-docker-1004 to the pool of jenkins slaves - [[phab:T150502|T150502]]
* 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt
* 10:12 hashar: gerrit: marked wikimedia/communications/WP-Victor read-only and [ARCHIVED] - [[phab:T107430|T107430]]
* 04:50 SMalyshev: update cherry-pick for https://gerrit.wikimedia.org/r/#/c/299825/8 on deployment-puppetmaster02.deployment-prep.eqiad.wmflabs


== 2017-08-02 ==
== 2022-07-15 ==
* 22:08 MaxSem: Running rebuildall.php on beta ruwiki
* 15:59 hashar: Built pcov php docker images [[phab:T280170|T280170]]
* 20:17 bearND: Update mobileapps to {{Gerrit|2d8e8f6}}
* 15:46 hashar: contint2001: `docker-system-prune-dangling.service`  it failed overnight cause Docker was not running. That should clear Icinga state # [[phab:T313119|T313119]]
* 11:31 hashar: Image snapshot-ci-jessie-1501673225 in wmflabs-eqiad is ready [[phab:T169602|T169602]]
* 14:05 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Switch to node16 jobs
* 10:51 hashar: Image snapshot-ci-jessie-1501670727 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 13:05 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]], this time actually.
* 09:02 hashar: Regenerating Nodepool Jessie image from scratch to get rid of tox 1.9.2 installed under /usr/local - [[phab:T169602|T169602]]
* 12:30 hashar: Starting docker on contint2001.wikimedia.org # [[phab:T313119|T313119]]
* 08:44 hashar: Image snapshot-ci-jessie-1501662758 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 12:20 hashar: rebuilding `php??` images for pcov https://gerrit.wikimedia.org/r/c/integration/config/+/694621 # [[phab:T280170|T280170]]
* 08:42 hashar: - [[phab:T169602|T169602]]
* 10:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813967
* 08:32 hashar: Regenerating Nodepool jessie image to upgrade tox from 1.9.2 to 2.5.0 - [[phab:T169602|T169602]]
* 10:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813932


== 2017-08-01 ==
== 2022-07-14 ==
* 15:45 hashar: Image snapshot-ci-jessie-1501601670 in wmflabs-eqiad is ready  && purging old instances [[phab:T161861|T161861]]
* 18:50 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]]
* 15:44 hashar: Debug: Executing '/usr/bin/npm install -g npm@3.8.3' -  [[phab:T161861|T161861]]
* 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for [[phab:T203215|T203215]]
* 15:34 hashar: Refreshing nodepool Jessie image to bump npm from 2.x to 3.8.x  [[phab:T161861|T161861]]
* 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
* 10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
* 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for [[phab:T268085|T268085]]
* 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI


== 2017-07-28 ==
== 2022-07-13 ==
* 21:11 MaxSem: Dropped table wikigrok_questions from beta enwiki
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813720
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|47a07e06e5e348902a3cdc51cff9c2bed87ab6df}}
* 20:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813707
* 00:17 Krinkle: Testing job insertion on beta cluster from deployment-tin triggers PHP Notice: Undefined index: uuid in EventBus/JobQueueEventBus.php:102, PHP Notice: Undefined index: sha1 in EventBus/JobQueueEventBus.php:99


== 2017-07-26 ==
== 2022-07-12 ==
* 21:35 Reedy: kill two long running update.php jobs on deployment-tin
* 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki ([[phab:T312865|T312865]])
* 13:39 zeljkof: Reloading Zuul to deploy {{Gerrit|8787b4bbe75691e35792864434f8e09f3aaf6e53}}
* 12:04 zeljkof: Reloading Zuul to deploy {{Gerrit|79781d8b08a6dbd4d2e2b5554fbb04154c0acf48}}
* 11:39 zeljkof: Reloading Zuul to deploy {{Gerrit|723ab49039714ae2507ef43a94884de1a6bb347e}}
* 11:31 hashar: realign installed debian packages on integration-slave-jessie-1001 and integration-slave-jessie-1002 -  [[phab:T171724|T171724]]
* 09:25 hashar: deployment-tin deleting temporary l10n cache from July 19th 20:09 at /tmp/scap_l10n_3608512748  1.5G
* 09:24 hashar: deployment-cache-upload04 deployment-cache-text04 upgraded logster 0.0.10-1~jessie1 -> 0.0.10-2~jessie1 - [[phab:T171318|T171318]]


== 2017-07-25 ==
== 2022-07-11 ==
* 16:51 halfak: deploying ores {{Gerrit|835d848}} [[phab:T171505|T171505]]
* 22:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/812934
* 19:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/812467


== 2017-07-24 ==
== 2022-07-10 ==
* 21:56 bearND: Update mobileapps to {{Gerrit|b608ec8}}
* 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet
* 15:03 hashar: Added webperformance Jenkins slave https://integration.wikimedia.org/ci/computer/webperformance/ with a single executor - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk
* 14:40 hashar: Booting integration-webperf instance 2CPU / 2GB RAM / 40G disk. Intended to host webperformance long running jobs . [[phab:T166756|T166756]]
* 11:02 hashar: Removing profile::swift::storage::labs class from deployment-ms-be03 and deployment-ms-be04 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]  [[phab:T171454|T171454]]
* 10:59 hashar: Removing class from deployment-trending01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:54 hashar: Removing classes from deployment-sca02 and deployment-sca03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:32 hashar: Removing profile::etcd from deployment-conf03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:12 hashar: Removing role::mathoid from deployment-mathoid to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:09 hashar: Removing role::changeprop from deployment-changeprop to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:06 hashar: Removing role::ocg from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:02 hashar: Removing role::mobileapps from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]


== 2017-07-21 ==
== 2022-07-09 ==
* 14:55 hashar: Jenkins: upgraded Android Emulator plugin with https://gerrit.wikimedia.org/r/#/c/366253/ && https://gerrit.wikimedia.org/r/#/c/366484/ - [[phab:T150623|T150623]]
* 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
* 14:12 hashar: added novaadmin to deployment-prep as a regular user. That lets MediaWiki OpenStack API list the instances [[phab:T171280|T171280]]
* 17:25 ori: Cherry-picked {{Gerrit|Ief73cc553}} (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
* 13:56 hashar: Created github mirror repo https://github.com/wikimedia/wikibase-wikiba.se  [[phab:T171160|T171160]]
* 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
* 10:46 hashar: Gerrit: created wikibase/wikibase.se repo for Amir1 / [[phab:T171160|T171160]]
* 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
* 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump  # ref [[phab:T312689|T312689]]


== 2017-07-20 ==
== 2022-07-07 ==
* 16:42 hashar: How to fix ssh access on beta cluster instances: https://phabricator.wikimedia.org/T171174#3456966
* 22:42 TheresNoTime: clear stuck beta deployment jobs (again), [[phab:T72597|T72597]]
* 15:30 hashar: deployment-prep : removing project wide puppet classes from https://horizon.wikimedia.org/project/puppet/  All are role::eventlogging::analytics::*
* 21:10 TheresNoTime: clear stuck beta deployment jobs, [[phab:T72597|T72597]]
* 15:08 hashar: removed profile::recommendation_api  from deployment-sca01  to try to fix the ssh access for mobrovac  [[phab:T171173|T171173]]  [[phab:T171174|T171174]]
* 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
* 14:57 zeljkof: reloading Zuul to deploy {{Gerrit|80b9d85}}
* 12:22 hashar: integration: rebooting `integration-agent-docker-1039` [[phab:T312534|T312534]]
* 14:31 hashar: deployment-prep: manually cleaned out the puppet master configuration. It was all screwed up.  Notably I removed bits about the puppetdb
* 10:20 zeljkof: Reloading Zuul to deploy {{Gerrit|80b9d855443a2f572d877b280783110684344c5d}}
* 09:17 hashar: Spawning and pooling integration-slave-docker-1003  as replacement to integration-slave-docker-1000 (broken)  - [[phab:T150502|T150502]]
* 09:03 hashar: Restoring castorby updating all jobs  to point to castor02 ( https://gerrit.wikimedia.org/r/366524 ) Starts with a cold cache :( - [[phab:T171148|T171148]]
* 08:53 hashar: Created castor02.integration.eqiad.wmflabs with puppet role role::ci::castor::server and adding it to Jenkins. Will then update the Jenkins jobs to point to it - [[phab:T171148|T171148]]
* 08:00 hashar: Disabled castor entirely via https://gerrit.wikimedia.org/r/366520 . The instance is broken - [[phab:T171148|T171148]]
* 07:55 hashar: Refreshing all Jenkins jobs defined in JJB in order to then disable castor entirely for [[phab:T171148|T171148]]
* 07:09 _joe_: rebooting castor, jobs are failing, and no one seems able to login
* 07:05 _joe_: adding myself to projectadmins for integration, trying to troubleshoot castor
* 01:38 thcipriani: scap on beta was failing because during the ldap downtime puppet created a shadow mwdeploy user, fixed using vipw and vigr


== 2017-07-19 ==
== 2022-07-05 ==
* 14:43 hashar: Jenkins: uploaded a patched android-emulator plugin for [[phab:T150623|T150623]]  and restarting Jenkins
* 14:17 dwalden: restarted mathoid service on deployment-docker-mathoid01
* 13:55 hashar: Jenkins: added JDK "Debian - OpenJdk 7" with  JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
* 11:39 hashar: Reloaded Zuul for `skip selenium for Wikibase repo/rest-api` https://gerrit.wikimedia.org/r/c/integration/config/+/811258
* 12:54 hashar: Gerrit: created repo integration/jenkinsci/android-emulator-plugin.git owned by access group integration-jenkinsci-android-emulator-plugin  which has Mholloway - [[phab:T170904|T170904]]
* 08:49 hauskatze: Diffusion rORES repository. Changed URI settings: enabled SSH push for mirroring; disabled HTTP {{!}} [[phab:T311390|T311390]]


== 2017-07-18 ==
== 2022-06-30 ==
* 16:26 halfak: manually restarted uwsgi-ores and celery-ores-worker on deployment-sca03
* 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at [[phab:T72597|T72597]]
* 16:19 halfak: manually installed "aspell-el" on deployment-sca03 (work around for ongoing puppet issues)
* 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
* 09:04 hashar: deleted integration-slave-trusty-1006
* 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053
* 03:57 twentyafterfour: Fixed deployment-imagescaler01 by cherry-picking https://gerrit.wikimedia.org/r/#/c/365891/ on deployment-puppetmaster02


== 2017-07-17 ==
== 2022-06-29 ==
* 18:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/365198
* 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
* 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')


== 2017-07-14 ==
== 2022-06-28 ==
* 20:16 Amir1: cpan[1]> install LWP::UserAgent on tin
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]


== 2017-07-13 ==
== 2022-06-27 ==
* 17:04 thcipriani: restarting jenkins for updates
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406


== 2017-07-12 ==
== 2022-06-24 ==
* 20:07 bearND: Update mobileapps to {{Gerrit|d30dae2}}
* 20:52 taavi: added `denisse` as a member
* 18:19 greg-g: where "things" == nodepool instance delete/creation
* 18:18 greg-g: things are back to a bad state, chase etc investigating
* 17:52 greg-g: nodepool is back to making instances and running jobs, thanks Cloud team
* 17:22 greg-g: CI is backed up, only one nodepoll instance running for the last long while, many in building
* 00:35 legoktm: deploying https://gerrit.wikimedia.org/r/364628


== 2017-07-11 ==
== 2022-06-23 ==
* 21:30 legoktm: deploying https://gerrit.wikimedia.org/r/364601
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021


== 2017-07-09 ==
== 2022-06-22 ==
* 01:15 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki ([[phab:T170026|T170026]], [[phab:T165716|T165716]])
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2017-07-07 ==
== 2022-06-21 ==
* 14:53 hashar: deployment-prep: change webproxy http://recommendation-api-beta.wmflabs.org/ to deployment-sca02 (has the proper security rule) - [[phab:T148129|T148129]]
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 14:53 hashar: deployment-prep: add port 9632 to security group "sca" https://horizon.wikimedia.org/project/access_and_security/security_groups/593/  - [[phab:T148129|T148129]]
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 14:03 hashar: Image snapshot-ci-trusty-1499435837 in wmflabs-eqiad is ready
* 13:57 hashar: Nodepool: updating snapshot-ci-trusty
* 13:56 hashar: Nodepool: uploaded new Ubuntu Trusty image


== 2017-07-06 ==
== 2022-06-20 ==
* 17:28 thcipriani: committed changes to modules/kafkatee on deployment-puppetmaster02 since having them uncommitted broke git-sync-upstream
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 16:20 hashar: Deleting Nodepool snapshot snapshot-ci-jessie-1499350442 -  faulty php7.0-sqlite package that breaks phan jobs - [[phab:T169904|T169904]]
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 15:29 hashar: deployment-cache-upload04 manually ran apt-get upgrade to downgrade ldap-utils and libldap-2.4-2  (caused puppet failure)
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 14:14 hashar: regenerating mediawiki-core-qunit-selenium-jessie jenkins job
* 12:05 hashar: deployment-prep created Web proxy for recommendation-api-beta.wmflabs.org -> http://10.68.20.183:9632  (deployment-sca01) for schana
* 02:38 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/363519


== 2017-07-04 ==
== 2022-06-17 ==
* 14:10 hashar: manually upgraded apache2 on deployment-puppetmaster02  see [[phab:T159254|T159254]]
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 13:33 hashar: beta cluster puppet is broken: Error: Could not send report: Connection refused - connect(2) for "deployment-puppetmaster02.deployment-prep.eqiad.wmflabs" port 8140
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 09:28 hashar: gerrit:  marking read-only  mediawiki/extensions/Nonlinear - [[phab:T169519|T169519]]


== 2017-07-03 ==
== 2022-06-16 ==
* 11:34 hashar: jenkins: refreshing all jobs and updating the castor-save bit ( https://gerrit.wikimedia.org/r/#/c/361843/ )
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]


== 2017-06-30 ==
== 2022-06-15 ==
* 08:16 hashar: Gerrit: changing repos to read-only: analytics/kraken analytics/kraken/deploy  analytics/vagrant/kraken - [[phab:T169303|T169303]]
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster


== 2017-06-29 ==
== 2022-06-14 ==
* 23:17 legoktm: deploying https://gerrit.wikimedia.org/r/362314
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production


== 2017-06-28 ==
== 2022-06-13 ==
* 15:55 hashar: beta: git gc  mediawiki repos in /srv/mediawiki-staging
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 15:47 hashar: beta: git -C /srv/deployment/ores/deploy/submodules/editquality gc  (saving 380MBytes)
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
* 15:33 hashar: running git gc under /srv/mediawiki-staging
* 14:43 hashar: pypi.python.org is back again - [[phab:T169091|T169091]]
* 14:33 elukey: running alter tables on the EL database in deployment-eventlogging03.deployment-prep.eqiad.wmflabs
* 14:06 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates and [[phab:T169091|T169091]]
* 14:04 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates
* 10:06 hashar: Unblocked beta cluster jenkins job. Have been stalled for a while


== 2017-06-27 ==
== 2022-06-12 ==
* 22:58 Amir1: cherry-picking gerrit:360891/3
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 22:42 Amir1: cherry-picking gerrit:360891/2
* 21:58 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'string' --property-id P34
* 18:31 hashar: Image snapshot-ci-jessie-1498587497 in wmflabs-eqiad is ready - [[phab:T169004|T169004]]
* 18:18 hashar: Regenerating Jessie nodepool image to hopefulyl bring back hhvm-tidy package  - [[phab:T169004|T169004]]
* 17:39 Amir1: running mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'external-id' --property-id P34


== 2017-06-26 ==
== 2022-06-10 ==
* 22:24 halfak: deploying ores-prod-deploy:82dfd56 to beta (note: [[phab:T168099|T168099]])
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 22:20 halfak: deploying ores-prod-deploy:82dfd56 to beta
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 20:33 bearND: Update mobileapps to {{Gerrit|0b05026}}
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 18:44 hashar: nodepool image-delete 1636  # Deletes snapshot-ci-trusty-1498491445 which lack nodejs when we still need it.
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check
* 18:23 twentyafterfour: renamed previously active image to 'image-ci-trusty_bad_20170626'
* 18:22 twentyafterfour: reverted nodepool image-ci-trusty to previous version 'image-ci-trusty-old_20170626'
* 15:41 hashar: Image snapshot-ci-trusty-1498491445 in wmflabs-eqiad is ready
* 15:34 hashar: Rebuilding nodepool image for trusty and regenerating snapshots
* 09:19 hashar: gerrit: marked wikimedia/bugzilla/* repos read-only


== 2017-06-24 ==
== 2022-06-09 ==
* 06:02 legoktm: deployment-flourine02 /srv partition is alerting on low disk space but once logs get automatically gzip'd it should be fine
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration


== 2017-06-23 ==
== 2022-06-08 ==
* 20:59 hasharAway: deployment-db03 reinstall ldap-utils, libldap-2.4-2  2.4.44+dfsg-4~bpo8+1 > 2.4.41+dfsg-1+wmf1
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 20:54 hasharAway: apt-get upgrade deployment-elastic06
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134


== 2017-06-22 ==
== 2022-06-06 ==
* 19:02 Amir1: cherry-picking gerrit:360891/1 ([[phab:T163922|T163922]])
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group .  That also makes him an owner to wikimedia/discovery/* - [[phab:T168588|T168588]]
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group .  That also makes him an owner to wikimedia/discovery/*
* 08:18 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x! - [[phab:T168511|T168511]]
* 08:16 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x!
* 08:12 hashar: deployment-prep: upgraded puppet to 3.8.5  on all instances


== 2017-06-21 ==
== 2022-06-02 ==
* 20:03 bearND: Update mobileapps to {{Gerrit|21f771d}}
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 19:54 hashar: deployment-tin stopped keyholder and armed it
* 11:26 hashar: Restarting Jenkins on contint2001
* 19:25 hashar: hard rebooting deployment-db04
* 11:19 hashar: Restarting Jenkins on releases1002
* 19:20 hashar: hard rebooting deployment-db03
* 18:52 hashar: Removing /etc/apt/sources.list.d/wikimedia_mariadb.list  (content: deb http://apt.wikimedia.org/wikimedia precise-wikimedia mariadb  )
* 18:51 hashar: fixing up apt config on deployment-db03 and deployment-db04 /  upgrade packages and kernel / reboot
* 17:02 hashar: upgrading kernel and puppet on deployment-mcs01 deployment-restbase01 and deployment-restbase02 - [[phab:T168541|T168541]]
* 17:00 hashar: upgrading kernel and puppet on deployment-changeprop and deployment-conf03 - [[phab:T168541|T168541]]
* 16:56 hashar: upgrading kernel and puppet on deployment-aqs01 deployment-aqs02 and deployment-aqs03 - [[phab:T168541|T168541]]
* 16:38 hashar: rebooting deployment-cache-upload04 and deployment-cache-text-04 - [[phab:T168541|T168541]]
* 16:29 hashar: upgrading deployment-apertium02 and deployment-eventlogging04  - [[phab:T168541|T168541]]
* 16:23 hashar: upgrade and reboot deployment-prometheus01
* 16:11 hashar: rebooting deployment-ms-fe02
* 16:11 hashar: rebooting deployment-ms-be04
* 16:09 hashar: rebooting deployment-ms-be03
* 16:03 hashar: upgrading deployment-ms-fe02  deployment-ms-be03 and deployment-ms-be04
* 15:57 hashar: apt-get upgrade and reboot of deployment-memc04 and deployment-memc05
* 15:52 hashar: rebooting deployment-etcd-01
* 15:48 hashar: apt-get upgrade deployment-etcd-01
* 15:35 hashar: deployment-prep changing Varnish director for citoid from citoid.wmflabs.org to citoid-beta.wmflabs.org  ( via https://horizon.wikimedia.org/project/prefixpuppet/ ) - [[phab:T168519|T168519]]
* 14:41 hashar: deployment-tmh01 is down for some reason
* 14:21 hashar: deployment-prep: force running puppet on all instances
* 14:17 hashar: finally fixed puppet on deployment-prep !
* 14:02 hashar: deployment-puppmaster (cd /etc/puppet && ln -s /var/lib/git/operations/puppet/manifests && ln -s /var/lib/git/operations/puppet/modules)
* 13:26 hashar: deployment-prep: puppet master got erroneously upgrade to puppet* 4.8. Roll it back to 3.8 which fail, and then back to 3.7!
* 12:47 hashar: broke deployment-prep puppet master while upgrading it :(
* 12:28 hashar: deployment-imagescaler01 removed puppetmaster and puppetmaster-common packages
* 12:04 hashar: apt-get dist-upgrade on deployment-mediawiki hosts
* 11:59 hashar: armed keyholder on deployment-tin and deployment-mira
* 11:15 hashar: deployment-cache-text04 : apt-get dist-upgrade
* 11:12 hashar: varnish fails on deployment-cache-text04
* 11:08 hashar: deployment-prep : rebooting deployment-tin deployment-mira deployment-cache-text04 deployment-cache-upload04
* 11:00 hashar: deployment-prep apt-get upgrade and reboot all hosts
* 10:21 hashar: deployment-zotero01  apt-get upgrade and rebooted
* 09:59 hashar: integration: removing swift / python-swift from integration-puppetmaster01
* 09:57 hashar: Upgrading puppet 3.7.2 .. 3.8.5 on integration-slave-docker-1001 and integration-slave-docker-1002
* 09:39 hashar: integration: deleting swift and and swift-storage-01  unused
* 09:38 hashar: upgrading/Rebooting all instances from integration project to catch up with Linux kernel upgrades


== 2017-06-20 ==
== 2022-05-31 ==
* 19:25 hashar: Nodepool rate being bumped from 1 query per 6 seconds to 1 query per 5 seconds ( https://gerrit.wikimedia.org/r/#/c/358601/ )
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* afk: deployment-tin stuck on post-merge queue for the past 13 hours, unstuck now
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2017-06-19 ==
== 2022-05-30 ==
* afk: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/360091/
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 08:29 hashar: Gerrit: added Ladsgroup to 'mediawiki' group - [[phab:T165860|T165860]]
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]


== 2017-06-18 ==
== 2022-05-28 ==
* 19:26 Reedy: Re-enabled beta-update-databases-eqiad as wikidatawiki takes < 10 minutes [[phab:T168036|T168036]] [[phab:T167981|T167981]]
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]
* 19:25 Reedy: A lot of items on beta wikidatawiki deleted [[phab:T168036|T168036]] [[phab:T167981|T167981]]


== 2017-06-16 ==
== 2022-05-27 ==
* 23:41 Reedy_: also deleting a lot of Property:P* pages on beta wikidatawiki [[phab:T168106|T168106]]
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 22:55 Reedy: deleting Q100000-Q200000 on beta wikidatawiki [[phab:T168106|T168106]]
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 19:04 Reedy: disabled beta-update-databases-eqiad because it's not doing much useful atm
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|18a50a707eac0bcdd88f48f2321af78ee399a4eb}}
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 14:40 hashar: integration-slave-jessie-1001  apt-get upgrade to downgrade python-pbr to 0.8.2 as pinned since [[phab:T153877|T153877]].  /usr/bin/unattended-upgrade magically upgraded it for some reason
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 06:49 Reedy: script upto `Processed up to page 336425 (Q235372)`... hopefully it's finished by morning
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 03:13 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 03:10 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep
* 02:23 Reedy: cherry-picked https://gerrit.wikimedia.org/r/#/c/354932/ onto beta puppetmaster


== 2017-06-15 ==
== 2022-05-26 ==
* 16:34 RainbowSprinkles: deployment-prep: Disabled database updates for awhile, running it by hand
* 18:33 dancy: Updated Jenkins beta-* job configs
* 10:39 hashar: apt-get upgrade on deployment-tin
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 00:52 thcipriani: deployment-tin jenkins agent borked for 4 hours, should be fixed now
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch


== 2017-06-14 ==
== 2022-05-25 ==
* 12:24 hashar: gerrit: marked mediawiki/skins/Donate has read-only ( https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/skins/Donate ) - [[phab:T124519|T124519]]
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 15:46 dancy: Restarted apache2 on gerrit1001


== 2017-06-13 ==
== 2022-05-24 ==
* 22:05 hashar: Zuul resarted manually from a terminal on contint1001. It does not have any statsd configuration so we will miss metrics for a bit till it is restarted properly.
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 21:13 hashar: Gracefully restarting Zuul
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 20:37 hashar: Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4)
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 20:31 hashar: Nodepool: deleted a bunch of Trusty instances. It scheduled lot of them that are taking slots in the pool. Better have jessie nodes to be spawned instead since there is high demand for them
* 20:19 hashar: deployment-prep: added Polishdeveloper to the "importer" global group. https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/Polishdeveloper  - [[phab:T167823|T167823]]
* 18:47 andrewbogott: root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter"
* 18:46 andrewbogott: using salt to "apt-get -y install facter" on all deployment-prep instances
* 18:38 andrewbogott: restarting apache2 on deployment-puppetmaster02
* 18:37 andrewbogott: doing a git fetch and rebase for deployment-puppetmaster02
* 17:00 elukey: hacking apache on mediawiki05 to test rewrite rules
* 16:04 Amir1: cherry-picked 357985/4 on puppetmaster
* 15:59 halfak: deployed ores-prod-deploy:862aea9
* 13:47 hashar: nodepool force running puppet for: lower min-ready for trusty [puppet] - https://gerrit.wikimedia.org/r/356466
* 10:53 elukey: rolling restart of all kafka brokers to pick up the new zookeper change (only deployment-zookeeper02 available)
* 10:36 elukey: delete deployment-zookeeper01 (old trusty instance, replaced with a jessie one)
* 09:50 elukey: big refactoring for zookeeper merged in operations/puppet - https://gerrit.wikimedia.org/r/#/c/354449 - ping the Analytics team for any issue


== 2017-06-12 ==
== 2022-05-23 ==
* 14:22 hashar: Image snapshot-ci-trusty-1497276913 in wmflabs-eqiad is ready
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 14:15 hashar: Nodepool: regenerating Trusty images to confirm that removal of keystone admin_token is a noop for nodepool - [[phab:T165211|T165211]]
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 12:44 hashar: Image snapshot-ci-jessie-1497270581 in wmflabs-eqiad is ready
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 12:30 hashar: nodepool: refreshing Jessie snapshot to upgrade HHVM from 3.12 to 3.18 - [[phab:T167493|T167493]] [[phab:T165074|T165074]]
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 08:47 hashar: deployment-prep : salt -v '*' cmd.run 'apt-get clean'
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2017-06-09 ==
== 2022-05-22 ==
* 20:30 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358092/1
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 18:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358067/3


== 2017-06-07 ==
== 2022-05-21 ==
* 17:49 elukey: forced /usr/local/bin/git-sync-upstream manually on puppetmaster02
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 17:30 elukey: manually fixed rebase issue for operations/puppet on puppetmaster02 (empty commit due to the change for scap3 and jobrunners)
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]
* 09:33 elukey: restart kafka brokers to pick up the new zookeeper settings
* 09:00 elukey: adding deployment-zookeeper02.eqiad.wmflabs to Hiera:deployment-prep
* 08:43 gehel: upgrading kibana to v5.3.3 on deployment-logstash2
* 08:35 gehel: rolling back to kibana 5.3.2, incompatible elasticsearch version
* 08:28 gehel: upgrading kibana to v5.4.1 on deployment-logstash2


== 2017-06-06 ==
== 2022-05-20 ==
* 14:34 hashar: deleting buildlog.integration.eqiad.wmflabs  was mean to receive Jenkins logs in ElasticSearch. We are experimenting with relforge1001.eqiad.wmnet now - [[phab:T78705|T78705]]
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
* 12:37 hashar: Removing HHVM from permanent Trusty slaves
* 10:44 elukey: running eventlogging_cleaner.py (https://gerrit.wikimedia.org/r/#/c/356383/) on eventlogging to test the cleaning of old events
* 09:24 hashar: Deleting deployment-phab02 instance. Has been shut off since April 23rd -  [[phab:T167090|T167090]]
* 07:51 hashar_: Fixed puppet on deployment-aqs instances


== 2017-06-05 ==
== 2022-05-19 ==
* 15:38 elukey: manually hacking deployment-jobrunner02.deployment-prep.eqiad.wmflabs to test a new config
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already


== 2017-06-02 ==
== 2022-05-18 ==
* 19:51 hashar: integration: granted ebernhardson sudo
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 12:12 hashar: jenkins: rebuild logstash plugin from HEAD of master for jenkins 2 back compat. logstash-1.2.0-4-gbcbc19e - [[phab:T78705|T78705]]
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]


== 2017-06-01 ==
== 2022-05-17 ==
* 20:14 bearND: Update mobileapps to {{Gerrit|c4dc72d}}
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 20:12 mdholloway: killed the running emulator processes on integration-slave-jessie-android to get it booting again following yesterday's gerrit outage
* 13:39 hashar: Gerrit: change integration.git project to "Rebase if Necessary" with "Allow content merges" - [[phab:T131008|T131008]]
* 13:10 hashar: Gerrit allow content merge for integration/config ( https://gerrit.wikimedia.org/r/#/admin/projects/integration/config )  - [[phab:T131008|T131008]]
* 08:03 hashar: Purged all mysql bin files from deployment-db03 ( rm -fR /srv/sqldata/T166060 ) - [[phab:T166060|T166060]]


== 2017-05-31 ==
== 2022-05-16 ==
* 20:21 hashar: Jenkins: upgrading git-client-plugin 2.4.5..2.4.6 [[phab:T166557|T166557]]
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 07:50 hashar: deployment-db04:  mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 07:49 hashar: deployment-db03: mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809


== 2017-05-30 ==
== 2022-05-14 ==
* 22:08 hasharAway: Changed integration/config.git submit type from "Fast forward only" to "Rebase if Necessary"  [[phab:T131008|T131008]]
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request


== 2017-05-29 ==
== 2022-05-13 ==
* 14:44 elukey: reverted previous config on redis01
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 14:36 elukey: set redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 300 on redis01 as test (rollback: redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 0)
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 10:22 hashar: force refreshed Nodepool Trusty images. Was stuck somehow
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 10:06 hashar: deployment-tin rm -fR /usr/src/hhvm  [[phab:T166492|T166492]]
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 09:51 hashar: deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json  [[phab:T166492|T166492]]
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2017-05-26 ==
== 2022-05-12 ==
* 09:20 elukey: installing hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 07:20 elukey: hacking on jobrunner02 in deployment-prep
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 01:28 bearND: Update mobileapps to {{Gerrit|db6493c}}
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2017-05-25 ==
== 2022-05-11 ==
* 19:46 hashar: deployment-tin manually cleaning disk space
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 16:44 elukey: restored hhvm on jobrunner02
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
* 16:03 bearND: Update mobileapps to {{Gerrit|946fe1f}}
* 10:33 elukey: manual install of hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02 to test a fix for the Redis.php lib
* 02:46 RainbowSprinkles: running `mwscript extensions/Flow/maintenance/FlowUpdateUserWiki.php --wiki=enwiki` in a screen on deployment-tin, probably going to take all night


== 2017-05-24 ==
== 2022-05-10 ==
* 16:04 hashar: rebooting integration-slave-trusty-1003 to catch up with kernel upgrade
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 12:22 hashar: deployment-prep: finished rebase of puppet.git
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 10:19 hashar: deployment-prep rebased  puppet repo with: git rebase -X theirs
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 10:10 hashar: deployment-prep : resetting puppet master to last known snapshot snapshot-20170523T0010  . All cherry picks got deleted
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 10:09 hashar: deployment-etcd-01: fixed puppet run
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:38 moritzm: updated puppet on deployment-puppetmaster02 to 3.8.5-2~bpo8+2
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377


== 2017-05-23 ==
== 2022-05-09 ==
* 16:55 RainbowSprinkles: there was no data
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 16:55 RainbowSprinkles: dropped flow_ext_ref from commonswiki on beta. schema migration is busted, going to let it recreate table
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 08:20 hashar: Updating Nodepool snapshot-ci-trusty
* 08:19 hashar: Regenerated Nodepool base image for Trusty. Got rid of hhvm from it


== 2017-05-22 ==
== 2022-05-08 ==
* 12:11 greg-g: ran git prune and rm'd the gc.log file
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 11:40 greg-g: gjg@deployment-tin:/srv/mediawiki/.git/gc.log has warning: There are too many unreachable loose objects; run 'git prune' to remove them.


== 2017-05-21 ==
== 2022-05-06 ==
* 12:05 Reedy: deployment-tin is back online
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]
* 10:41 Reedy: disabled jerkins on deployment-tin again
* 09:10 greg-g: beta-update-database-eqiad has been hitting the timelimit since May 19th
* 09:02 Reedy: brought deployment-tin back online a while ago


== 2017-05-20 ==
== 2022-05-05 ==
* 09:10 greg-g: executers are running again
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 09:02 greg-g: All executers in Jenkins are "offline" including the permament ones
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins builds from https://integration.wikimedia.org/ci/view/Beta/


== 2017-05-19 ==
== 2022-05-04 ==
* 19:05 mutante: fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab. remove again from instance level for phab-01
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 18:40 mutante: deployment-phab01 still has puppet error "Could not find class role::phabricator::main" and that should simply be removed from it, but i can NOT find it in Horizon, i checked instance config, project config, the "Other" section, the "All classes" tab. Because it's gone. But how do i fix the instance config then?
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 18:39 mutante: applying role::phabricator_server on instance deployment-phab01 (it had error, could not find role::phabricator::main and the name changed in role/profile conversion)
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2017-05-15 ==
== 2022-05-03 ==
* 10:46 addshore: enabled beta-code-update-eqiad for some testing
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 10:38 addshore: temporarily disabled beta-code-update-eqiad for some testing
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2017-05-13 ==
== 2022-05-02 ==
* 20:31 bd808: Deleted stuck mediawiki-core-doxygen-publish job. Jenkins had it marked for a particular nodepool instance that was offline.
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]


== 2017-05-12 ==
== 2022-04-29 ==
* 13:12 hashar: Trying to refresh Nodepool Jessie image. Should get HHVM pinned to 'experimental' component => 3.12.x
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]


== 2017-05-11 ==
== 2022-04-28 ==
* 20:43 hashar: nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit.  Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. [[phab:T165074|T165074]]
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 12:57 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/353282/
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC


== 2017-05-10 ==
== 2022-04-27 ==
* 20:28 bearND: Update mobileapps to {{Gerrit|75b135e}}
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 18:32 mutante: deployment-tin/mira: the change of the role class name was because of https://gerrit.wikimedia.org/r/#/c/344728/ which moved deployment::server to profile/role structure. both instances configured accordingly now. the remaining issue with "id_rsa.bromine" should be all unrelated
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029
* 18:28 mutante: deployment-mira: configure puppet config in horizon, remove "role::deployment::server", use correct new name "role::deployment_server" (moved to profile). (a bit tricky because then in Horizon it seems to disappear from the "others" section, but if you click the "all" tab you get to see the class names
* 18:12 mutante: deployment-tin: puppet run now ok, except ":Upload/File[/var/lib/releases/.ssh/id_rsa.bromine.eqiad.wmnet]: Could not evaluate:" this should be an unrelated issue
* 18:05 mutante: deployment-tin: configure to use role::deployment_server (instead of deployment::server), for some reason now Horizon shows _nothing_ under "other classes" where this was before
* 17:58 mutante: deployment-tin: deleting puppet lock file (claimed it was running but also didnt run since > 900 min), looking at fixing deployment::server role name change
* 15:26 elukey: refresh cherry pick gerrit/352582 on puppet master (rebase -i to remove, then cherry pick)
* 14:34 elukey: cherry pick gerrit/352582 to puppet master
* 12:35 hashar: deployment-prep: git -C /srv/mediawiki-staging/php-master/extensions rm --cached SemanticFormsInputs
* 08:04 hashar: merging 'composer test' into mwext-testextension-* jobs https://gerrit.wikimedia.org/r/#/c/352160/ - [[phab:T161895|T161895]]


== 2017-05-09 ==
== 2022-04-26 ==
* 12:44 hashar: deployment-ircd upgrading puppet 3.7.2 => 3.8.5
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 12:19 hashar: Unbroke puppet on deployment-irc and deployment-urldownloader . Both choked on a ruby one-liner, fixed via https://gerrit.wikimedia.org/r/#/c/336840/
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2017-05-08 ==
== 2022-04-25 ==
* 21:42 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/351131/
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 00:57 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/351130
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI


== 2017-05-06 ==
== 2022-04-20 ==
* 01:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/352154
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service


== 2017-05-05 ==
== 2022-04-18 ==
* 20:11 hasharDisappear: Mass pushing addition of jakub-onderka/php-console-highlighter  to all mediawiki extensions having php-parallel-lint ( example: https://gerrit.wikimedia.org/r/#/c/352215/ )
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 09:20 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350817 on deployment-puppetmaster02
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 09:17 addshore: temporarily enabled beta-code-update-eqiad
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 09:12 addshore: temporarily disabled beta-code-update-eqiad
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.
* 08:30 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350220/ on deployment-puppetmaster02


== 2017-05-04 ==
== 2022-04-14 ==
* 10:47 hashar: puppet ca destroy deployment-zookeeper01.eqiad.wmflabs
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 10:46 hashar: puppet ca destroy deployment-ores-redis-02.deployment-prep.eqiad.wmflabs  (no such instance)
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 10:46 hashar: puppet ca sign deployment-ores-redis-02.deployment-prep.eqiad.wmflabs
* 10:39 hashar: Removing puppetmaster: puppetmaster.thumbor.eqiad.wmflabs  from deployment-imagescaler01 - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'  - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'
* 10:31 hashar: deployment-phab01 / deployment-imagescaler01 rm /etc/puppet/puppet.conf.d/10-self.conf  - [[phab:T153319|T153319]]
* 10:29 hashar: Unbroke puppet on deployment-imagescaler01 and removing role::puppetmaster::self - [[phab:T153319|T153319]]
* 10:16 hashar: Unbroke puppet on deployment-phab01 - [[phab:T153319|T153319]]
* 07:30 hashar: deployment-prep: adding TTO (This, that and the other) as a project member to grant shell access - [[phab:T163887|T163887]]


== 2017-05-03 ==
== 2022-04-12 ==
* 17:39 mdholloway: (this concerns integration-slave-jessie-android)
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 17:37 mdholloway: enabled automatic Android component installation for the Android Gradle plugin, rebuilt the SDK, and deleted the old one
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 15:54 hashar: Granted sudo right for Niedzielski accounts on Android CI slave.  Already has it with the other labs account Sniedzielski - [[phab:T164388|T164388]]
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 15:38 hashar: Granted mdholloway (mobile team) full sudo access on integration labs project so he can reach integration-slave-jessie-android -  [[phab:T164388|T164388]]
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141


== 2017-05-02 ==
== 2022-04-08 ==
* 21:14 hashar: Manually cancelled a few mediawiki-core-jsduck-publish and mediawiki-core-doxygen-publish job in Jenkins build queue. They seems to deadlock Jenkins somehow :(
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 19:59 hashar: Regenerate jobs selenium-GettingStarted from JJB - [[phab:T164296|T164296]]
* 19:51 hashar: Jenkins: rolling back Performance plugin from 2.2 to 2.0 due to an exception / failure to find a junit xml file.  [[phab:T164296|T164296]]
* 19:02 hashar: Added multichill ( https://github.com/multichill ) to the Wikimedia Github organization
* 10:21 godog: bounce varnish and varnish-frontend on deployment-cache-upload04
* 10:16 godog: upgrade scap on deployment-tin to overcome AttributeError: Lock instance has no attribute 'get_lock_excuse'
* 09:41 godog: flip deployment-cache-upload04 to deployment-ms-fe02 - [[phab:T162247|T162247]]
* 08:17 hashar: Reconfigured all Jenkins jobs via jjb


== 2017-05-01 ==
== 2022-04-07 ==
* 20:39 hashar: Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ )
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 20:26 hashar: nodepool: deleting alien instance ci-trusty-wikimedia-631443 4e66ad7e-b9d3-4af1-b559-{{Gerrit|3f54968d376e}}
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]
* 02:49 TimStarling: on puppetmaster02 manually updating /etc/conftool/data-local
* 02:37 TimStarling: on puppetmaster02 updated cherry pick for https://gerrit.wikimedia.org/r/#/c/347360


== 2017-04-27 ==
== 2022-04-06 ==
* 18:18 urandom: deployment-prep: restarting cassandra-metrics-collector on deployment-restbase0[1-2]
* 20:03 thcipriani: rebooting phabricator
* 07:26 Amir1: cherry-picking 348184/4 ([[phab:T161563|T161563]])
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]


== 2017-04-26 ==
== 2022-04-04 ==
* 23:36 urandom: removing r/350485 from deployment-prep
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 21:53 urandom: cherry-picking r/350485 to deployment-prep
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 20:20 bearND: Update mobileapps to {{Gerrit|14bd4a5}}
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 15:24 godog: add new deployment-ms-be0[34] backends to swift in deployment-prep - [[phab:T162247|T162247]]
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul


== 2017-04-25 ==
== 2022-04-02 ==
* 21:57 halfak: deployed ores {{Gerrit|cc12103}}
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
* 06:46 Amir1: uncherry-pick {{Gerrit|f6ce64e99a}} and {{Gerrit|225b8d4e82}} ([[phab:T161563|T161563]])


== 2017-04-22 ==
== 2022-03-31 ==
* 20:17 hashar: Added FlorianSW to Github organization "wikimedia" (no team though)
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957


== 2017-04-21 ==
== 2022-03-29 ==
* 12:25 hashar: [[phab:T104048|T104048]] zuul enqueue --trigger gerrit --pipeline postmerge --project AhoCorasick --change 345433,1
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 09:32 hashar: Zuul: deploying "Decouple repos from mediawiki gate queue"  {{Gerrit|7a79f752363a}}  / [[phab:T107529|T107529]]
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 09:30 elukey: hack reverted on tin and scap pull performed on jobrunner02
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771


== 2017-04-20 ==
== 2022-03-28 ==
* 17:09 elukey: reverted hack on deployment-tin (apparently no effects on the jobrunner)
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 16:41 elukey: temporary disable puppet on deployment-tin to remove jobrunner02 from scap dsh; manually enable persistent connection between it and rdb redis hosts
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]


== 2017-04-19 ==
== 2022-03-27 ==
* 16:34 hashar: deleted nodepool alien ci-jessie-wikimedia-613597
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]
* 09:20 hashar: apt-get upgrade deployment-tin  deployment-mira
* 09:16 hashar: apt-get upgrade on deployment-mx deployment-redis01 deployment-redis02 deployment-cache-text04
* 02:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/348896


== 2017-04-18 ==
== 2022-03-26 ==
* 14:29 hashar: unbreaking integration puppetmaster. Broke it when upgrading the puppet package :(
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 14:09 hashar: integration: upgrade puppet on Jessie permanent slaves  3.7.2 -> 3.8.5  (and add ruby-rgen).  Done via: salt -v '*' pkg.upgrade
* 13:17 elukey: upgrade deployment-jobrunner02 to hhvm 3.18.2+wmf2 - [[phab:T162354|T162354]]
* 10:07 godog: upgrade swift to 2.2.0 on deployment-ms*


== 2017-04-14 ==
== 2022-03-25 ==
* 12:29 hashar: Delete integration-c1 instance (32GB RAM) on labvirt1004. It was used as a workaround for [[phab:T161006|T161006]]
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 08:17 hashar: beta: cherry picking again 348184/4 'service: use gzip for logging in uwsgi' for [[phab:T161563|T161563]]
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 08:03 hashar: beta: resetting puppetmaster to last good tag snapshot-20170414T0030 A cherry pick for [[phab:T161563|T161563]] end up dropping three patches which broke other parts of the infrastructure
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 07:52 hashar_: Puppet failing on deployment-tin and deployment-mira . Some patches have been dropped from the puppet master :-((
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 00:59 Amir1: three cherry-picks failed to merge, skipped them {{Gerrit|93dad5bec8e937ef93bdd63046b0bbbf14ad9722}} {{Gerrit|92c7d0b002a02ff46a00e79d6d89fe83d5f65c17}} {{Gerrit|21d60a478ffb21160049679eea235bdb1010a489}}
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord
* 00:45 Amir1: cherry-picking 348184/1 ([[phab:T161563|T161563]])


== 2017-04-13 ==
== 2022-03-24 ==
* 15:37 hashar: deployment-mediawiki04 clearing /var/cache/hhvm/fcgi.hhbc.sq3
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 15:15 hashar: Deployed  mediawiki-core-qunit-selenium-jessie job (runs qunit + selenium with webdriverio)  https://gerrit.wikimedia.org/r/#/c/347587/ - [[phab:T139740|T139740]]
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck


== 2017-04-12 ==
== 2022-03-23 ==
* 15:14 hashar: rm -fR  /mnt/home/jenkins-deploy/.android/build-cache/*  # [[phab:T162635|T162635]]
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 14:56 hashar: integration-slave-jessie-1001 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 14:54 hashar: integration-slave-jessie-1002 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 10:37 hashar: Jenkins email-ext plugin got upgraded. Some groovy templating might be prevented and would have to be reviewed/approved via https://integration.wikimedia.org/ci/scriptApproval/
* 08:52 hashar: Cancelled bunch of mediawiki-core-doxygen-publish jobs that were keeping the queue busy/deadlocked builds.   Should be moved to poll scm instead ( [[phab:T115755|T115755]] )


== 2017-04-11 ==
== 2022-03-22 ==
* 15:59 hashar: integration-config-tox-jessie job is broken due to the JJB upgrade
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 15:40 hashar: Upgraded JJB to latest master {{Gerrit|4f77324f}}  with a couple cherrypicks on top of that. 022738f8...edebce7f  [[phab:T162674|T162674]]
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945
* 15:36 hashar: Updating selenium-* jobs configuration for the performance plugin due to JJB upgrade [[phab:T162674|T162674]]
* 15:24 hashar: Adding parameter ZUUL_VOTING to all Jenkins jobs due to JJB upgrade [[phab:T162674|T162674]]
* 15:13 hashar: Forced updated jenkins-job-builder 86478421...022738f8 - [[phab:T162674|T162674]]
* 13:44 hashar: Forced updated jenkins-job-builder 1639a86e...86478421 - [[phab:T162674|T162674]]
* 13:44 hashar: Updating all Jenkins jobs using the git plugin due to JJB change  {{Gerrit|cdfeb7bf66b0eacfed3eaf2a77813d65ab0e29f2}} -  [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e  - [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e
* 10:41 hashar: Enable webdriver.io  browser tests for MediaWiki core - https://gerrit.wikimedia.org/r/#/c/324719/ - [[phab:T139740|T139740]]
* 09:50 hashar: Regenerating MediaWiki doxygen documentations for all 1.23.x releases.
* 08:55 hashar: Retriggering MediaWiki doxygen publishing job for 1.26.0 - [[phab:T162506|T162506]] : zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.26.0 --newrev {{Gerrit|981ec62244e4806a16bda804dda1d14cb5d7f193}}


== 2017-04-10 ==
== 2022-03-21 ==
* 21:17 hashar: marked a nodepool node online manually. The instance was up but Jenkins failed to reach it due to some SEVERE: I/O error in channel
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 20:52 hashar: integration-slave-jessie-1001 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete # [[phab:T162635|T162635]]
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]
* 20:49 hashar: integration-slave-jessie-1002 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete  # [[phab:T162635|T162635]]
* 20:08 bearND: Update mobileapps to {{Gerrit|1695900}}


== 2017-04-06 ==
== 2022-03-18 ==
* 16:36 halfak: staging ores:554ea12
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 12:23 hashar: Image snapshot-ci-trusty-1491480759 in wmflabs-eqiad is ready
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 12:13 hashar: Updating Nodepool Trusty image to let Linux overcommit memory ( https://gerrit.wikimedia.org/r/#/c/346634/ )
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)


== 2017-04-05 ==
== 2022-03-17 ==
* 13:34 ema: testing possible fix for [[phab:T162035|T162035]] on deployment-ms-fe01
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]


== 2017-04-04 ==
== 2022-03-16 ==
* 21:29 hashar: contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 21:26 hashar: contint2001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 20:58 hashar: integration: purging precise cow images from integration-slave-jessie-1001 and integration-slave-jessie-1002  ( https://gerrit.wikimedia.org/r/#/c/345836/ )
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 20:58 hashar: rebased integration puppet master
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep
* 20:02 legoktm: deploying https://gerrit.wikimedia.org/r/346348


== 2017-04-03 ==
== 2022-03-15 ==
* 20:43 bearND: Update mobileapps to {{Gerrit|fdd4e31}}
* 18:26 brennen: gitlab: removed most existing /people groups
* 20:39 hashar: Nodepool: holding instance ci-trusty-wikimedia-597386  in an attempt debug Wikibase/Scribunto memory usage exploding [[phab:T125050|T125050]]
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 20:37 hashar: jenkins: disabled/reenabled gearman plugin to unlock the beta cluster related jobs
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 09:17 hashar: deployment-jobrunner02 : cherry picked a monkey patch for Redis::close() to prevent it from sending QUIT command ( https://gerrit.wikimedia.org/r/#/c/346117/ ) - [[phab:T125735|T125735]]
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2017-04-01 ==
== 2022-03-14 ==
* 09:48 Sagan: puppet on deployment-tin looks like it is not running properly
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2017-03-29 ==
== 2022-03-11 ==
* 23:51 Krinkle: Free up space on integration-slave-jessie-1001 by removing old /srv/jenkins-workspace and /srv/pbuilder dirs
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
* 19:57 thcipriani: added --force flag for scap in beta-scap-eqiad temporarily
* 18:41 ebernhardson: upgrading elasticsearch and kibana to 5.1.2 on deployment-logstash2 to test puppet+integration prior to prod deployment
* 15:18 hashar: Delete a 32GB instance integration-ci  - [[phab:T161006|T161006]]


== 2017-03-28 ==
== 2022-03-10 ==
* 19:53 hashar: Populating package manager cache of oojs-ui-npm-run-jenkins-node-6-jessie  by manually triggering a build with ZUUL_PIPELINE=postmerge  [[phab:T155483|T155483]]
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 19:34 hashar: Migrate oojs/ui to just run 'npm jenkins' https://gerrit.wikimedia.org/r/345203 /  [[phab:T155483|T155483]]
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 16:05 halfak: deployed ores:18beebf ([[phab:T160638|T160638]])
* 20:03 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/768843
* 13:22 gehel: restarting elasticsearch on deployment-elastic05 to reload log4j configuration
* 15:12 hashar: updating Quibble jenkins jobs
* 10:28 hashar: Jenkins: installing Android Lint plugin 2.4 - [[phab:T161305|T161305]]
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 07:42 hashar: nodepool cleared a couple alien instances
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2017-03-27 ==
== 2022-03-09 ==
* 17:02 ebernhardson: cherry pick https://gerrit.wikimedia.org/r/344964 to puppetmaster to test upgrade to logstash 5.x
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 11:10 hashar: Image snapshot-ci-jessie-1490612363 in wmflabs-eqiad is ready
* 10:59 hashar: Updating Nodepool Jessie image to include PhantomJS (take two) - [[phab:T137112|T137112]]
* 10:58 hashar: Image snapshot-ci-jessie-1490611594 in wmflabs-eqiad is ready
* 10:47 hashar: Updating Nodepool Jessie image to include PhantomJS - [[phab:T137112|T137112]]
* 10:20 hashar: Restarting Jenkins to drop the Throttle Concurrent Builds plugin - [[phab:T158596|T158596]]


== 2017-03-25 ==
== 2022-03-08 ==
* 10:46 Amir1: deleting deployment-ores-redis ([[phab:T160762|T160762]])
* 20:31 brennen: requiring 2fa for all users under /repos
* 10:39 Amir1: changing ores redis address to deployment-ores-redis-01 ([[phab:T160762|T160762]])
* 10:02 Amir1: deleted deployment-ores-redis-02


== 2017-03-24 ==
== 2022-03-07 ==
* 21:34 Amir1: launching deployment-ores-redis-02 ([[phab:T160762|T160762]])
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2017-03-23 ==
== 2022-03-04 ==
* 16:07 mobrovac: restbase deploying {{Gerrit|752ca4b7}}
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 15:52 hashar: Deleting integration-slave-trusty-1011 m1.large. One less perm slave to take care about
* 19:13 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/768068
* 14:02 hashar: deployment-ms-be01 and deployment-ms-be02 : Lower Swift replicator on, upgrade package, reboot hosts. [[phab:T160990|T160990]]


== 2017-03-22 ==
== 2022-03-03 ==
* 09:45 hashar: beta: purging all Linux kernel from Swift instances
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 08:48 hashar: deployment-ms-be01: swift-init reload all - [[phab:T160990|T160990]]
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 08:45 hashar: deployment-ms-be01: swift-init reload container - [[phab:T160990|T160990]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 08:43 hashar: deployment-ms-be01: swift-init reload object - [[phab:T160990|T160990]]
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2017-03-21 ==
== 2022-03-02 ==
* 16:47 halfak: halfak@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 16:07 Amir1: ladsgroup@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 11:27 hashar: integration: purging old packages on permanent slaves, mostly old kernels:  apt-get autoremove --purge
* 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
* 09:06 hashar: CI deploying config hack "High priority test pipeline"  :  https://gerrit.wikimedia.org/r/343318  - [[phab:T160667|T160667]]
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04


== 2017-03-20 ==
== 2022-02-28 ==
* 20:51 andrewbogott: migrating deployment-urldownloader to labvirt1013
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 20:45 andrewbogott: migrating deployment-pdf01 to labvirt1011
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 20:14 andrewbogott: migrating deployment-puppetmaster02 to a different labvirt
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin
* 20:09 bearND: Update mobileapps to {{Gerrit|c0ab01d}}
* 08:51 hashar: Jenkins: depooling / deleting Precise instances.


== 2017-03-17 ==
== 2022-02-26 ==
* 14:08 hashar: salt -v '*precise*' cmd.run 'puppet agent --disable "Pending shutdown on March 20th - [[phab:T158652|T158652]]"'
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases


== 2017-03-16 ==
== 2022-02-24 ==
* 21:48 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/343113/
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 15:03 hashar: deployment-prep setting role::logging::mediawiki::udp2log::rotate: 15  in project wide hiera configuration
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2017-03-15 ==
== 2022-02-23 ==
* 20:29 bearND: Update mobileapps to {{Gerrit|bb8fcf2}}
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 19:02 niedzielski: Reloading Zuul to deploy {{Gerrit|f1c9073c6bda0bc83329c5178cb53613479ba200}}
* 20:10 dancy: Updating scap in beta
* 15:55 Reedy: Removed hhvm statcache cherrypick from beta puppetmaster
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 11:09 elukey: Restore prod version of memcached on deployment-memc04 after experiment (I installed a new version a while ago)
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:22 elukey: created instances deployment-aqs0[23] to have better testing for the AQS beta environment
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # [[phab:T252071|T252071]]
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=hewiktionary
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=dewiktionary
* 09:08 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary
* 08:56 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary // (ParameterTypeException, [[phab:T160503|T160503]])
* 08:50 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (3 sites added)
* 08:49 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https --load-from=https://deployment.wikimedia.beta.wmflabs.org/w/api.php
* 08:49 addshore: addshore@deployment-tin mwscript sql.php --wiki=enwiktionary "TRUNCATE sites; TRUNCATE site_identifiers;"
* 08:44 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=dewiktionary --site-group=wiktionary // (0 sites added)
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (1 site added)


== 2017-03-14 ==
== 2022-02-22 ==
* 19:22 thcipriani: removed alien nodepool instance via: openstack server delete ci-jessie-wikimedia-566503
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 10:15 hashar: Added Niedzielski to integration.
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])
* 09:54 hashar: Jenkins: dropping Sniedzielski  more specific permissions. Account is already in wmf ldap group


== 2017-03-13 ==
== 2022-02-21 ==
* 13:19 hashar: Depooled Precise instances from Jenkins [[phab:T158652|T158652]] leaving the instances up for now.
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 11:38 hashar: Deleting php53lint jobs. Replacing them with php55 equivalents
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 09:39 hashar: upgrading puppet on deployment-pdf01
* 07:27 hashar: Refreshing all Jenkins jobs
* 09:30 hashar: Removing old kernel packages from deployment-pdf01 to free up disk space
* 08:55 hashar: Deleting deployment-copper  Fails puppet due to broken OpenStack metadata  http://169.254.169.254/openstack/2015-10-15/meta_data.json  (fails)  and  no more needed (per elukey )


== 2017-03-10 ==
== 2022-02-20 ==
* 12:29 kart_: Beta: [[phab:T159800|T159800]]: Update DB index for [[phab:T146450|T146450]]
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo


== 2017-03-09 ==
== 2022-02-19 ==
* 16:20 gehel: upgrading elasticsearch on deployment-prep to v5.1.2
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 09:39 hashar: deployment-prep: rebasing puppet master. Got stall due to a submodule update apparently
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]


== 2017-03-08 ==
== 2022-02-18 ==
* 22:45 Reedy: https://gerrit.wikimedia.org/r/#/c/341916/ cherry picked onto deployment-puppetmaster02
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}


== 2017-03-07 ==
== 2022-02-17 ==
* 22:39 hashar: upgrading jenkins02.ci-staging to jenkins 2.x
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 15:26 hashar: ci-staging, enabling puppet master auto signing ( puppetmaster::autosigner: true )
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 08:25 hashar: Image snapshot-ci-jessie-1488874660 in wmflabs-eqiad is ready (Chromium 55->56 among others) - [[phab:T153038|T153038]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 08:16 hashar: Pushing new Jessie image: image-jessie-20170306T224719Z.qcow2
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 11:12 hashar: Bringing deployment-deploy03 back
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome


== 2017-03-06 ==
== 2022-02-16 ==
* 19:03 addshore: mwscript sql.php --wiki=aawiki "CREATE DATABASE cognate_wiktionary"
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 16:03 hashar: Jenkins upgrading  "Git client plugin" 1.19.6 to 2.3.0
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}


== 2017-03-02 ==
== 2022-02-15 ==
* 20:47 hashar: deployment-prep: restarted apache/puppet master. Maybe that will fix ssh_known_hosts being emptied from time to time [[phab:T159332|T159332]]
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 19:32 thcipriani: snapshot-ci-jessie updated for nodepool
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 19:15 thcipriani: running: nodepool image-update wmflabs-eqiad snapshot-ci-jessie to manually update the ci-jessie snapshot for nodepool
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 18:26 godog: integration update composer on '*slave*'
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 11:52 hashar: gerrit: killed a stalled connection: {{Gerrit|dd511e52}} Feb-27 07:11  git-receive-pack '/mediawiki/services/zotero/translators'
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 09:53 hashar: Image snapshot-ci-jessie-1488447340 in wmflabs-eqiad is ready
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}
* 09:29 hashar: Image snapshot-ci-trusty-1488446586 in wmflabs-eqiad is ready
* 09:18 hashar: upgrading composer on permanent slaves for [[phab:T125343|T125343]] : salt -v '*slave*' cmd.run 'cd /srv/deployment/integration/composer && git pull'
* 09:16 hashar: upgrade composer to 1.1.0 https://gerrit.wikimedia.org/r/#/c/339645/
* 08:40 elukey: upgrading apache2 on deployment-mediawiki* - latest debian DSA, introduces https://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions (risk of HTTP 400 responses regression, contact elukey or moritzm if you see any issue)


== 2017-03-01 ==
== 2022-02-14 ==
* 19:09 addshore: "mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki he wiktionary hewiktionary he.wiktionary.beta.wmflabs.org" [[phab:T158628|T158628]]
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 17:11 hashar: cleaned out Jenkins security matrix to drop users that are no more used/inexistent -- [[phab:T69027|T69027]]
* 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 14:13 hashar: deployment-prep : on deployment-tin removed empty dir /etc/ssh/userkeys/root.d . Causes puppet noise
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 12:21 hashar: deployment-prep cleaning out git repos on deployment-tin
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 10:00 legoktm: deployed https://gerrit.wikimedia.org/r/340280 to slaves
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon
* 04:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/340465
* 01:03 Reedy: beta-scap-eqiad giving Host key verification failed


== 2017-02-28 ==
== 2022-02-12 ==
* 19:43 thcipriani: deployment-puppetmaster02 puppetmaster running again, apache2 was refusing to start with: Invalid command 'SSLOpenSSLConfCmd' -- installed apache from wmf repo instead of debian fixed it
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 08:36 hashar: nodepool deleted alien instances 541585 541586 and 541587


== 2017-02-27 ==
== 2022-02-10 ==
* 21:36 bearND: Update mobileapps to {{Gerrit|c924126}}
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602


== 2017-02-25 ==
== 2022-02-09 ==
* 03:50 MaxSem: deployment-prep Deleted January logs from deployment-fluorine02, was running out of space
* 15:22 taavi: deleted shutoff deployment-mx02


== 2017-02-24 ==
== 2022-02-08 ==
* 13:56 hashar: Log refresh Nodepool instances to deploy slave script update to be able to merge mediawiki/composer.json into vendor/composer.json  6527f49..a7728a5  https://gerrit.wikimedia.org/r/#/c/339202/ [[phab:T158674|T158674]]
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 13:52 hashar: deployed slave script update to be able to merge mediawiki/composer.json into vendor/composer.json  6527f49..a7728a5  https://gerrit.wikimedia.org/r/#/c/339202/ [[phab:T158674|T158674]]
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03


== 2017-02-23 ==
== 2022-02-07 ==
* 18:35 greg-g: 18:29 <  chasemp> !log labnodepool1001:~# service nodepool restart
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 09:27 hashar: Clearing skins from testextension jobs [[phab:T117710|T117710]] salt -v '*slave*' cmd.run 'rm -fR /srv/jenkins-workspace/workspace/mwext-testextension*/src/skins/*'
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550


== 2017-02-22 ==
== 2022-02-04 ==
* 20:58 hashar: Deleted jenkins job pplint-HEAD. Fully replaced by rake / puppet-syntax gem - [[phab:T154894|T154894]]
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
* 20:54 hashar: Deleted jenkins job erblint-HEAD. Fully replaced by rake / puppet-syntax gem - [[phab:T154894|T154894]]


== 2017-02-20 ==
== 2022-02-03 ==
* 14:53 hashar: integration: applying role::ci::slave::saucelabs to saucelabs-01
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 12:50 hashar: integration-slave-jessie-1001 downgraded cowbuilder to 0.73 from jessie to match integration-slave-jessie-1002
* 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]


== 2017-02-17 ==
== 2022-02-02 ==
* 14:07 hashar: integration: deleting "repository" instance. No time to figure out how to ship Sonatype Nexus to it. [[phab:T147635|T147635]]
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta


== 2017-02-16 ==
== 2022-02-01 ==
* 18:34 greg-g: chase restarted nodepool, the daemon crashed
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 18:32 greg-g: no active nodepool instances listed in Jenkin's view: https://integration.wikimedia.org/ci/ but zuul has plenty to do https://integration.wikimedia.org/zuul/
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 16:56 hashar: integration: provisioned browsertests-1001 with role::ci::slaves::browsertests . Added it to Jenkins with label  BrowserTests
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]
* 16:33 halfak: deploying ores:e9bbda3
* 16:30 hashar: integration: created browsertests-1001  intended to run the daily browser tests later on


== 2017-02-15 ==
== 2022-01-31 ==
* 15:47 hashar: Zuul reducing gate-and-submit minimum amount of changes to process from the wrong 12 down to 2.  In case of repeating failures it would  end up running jobs for only two jobs which would prevent cancelling jobs for up to 11 changes!
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]


== 2017-02-14 ==
== 2022-01-28 ==
* 14:38 hashar: Updating castor-save publish job to properly capture composer cache on Jessie ( it is in ~/.composer/cache for some reason)  [[phab:T156359|T156359]]
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`


== 2017-02-13 ==
== 2022-01-27 ==
* 21:25 bearND: Update mobileapps to {{Gerrit|3af473f}}
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2  # [[phab:T299431|T299431]]
* 20:15 hashar: Image snapshot-ci-jessie-1487016035 in wmflabs-eqiad is ready
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 20:01 hashar: Updating Nodepool Jessie snapshot to update the Parsoid zuul-cloner map ( https://gerrit.wikimedia.org/r/#/c/337430/ )
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:25 hashar: Changing Jenkins slave contint1001 working dir  from /srv/ssd/jenkins-slave to /srv/jenkins-slave ( https://gerrit.wikimedia.org/r/#/c/337286/ )
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins


== 2017-02-10 ==
== 2022-01-26 ==
* 22:11 halfak: deployed ores:a15ec90
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 21:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/337079
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 16:18 thcipriani: deployment-puppetmaster02:/var/lib/git/operations/puppet removed untracked file "how", updated submodules
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 14:49 hashar: rebase beta puppet master. Fixed conflicts with https://gerrit.wikimedia.org/r/#/c/321096/ and https://gerrit.wikimedia.org/r/#/c/312523/
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 11:36 hashar: Pruning some old caches from castor.integration.eqiad.wmflabs  (eg node-4  jobs are gone)
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2017-02-09 ==
== 2022-01-25 ==
* 22:22 greg-g: manually kicked off a bunch of selenium tests after tgr and Reedy fixed [[phab:T157636|T157636]]
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 18:36 Reedy: someone should kill the ee_prototypewiki db from beta
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 16:40 Amir1: deploying {{Gerrit|030c269}} ores to sca03
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 11:11 legoktm[NE]: deploying https://gerrit.wikimedia.org/r/336615 and https://gerrit.wikimedia.org/r/336779
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2017-02-08 ==
== 2022-01-24 ==
* 22:26 mdholloway: mobileapps deployed {{Gerrit|0efa7b8}} in the beta cluster
* 21:44 Reedy: unstick beta ci jobs
* 14:14 hashar: integration-slave-jessie-1001 upgrading cowbuilder
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 09:20 hashar: deployment-fluorine02  upgraded packages, deleted old files from /srv/mw-log/archive
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2017-02-07 ==
== 2022-01-22 ==
* 17:49 halfak: deploying ores {{Gerrit|7c80636}}
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 09:02 hashar: Hard rebooting integration-slave-jessie-1001 .  I messed up with the DHCP client :(
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts


== 2017-02-06 ==
== 2022-01-21 ==
* 21:31 bearND: Update mobileapps to {{Gerrit|034a391}}
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`


== 2017-02-04 ==
== 2022-01-20 ==
* 21:37 halfak: deploying ores {{Gerrit|7c80636}}
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 21:24 halfak: deploying ores {{Gerrit|691b340}}
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2017-02-03 ==
== 2022-01-19 ==
* 11:09 hashar: beta: removed old kernels from deployment-redis02  to free up disk space
* 18:20 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 10:42 hashar: Image ci-jessie-wikimedia-1486115643 in wmflabs-eqiad is ready [[phab:T156923|T156923]]
* 17:31 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 10:12 hashar: Image ci-jessie-wikimedia-1486115643 in wmflabs-eqiad is ready  [[phab:T156923|T156923]]
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]
* 09:54 hashar: Regenerate Nodepool Jessie snapshot.  Would get a new HHVM version [[phab:T156923|T156923]]


== 2017-02-02 ==
== 2022-01-18 ==
* 21:56 hashar: integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-amd64.cow  it was not properly provisioned causing build to fail (eg lack of /etc/hosts) Running puppet to reprocvision it (poke [[phab:T156651|T156651]])
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 16:26 Amir1: deploying {{Gerrit|9fd75a1}} ores in beta
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 16:17 hashar: integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-i386.cow/ it was not properly provisioned causing build to fail (eg lack of /etc/hosts)  Running puppet to reprocvision it (poke [[phab:T156651|T156651]])
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # [[phab:T299389|T299389]]
* 14:15 hashar: Nodepool: delete the image building of Jessie (image id 1322) to prevent a faulty HHVM version from being added. [[phab:T156923|T156923]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 00:52 tgr: added mhurd as member
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2017-02-01 ==
== 2022-01-17 ==
* 21:43 bearND: Update mobileapps to {{Gerrit|e48a88c}}
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 18:51 thcipriani: nodepool delete-image 1320 per [[phab:T156923|T156923]]
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 14:53 gehel: deployment-elastic* fully migrated to Jessie and /srv as data partition - [[phab:T151326|T151326]]
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
* 14:52 gehel: killing test node deployment-elastic08 - [[phab:T151326|T151326]]
* 14:32 gehel: shutting down and reimaging deployment-elastic07 - [[phab:T151326|T151326]]
* 14:06 gehel: shutting down and reimaging deployment-elastic06 - [[phab:T151326|T151326]]
* 13:34 gehel: shutting down and reimaging deployment-elastic05 - [[phab:T151326|T151326]]
* 13:29 gehel: starting deployment-elastic* migration to jessie and moving data partition to /srv ([[phab:T151326|T151326]] / [[phab:T151328|T151328]])
* 13:18 moritzm: upgraded deployment-prep to hhvm 3.12.12


== 2017-01-31 ==
== 2022-01-14 ==
* 22:12 thcipriani: started mysql on all integration precise instances via salt -- was stopped for some reason
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 01:59 bd808: nodepool is full of instance stuck in "delete"
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 01:53 bd808: https://integration.wikimedia.org/zuul/ showing huge backlogs but https://integration.wikimedia.org/ci/ looks mostly idle
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta


== 2017-01-26 ==
== 2022-01-13 ==
* 14:25 hashar: Created Github repo for Gerrit replication https://github.com/wikimedia/mediawiki-libs-phpstorm-stubs  [[phab:T153252|T153252]]
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 13:49 hashar: Gerrit creating  mediawiki/libs/phpstorm-stubs to fork https://github.com/JetBrains/phpstorm-stubs for [[phab:T153252|T153252]]
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2017-01-24 ==
== 2022-01-12 ==
* 11:04 hashar: Deleting integration-publisher (Precise) replaced by integration-publishing (Jessie). [[phab:T156064|T156064]] [[phab:T143349|T143349]]
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/


== 2017-01-23 ==
== 2022-01-11 ==
* 23:41 bearND: Update mobileapps to {{Gerrit|66ef3c2}}
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 21:05 hashar: Created integration-publishing Jessie instance 10.68.23.254 with puppet class role::ci::publisher::labs .  Meant to replace Precise instance integration-publisher [[phab:T156064|T156064]]
* 12:45 hashar: Image ci-jessie-wikimedia-1485174573 in wmflabs-eqiad is ready  {{!}} should no more spawn varnish on boot
* 09:02 hashar: Archiving Gerrit project wikidata/gremlin marking it read-only [[phab:T155829|T155829]]
* 07:15 _joe_: cherry-picking the move of base to profile::base


== 2017-01-21 ==
== 2022-01-10 ==
* 21:20 hashar: integration: updating slave scripts for https://gerrit.wikimedia.org/r/#/c/333389/
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 21:08 bd808: Puppet failures on deployment-restbase0[12] seem to be some sort of hang of the Puppet process itself. Run prints "Finished catalog run in 2n.nn seconds" but Puppet doesn't terminate for about a minute longer. The only state change logged is cassandra-metrics-collector service start.
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]


== 2017-01-20 ==
== 2022-01-05 ==
* 10:14 hashar: puppet fails on "integration" labs instances due to an attempt to unmount the non existing NFS /home.  Filled [[phab:T155820|T155820]]
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 09:18 hashar: beta: reset workspace of /srv/mediawiki-staging/php-master/extensions/reCaptcha  it had a .gitignore local hack for some reason
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 09:05 hashar: integration restarted mysql on trusty permanent slaves [[phab:T141450|T141450]] [[phab:T155815|T155815]]  salt -v '*trusty*' cmd.run 'service mysql start'
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2017-01-19 ==
== 2022-01-04 ==
* 22:11 Krenair: added bunch of others to the same group per request. we should figure out how to make this process sane somehow
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 22:06 Krenair: added nuria to deploy-service group on deployment-tin
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 16:56 hashar: rebased puppet master on integration and deployment-prep Trivial conflict between https://gerrit.wikimedia.org/r/#/c/312523/ and a lint change
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383
* 09:36 hashar: Nuking workspaces of all mwext-testextension-hhvm-composer* jobs. Lame attempt for [[phab:T155600|T155600]]. salt -v '*slave*' cmd.run 'rm -fR /srv/jenkins-workspace/workspace/mwext-testextension-hhvm-composer*'


== 2017-01-18 ==
== 2022-01-03 ==
* 10:49 hashar: Disconnected/connected Jenkins Gearman clientThe beta cluster builds had a deadlock.
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001Restarted CI Jenkins.
* 10:39 hashar: Image ci-jessie-wikimedia-1484735445 in wmflabs-eqiad is ready (add python-conftool to hopefully have puppet rspec pass on https://gerrit.wikimedia.org/r/#/c/332475/ )
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


== 2017-01-17 ==
* 21:47 urandom: deployment-prep restarting Cassandra on deployment-restbase02
* 21:46 urandom: deployment-prep restarting Cassandra on deployment-restbase01
* 19:02 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/332534/
* 18:25 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/332521/
* 18:07 urandom: deployment-prep restarting Cassandra on deployment-restbase01
* 17:50 urandom: re-enabling puppet on deployment-restbase02
* 17:47 urandom: re-enabling puppet on deployment-restbase01
* 10:32 hashar: Refreshing all jobs in Jenkins 'jenkins-jobs --conf jenkins_jobs.ini update config/jjb'


== 2017-01-16 ==
{{SAL-archives/Release Engineering}}
* 09:33 hashar: integration  nuked the Zuul merger path for SelectTag mw extension ( on scandium /srv/ssd/zuul/git/mediawiki/extensions/SelectTag )  Failed to merge https://gerrit.wikimedia.org/r/#/c/331974/


== 2017-01-12 ==
* 00:33 legoktm: deploying https://gerrit.wikimedia.org/r/331796 and https://gerrit.wikimedia.org/r/331795
== 2017-01-11 ==
* 18:07 urandom: restarting restbase cassandra nodes
* 18:01 urandom: disabling puppet on restbase cassandra nodes to experiment with prometheus exporter
== 2017-01-10 ==
* 23:07 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/331099
== 2017-01-08 ==
* 05:20 Krenair: deployment-stream: live hacked /usr/lib/python2.7/dist-packages/socketio/handler.py a bit (added apostrophes) to try to make rcstream work
== 2017-01-07 ==
* 10:17 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki ([[phab:T139110|T139110]])
== 2017-01-06 ==
* 16:31 hashar: Nodepool Image ci-jessie-wikimedia-1483719758 in wmflabs-eqiad is ready
* 16:24 hashar: Nodepool Image ci-trusty-wikimedia-1483719370 in wmflabs-eqiad is ready
* 04:56 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/330843
== 2017-01-05 ==
* 17:20 hashar: Dropping puppet source from https://doc.wikimedia.org/puppetsource/ . contint1001: sudo rm -fR /srv/org/wikimedia/doc/puppetsource  ([[phab:T143233|T143233]])
== 2017-01-04 ==
* 21:29 mutante: deployment-cache-text-04 - running acme-setup command to debug  .. Creating CSR /etc/acme/csr/beta_wmflabs_org.pem
* 21:26 Krenair: trying to troubleshoot puppet by stopping nginx then letting puppet start it
* 21:05 mutante: deployment-cache-text04 stopping nginx service, running puppet to debug dependency issue
* 09:41 hashar: integration: pruning /srv/pbuilder/aptcache/  on Jessie perm slaves
== 2017-01-02 ==
* 11:22 hashar: Nodepool Image ci-jessie-wikimedia-1483355768 in wmflabs-eqiad is ready
* 11:17 hashar: Jessie images have the wrong python-pbr version ( [[phab:T153877|T153877]] ) causing zuul-cloner to fail. Refreshing image
* 10:02 hashar: Nodepool Image ci-jessie-wikimedia-1483350885 in wmflabs-eqiad is ready
* 09:57 hashar: Nodepool Image ci-trusty-wikimedia-1483350368 in wmflabs-eqiad is ready
== 2016-12-27 ==
* 05:00 Amir1: deploying {{Gerrit|5230e7d}} in ores beta node ([[phab:T154168|T154168]])
== 2016-12-26 ==
* 12:09 hashar: beta: restarted varnish.service and varnish-frontend.service on deployment-cache-text04
== 2016-12-24 ==
* 09:02 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/329038
== 2016-12-23 ==
* 12:18 legoktm: deploying https://gerrit.wikimedia.org/r/328886
== 2016-12-22 ==
* 22:11 thcipriani: disable production l10nupdate for deployment freeze
== 2016-12-21 ==
* 05:57 Krinkle: Jenkins "Collapsing Console Sections" for PHPUnit was broken since "-d zend.enable_gc=0" was added to phpunit.php invocation. Updated pattern in Jenkins system configuration.
== 2016-12-19 ==
* 21:21 andrewbogott: and also python-functools32_3.2.3.2-3~bpo8+1_all.deb
* 21:20 andrewbogott: upgrading to python-jsonschema_2.5.1-5~bpo8+1_all.deb on deployment-eventlogging03
* 20:51 andrewbogott: upgrading to python-requests_2.12.3-1_all.deb ./python-urllib3_1.19.1-1_all.deb on deployment-mediawiki04 and deployment-tin
* 09:35 legoktm: deploying https://gerrit.wikimedia.org/r/328145
* 08:00 legoktm: deploying https://gerrit.wikimedia.org/r/288819 https://gerrit.wikimedia.org/r/276065 https://gerrit.wikimedia.org/r/328136
* 02:25 legoktm: deploying https://gerrit.wikimedia.org/r/327692
== 2016-12-16 ==
* 22:34 legoktm: deploying https://gerrit.wikimedia.org/r/327202
* 14:33 hashar: Nodepool Image ci-jessie-wikimedia-1481897950 in wmflabs-eqiad is ready
* 14:25 hashar: Nodepool Image ci-trusty-wikimedia-1481897961 in wmflabs-eqiad is ready
* 14:19 hashar: Refreshing Nodepool images. The snapshots were broken due to mariadb-client failing to upgrade
* 13:45 hashar: integration / contintcloud : remove security rules of labs projects that allowed gallium (phased out) [[phab:T95757|T95757]]
* 13:44 hashar: integration / contintcloud : update security rules of labs projects to allow contint2001
* 13:15 hashar: integration: update sudo policy for debian-glue to keep the env variable SHELL_ON_FAILURE (for https://gerrit.wikimedia.org/r/#/c/327720/ )
* 10:15 hashar: integration: apt-get upgrade on all permanent slaves
* 10:13 hashar: integration-slave-docker-1000  changed docker::version from no more existent '1.12.3-0~jessie' to simply 'present'. Will have to manually upgrade it from now on.  [[phab:T153419|T153419]]
* 10:04 hashar: deployment-puppetmaster02  updated puppet repo. Was stall due to a bump of the mariadb submodule
== 2016-12-15 ==
* 21:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/324368
* 19:23 marxarelli: Manually rebasing and re-applying cherry picks for operations/puppet on integration-puppetmaster01.eqiad.wmflabs
* 16:08 hashar: deployment-phab02 : apt-get upgrade  [[phab:T147818|T147818]]
* 14:48 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki ([[phab:T139110|T139110]])
* 11:41 zeljkof: Reloading Zuul to deploy 327473
== 2016-12-14 ==
* 12:38 elukey: created deployment-copper on deployment-prep as temporary test
== 2016-12-13 ==
* 22:52 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/327119
* 21:15 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/327048
* 09:42 hashar: Updating MediaWiki Jenkins jobs to support injecting skin dependencies [[phab:T151593|T151593]]
* 02:17 legoktm: deploying https://gerrit.wikimedia.org/r/326880
* 02:10 legoktm: deploying https://gerrit.wikimedia.org/r/326877
== 2016-12-09 ==
* 04:01 legoktm: deploying https://gerrit.wikimedia.org/r/326070
* 03:45 legoktm: deploying https://gerrit.wikimedia.org/r/326069
== 2016-12-08 ==
* 23:35 legoktm: deploying https://gerrit.wikimedia.org/r/326048 https://gerrit.wikimedia.org/r/326050
* 22:32 legoktm: deploying https://gerrit.wikimedia.org/r/325930
* 21:14 legoktm: deploying https://gerrit.wikimedia.org/r/326032
* 21:08 legoktm: deploying https://gerrit.wikimedia.org/r/326020
* 20:27 legoktm: deploying https://gerrit.wikimedia.org/r/325974
* 20:19 legoktm: deploying https://gerrit.wikimedia.org/r/326016
* 20:11 legoktm: deploying https://gerrit.wikimedia.org/r/326015
* 19:51 legoktm: deploying https://gerrit.wikimedia.org/r/326009
* 19:44 legoktm: deploying https://gerrit.wikimedia.org/r/325912 https://gerrit.wikimedia.org/r/326006
* 15:33 hashar: Image ci-jessie-wikimedia-1481210905 in wmflabs-eqiad is ready : Notice: /Stage[main]/Main/Package[netcat-openbsd]/ensure: ensure changed 'purged' to 'present'
* 15:28 hashar: Updating Nodepool Jessie image to ship `netcat`  [[phab:T151469|T151469]] [[phab:T152684|T152684]]
* 10:31 hashar: Image ci-trusty-wikimedia-1481192772 in wmflabs-eqiad is ready
* 10:21 hashar: Refreshing Nodepool base image for Trusty. Was blocked on a mariadb upgrade, should also acquire network faster [[phab:T113342|T113342]]
* 09:45 legoktm: deploying https://gerrit.wikimedia.org/r/325903
* 08:48 hashar: Image ci-jessie-wikimedia-1481186016 in wmflabs-eqiad is ready  [[phab:T113342|T113342]]
* 05:31 legoktm: legoktm@integration-saltmaster:~$ sudo salt '*jessie*' cmd.run 'puppet agent -tv'
* 05:26 legoktm: cherry-picked https://gerrit.wikimedia.org/r/#/c/325877/ onto integration-puppetmaster01
* 03:26 legoktm: deploying https://gerrit.wikimedia.org/r/325873
== 2016-12-07 ==
* 15:04 hashar: Image ci-trusty-wikimedia-1481122712 in wmflabs-eqiad is ready  [[phab:T117418|T117418]]
* 02:29 matt_flaschen: foreachwikiindblist FlowFixInconsistentBoards complete
* 02:27 matt_flaschen: Started (foreachwikiindblist flow.dblist extensions/Flow/maintenance/FlowFixInconsistentBoards.php) 2>&1 {{!}} tee FlowFixInconsistentBoards_2016-12-06.txt on deployment-tin
== 2016-12-06 ==
* 21:20 hashar: Image ci-jessie-wikimedia-1481058839 in wmflabs-eqiad is ready [[phab:T113342|T113342]]
* 21:13 hashar: Refresh Nodepool Jessie snapshot which boot 3 times faster. Will help get nodes available faster [[phab:T113342|T113342]]
* 16:33 hashar: Nodepool imported a new Jessie image 'jessie-[[phab:T113342|T113342]]' with some network configuration hotfix. Will use for debugging. [[phab:T113342|T113342]]
* 09:08 Reedy: running foreachwiki update.php on beta
== 2016-12-05 ==
* 20:43 hashar: Image ci-jessie-wikimedia-1480969940 in wmflabs-eqiad is ready (include trendingedits::packages  which explicitly define the installation of librdkafka-dev' )
* 09:52 elukey: add https://gerrit.wikimedia.org/r/#/c/324642/ to the deployment-prep's puppet master to test nutcracker
* 09:39 hashar: beta-update-databases-eqiad fails due to CONTENT_MODEL_FLOW_BOARD not registered on the wiki. [[phab:T152379|T152379]]
* 08:44 hashar: Image ci-jessie-wikimedia-1480926961 in wmflabs-eqiad is ready  [[phab:T113342|T113342]]
* 08:35 hashar: Pushing new Jessie image to Nodepool that is supposedly boot 3x times faster [[phab:T113342|T113342]]
== 2016-12-04 ==
* 15:25 Krenair: Found a git-sync-upstream cron on deployment-mx for some reason... commented for now, but wtf was this doing on a MX server?
== 2016-12-03 ==
* 23:07 legoktm: deploying https://gerrit.wikimedia.org/r/325132
* 10:48 legoktm: deploying https://gerrit.wikimedia.org/r/325093 and https://gerrit.wikimedia.org/r/325094
== 2016-12-02 ==
* 14:40 hashar: added Tobias Gritschacher to Gerrit "integration" group so he can +2 patches on integration/* repositories \O/
== 2016-12-01 ==
* 18:20 elukey: removing https://gerrit.wikimedia.org/r/#/c/305536 from the puppet master via rebase -i (no-op for beta)
* 18:11 elukey: adding https://gerrit.wikimedia.org/r/#/c/305536/3 to the puppet master
* 14:16 hashar: Image ci-jessie-wikimedia-1480601060 in wmflabs-eqiad is ready  {{!}} [[phab:T152096|T152096]]
== 2016-11-30 ==
* 17:22 gehel: restart of logstash on deployment-logstash2 - upgrade to Java 8 - [[phab:T151325|T151325]]
* 17:11 gehel: rolling restart of deployment-elastic0* - upgrade to Java 8 - [[phab:T151325|T151325]]
* 11:22 hashar: Gerrit hide mediawiki/extensions/JsonData/JsonSchema Empty since 2013
* 11:20 hashar: Gerrit made mediawiki/extensions/GuidedTour/guiders read-only (per README.md, no more used)
* 11:18 hashar: Gerrit  mediawiki/extensions/CentralNotice/BannerProxy.git  Empty since 2014
== 2016-11-29 ==
* 15:23 hashar: Image ci-jessie-wikimedia-1480432368 in wmflabs-eqiad is ready
* 14:30 hashar: Image ci-trusty-wikimedia-1480429423 in wmflabs-eqiad is ready  [[phab:T151879|T151879]]
* 14:24 hashar: Refreshing Nodepool Trusty snapshot to get php5-xsl installed [[phab:T151879|T151879]]
== 2016-11-28 ==
* 09:48 hashar: Image ci-trusty-wikimedia-1480326016 in wmflabs-eqiad is ready
* 09:39 hashar: Regenerated Nodepool image for Trusty. It no more includes apache::mod::php5 which broke the build and is not needed on Trusty ( https://gerrit.wikimedia.org/r/323803  )
* 09:15 elukey: cherry-pick of https://gerrit.wikimedia.org/r/#/c/323517 to deployment-puppetmaster02 to test
== 2016-11-26 ==
* 16:15 Reedy: killed /srv/jenkins-workspace/workspace/mediawiki-core-*/src and /srv/jenkins-workspace/workspace/mwext-*/src from integration slaves to get rid of borked MW dirs
* 15:51 Reedy: deleted /srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src on integration-slave-trusty-1006 to force a reclone
* 14:14 Reedy: moved old /srv/mediawiki-staging/php-master to /tmp/php-master, recloned MW Core, copied in LocalSettings, skins, vendor and extensions. [[phab:T151676|T151676]]. scap sync-dir running
* 13:05 Reedy: marked deployment-tin as offline due to [[phab:T151670|T151670]]
== 2016-11-24 ==
* 20:49 hashar: make contint1001 Jenkins slave to only builds jobs with a label matching the node  https://integration.wikimedia.org/ci/computer/contint1001/configure  [[phab:T86659|T86659]]
* 15:46 elukey: removing https://gerrit.wikimedia.org/r/#/c/322268/ from the list of cherry picks on puppet master since it is not the right way to go
* 08:58 elukey: rebased puppet operations git repo on  deployment-puppetmaster to refresh https://gerrit.wikimedia.org/r/#/c/322268/
== 2016-11-23 ==
* 15:04 Krenair: fixed puppet on deployment-cache-text04 by manually enabling experimental apt repo, see [[phab:T150660|T150660]]
* 10:57 hashar: Terminating deployment-apertium01 again [[phab:T147210|T147210]]
== 2016-11-22 ==
* 19:31 hashar: beta: rebased puppet master
* 19:30 hashar: beta: dropping cherry pick for the PDF render by mobrovac ( https://gerrit.wikimedia.org/r/#/c/305256/ ). Got merged
* {{SAL entry|1=08:29 hashar: Deleting shut off instances: integration-puppetmaster , deployment-puppetmaster , deployment-pdf02 , deployment-conftool  - T150339}}
== 2016-11-21 ==
* {{SAL entry|1=12:46 hashar: beta: Cherry picked puppet fix for udp2log https://gerrit.wikimedia.org/r/#/c/322639/  T151169}}
== 2016-11-19 ==
* {{SAL entry|1=00:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/322370}}
== 2016-11-18 ==
* {{SAL entry|1=15:42 elukey: cherry picked https://gerrit.wikimedia.org/r/#/c/322268 on puppet master}}
== 2016-11-17 ==
* {{SAL entry|1=22:07 mutante: re-enabled puppet on contint1001 after live Apache fix}}
* {{SAL entry|1=11:34 hasharLunch: Deleted instance deployment-apertium01 . Was Trusty and lacked packages, replaced by a Jessie one ages ago. T147210}}
== 2016-11-16 ==
* {{SAL entry|1=20:53 elukey: restored apache2 config on deployment-mediawiki06}}
* {{SAL entry|1=20:28 elukey: temporary increasing verbosity of mod_rewrite on deployment-mediawiki06 as test}}
* {{SAL entry|1=20:02 Krenair: mysql master back up, root identity is now unix socket based rather than password}}
* {{SAL entry|1=19:57 Krenair: taking mysql master down to fix perms}}
* {{SAL entry|1=13:02 hashar: Restarted HHVM on deployment-mediawiki05 was not honoring requests T150849}}
* {{SAL entry|1=12:24 hashar: beta: created dewiktionary table on the Database slave. Restarted replication with START SLAVE;    T150834  T150764}}
* {{SAL entry|1=10:39 hashar: Removing revert b47ce21cec3a4340dd37c773210a514350f10297 from deployment-tin and reenabling jenkins job.  https://gerrit.wikimedia.org/r/321857 will get it fixed}}
* {{SAL entry|1=10:26 hashar: Reverting mediawiki/core b47ce21cec3a4340dd37c773210a514350f10297 on beta cluster T150833}}
* {{SAL entry|1=09:51 hashar: marking deployment-tin offline so I can live hack mediawiki code / scap for T150833 and T15034}}
* {{SAL entry|1=09:12 hashar: deployment-mediawiki04 stopping hhvm}}
* {{SAL entry|1=09:12 hashar: deployment-mediawiki04 stopping hhv}}
* {{SAL entry|1=08:59 hashar: beta database update broken with: MediaWiki 1.29.0-alpha Updater\n\nYour composer.lock file is up to date with current dependencies!}}
* {{SAL entry|1=07:52 Krenair: the new mysql root password for -db04 is at /tmp/newmysqlpass as well as in a new file in the puppetmaster's labs/private.git}}
* {{SAL entry|1=06:34 twentyafterfour: restarting hhvm on deployment-mediawiki04}}
* {{SAL entry|1=06:33 Amir1: ladsgroup@deployment-mediawiki05:~$ sudo service hhvm restart}}
* {{SAL entry|1=06:30 mutante: restarting hhvm on deployment-mediawiki06}}
== 2016-11-15 ==
* {{SAL entry|1=16:03 hasharAway: adding thcipriani to the labs "git" project maintained by paladox}}
== 2016-11-14 ==
* {{SAL entry|1=08:16 Amir1: cherry-picking 321096/3 in beta puppetmaster}}
== 2016-11-12 ==
* 14:02 Amir1: cherry-picked gerrit change 321096/2 in puppetmaster
== 2016-11-11 ==
* 23:48 bd808: Updated _template/logstash on deployment-logstash2 to include change from https://gerrit.wikimedia.org/r/#/c/320441/
* 23:44 bd808: Cherry-picked https://gerrit.wikimedia.org/r/#/c/320441/ for testing on deployment-logstash2
* 21:27 hashar: deployment-tin  deleted /var/lock/scap . Was left over after beta-scap-eqiad job got abruptly aborted
== 2016-11-10 ==
* 09:33 hashar: Image ci-jessie-wikimedia-1478770026 in wmflabs-eqiad is ready
* 09:26 hashar: Regenerate Nodepool base image for Jessie and refreshing snapshot image
== 2016-11-09 ==
* 20:27 Krenair: removed default SSH access from production host 208.80.154.135, the old gallium IP
* 16:34 Reedy: deployment-tin no longer offline, jenkins running jobs now
* 16:11 Reedy: marking deployment-tin.eqiad  as offline to test -labs -> beta config rename
== 2016-11-08 ==
* 10:23 hashar: refreshing all jenkins jobs to clear out potential live hack I made but can't remember on which jobs I did
== 2016-11-07 ==
* 14:01 gilles: Pointing deployment-imagescaler01.eqiad.wmflabs' puppet to puppetmaster.thumbor.eqiad.wmflabs
== 2016-11-04 ==
* 13:20 hashar: gerrit: created mediawiki/extensions/PageViewInfo.git  and renamed user group extension-WikimediaPageViewInfo to extension-PageViewInfo T148775
* 12:57 hashar: Image ci-jessie-wikimedia-1478263647 in wmflabs-eqiad is ready (bring in java for maven projects)
* 12:49 dcausse: deployment-prep reloading nginx on deployment-elastic0[5-7] to fix ssl cert issue
* 09:28 hashar: Delete integration-slave-jessie-1003 , only have a few jobs running on permanent Jessie slaves - T148183
* 09:26 hashar: Delete zuul-dev-jessie.integration.eqiad.wmflabs  was for testing Zuul on Jessie and it works just fine on contint1001 :]  T148183
* 09:25 hashar: Delete integration-slave-trusty-1012 one less permanent slave since some load has been moved to Nodepool  T148183
* 09:24 hashar: Delete integration-slave-trusty-1016 not pooled in Jenkins anymore T148183
== 2016-11-03 ==
* 15:05 Amir1: deploy 0caa589 in ores to deployment-sca03
* 14:52 Amir1: deploying ores 0caa589 in deployment-sca03
* 11:32 hashar: deployment-apertium01 manually cleared puppet.conf
* 11:29 hashar: deployment-apertium01 fails puppet du to wrong certificate bah
* 07:22 Krenair: fiddled with jenkins jobs in mediawiki-core-doxygen-publish to try to get stuff moving in the postmerge queue again
* 05:04 Krenair: beginning to move the rest of beta to the new puppetmaster
* 01:53 mutante: followed instructions at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Gearman_deadlock
* 01:53 mutante: disabling and re-enabling gearman, zuul is not working and could be gearman deadlock
== 2016-11-02 ==
* 22:06 hashar: hello stashbot
* 18:51 Krenair: armed keyholder on -tin and -mira
* 18:50 Krenair: started mysql on -db boxes to bring beta back online
* 10:54 hashar: Image ci-jessie-wikimedia-1478083637 in wmflabs-eqiad is ready
* 10:47 hashar: Force refresh Nodepool snapshot for Jessie  so it get doxygen included T119140
== 2016-11-01 ==
* 22:22 Krenair: started mysql on -db03 to hopefully pull us out of read-only mode
* 22:21 Krenair: started mysql on -db04
* 22:19 Krenair: stopped and started udp2log-mw on -fluorine02
* 22:10 hashar: Armed keyholder on deployment-tin . Instance had 20 minutes uptime and apparently keyholder does not self arm
* 22:00 Krenair: started moving nodes back to the new puppetmaster
* 02:55 Krenair: Managed to mess up the deployment-puppetmaster02 cert, had to move those nodes back
== 2016-10-31 ==
* 20:57 Krenair: moving some nodes to deployment-puppetmaster02
* 16:57 bd808: Added Niharika29 as project member
== 2016-10-27 ==
* 20:51 hashar: reboot integration-puppetmaster01
* 18:50 bd808: stashbot has replaced qa-morebots in this channel as the sole bot handling !log messages
* 18:46 bd808: Testing dual page wiki logging by stashbot. (check #3)
* 18:36 bd808: !log deployment-prep Testing dual page wiki logging by stashbot. (second attempt)
* 18:14 bd808: !log deployment-prep Testing dual page wiki logging by stashbot.
* 10:30 hashar: integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing for T149294: salt -v '*slave-trusty*' cmd.run 'deluser jenkins-deploy kvm'
* 10:29 hashar: integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing:  salt -v '*slave-trusty*' cmd.run 'groupdeluser jenkins-deploy kvm'
* 10:25 hashar: integration: purge Android packages from Trusty slaves for T149294 : salt -v '*slave-trusty*' cmd.run 'apt-get --yes remove --purge gcc-multilib lib32z1 lib32stdc++6 qemu'
== 2016-10-25 ==
* 19:21 hasharAway: Python PyPi mirror has some issue. Impacts all CI jobs relying on tox  https://status.python.org/
* 10:39 elukey: cherry picked https://gerrit.wikimedia.org/r/#/c/314519/ and https://gerrit.wikimedia.org/r/#/c/306943/ to deployment-puppetmaster
== 2016-10-24 ==
* 16:19 andrewbogott: upgrading deployment-puppetmaster to puppet 3.8.5 packages
* 09:14 hashar: rebasing integration puppet master
== 2016-10-21 ==
* 09:42 gehel: decommission of deployment-elastic08 - T147777
== 2016-10-20 ==
* 23:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/317083
* 20:53 legoktm: deploying https://gerrit.wikimedia.org/r/317022
== 2016-10-14 ==
* 21:13 matt_flaschen: Ran START SLAVE to restart replication after columns created directly on replica were deleted.
* 20:53 bd808: Dropped lu_local_id, lu_global_id from replica db which were added improperly
* 20:37 matt_flaschen: Applied CentralAuth's patch-lu_local_id.sql migration for T148111, to sql --write
* 20:09 bd808: Applied CentralAuth's patch-lu_local_id.sql migration for T148111
* 11:30 dcausse: deployment-prep running sudo update-ca-certificates --fresh on deployment-ton to fix curl error code 60 in cirrus maint script (T145609)
== 2016-10-13 ==
* 21:21 hashar: Deleted CI slaves integration-slave-jessie-1004 integration-slave-jessie-1005 integration-slave-trusty-1013 integration-slave-trusty-1014 integration-slave-trusty-1017 integration-slave-trusty-1018
* 20:12 hashar: Switching composer-hhvm / composer-php55 to Nodepool  https://gerrit.wikimedia.org/r/#/c/306727/  T143938
* 16:23 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 16:06 godog: add settings to duplicate traffic to thumbor in beta and restart swift-proxy
* 16:03 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315648/ on deployment-puppetmaster
* 15:35 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:38 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/5 on deployment-puppetmaster
* 14:34 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:32 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/4 on deployment-puppetmaster
* 14:32 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:27 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/ on deployment-puppetmaster
* 14:22 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 13:42 gilles: Cherry picking https://gerrit.wikimedia.org/r/#/c/315248/ on deployment-puppetmaster
== 2016-10-12 ==
* 13:37 elukey: upgraded memcached on deployment-memc04 to 1.4.28-1.1+wmf1 as part of a perf experiment (T129963) - rollback: wipe https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep/host/deployment-memc04, apt-get remove memcached on deployment-memc04, puppet run
== 2016-10-11 ==
* 21:35 hasharAway: Force pushed Zuul patchqueue  5628f95...fc6a118 HEAD -> patch-queue/debian/precise-wikimedia
* 14:37 hashar: Mysql was down on Precise slaves. Apparently rebooted 17 days ago and I guess mysql does not spawn on boot. Restarted mysql on all Precise via: salt -v '*slave-precise*' cmd.run 'start mysql'
* 09:35 godog: reboot deployment-imagescaler01 to enable memory cgroup
* 08:29 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/313387/ Filter out refs/meta/config from all pipelines  T52389
== 2016-10-10 ==
* 15:45 dcausse: deployment-prep deployment-elastic0[5-8]: reduce the number of replicas to 1 max for all indices
== 2016-10-07 ==
* 20:10 hashar: Created repository.integration.eqiad.wmflabs to play/Test Sonatype Nexus
* 20:10 hashar: rebooting integration-puppetmaster01
* 07:55 hashar: Upgrading Nodepool image for Jessie
== 2016-10-06 ==
* 14:45 hashar: deployment-mira disarmed/rearmed keyholder in an attempt to clear a Shinken alarm
* 12:16 hashar: Jenkins slave deployment-tin.eqiad , removing label "deployment-tin.eqiad"  it has "BetaClusterBastion" and all jobs are bound to it already
== 2016-10-05 ==
* 19:33 andrewbogott: removing mediawiki::conftool from deployment-mediawiki04, deployment-mediawiki06, deployment-mediawiki05
== 2016-10-04 ==
* 19:43 andrewbogott: removed contint::slave_scripts and associated files from deployment-sca01 and  deployment-sca02
* 16:22 bd808: Restarted puppetmaster process on deployment-puppetmaster
* 16:20 bd808: deployment-puppetmaster: removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/305256/; conflicts with upstream changes
* 15:01 godog: shutdown deployment-poolcounter02, replaced by deployment-poolcounter04 - T123734
* 09:03 hashar: Regenerating configuration of all Jenkins job due to https://gerrit.wikimedia.org/r/#/c/313306/
* 01:14 twentyafterfour: New scap command line autocompletions are now installed on deployment-tin and deployment-mira refs T142880
== 2016-10-03 ==
* 22:40 thcipriani: manual rebase on deployment-puppetmaster:/var/lib/git/operations/puppet
* 22:05 thcipriani: reapplied beta::deployaccess to mediawiki servers
* 21:42 cscott: updated OCG to version 0bf27e3452dfdc770317f15793e93e6e89c7865a
* 21:36 cscott: starting OCG deploy
* 13:43 hashar: Added integration-slave-trusty-1014  back in the pool
* 13:41 hashar: Tip of the day: to reboot an instance and bypass molly-guard: /sbin/reboot
* 13:39 hashar: integration-slave-trusty-1014  upgrading packages, clean up and rebooting it
* 13:37 hashar: marked integration-slave-trusty-1014 offline. Cant run job / get stuck somehow
* 10:21 godog: add role::prometheus::node_exporter to classes in hiera:deployment-prep T144502
== 2016-10-01 ==
* 09:41 hashar: beta: shutdown deployment-db1 and deployment-db2 . Databases have been migrated to other hosts T138778
== 2016-09-29 ==
* 15:43 hashar: logstash-beta: refreshed the field list via https://logstash-beta.wmflabs.org/app/kibana#/settings/indices/logstash
* 13:52 hashar: Restarted jobrunner / jobchron on deployment-jobrunner02 . Were no more logging to /var/log/mediawiki/ somehow
* 13:51 hashar: Restarted udp2log on deployment-fluorine02
* 10:50 legoktm: deploying https://gerrit.wikimedia.org/r/313384
* 10:37 hashar: Jenkins upgrade AnsiColor plugin from 0.3.1 to 0.4.2
* 10:28 hashar: Upgrading Jenkins plugins with zeljkof :]
* 08:59 hashar: Hopefully going to get beta fixed via mw/core revert patch https://gerrit.wikimedia.org/r/313373
== 2016-09-28 ==
* 23:56 MaxSem: Deleted varnish cache files on deployment-cache-upload04 to free up space, disk full
* 21:48 hasharAway: deployment-tin:  service nscd  restart
* 21:43 hasharAway: beta cluster update database is broken :/  Filled T146947 about it
* 21:25 hasharAway: deployment-tin: sudo -H -u www-data php5 /srv/mediawiki-staging/multiversion/MWScript.php update.php --wiki=commonswiki --quick
* 21:18 hasharAway: https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ is broken for unkwnon reason :(
* 20:48 hasharAway: Deleted deployment-tin02 via Horizon. Replaced by deployment-tin
* 20:19 hasharAway: restarted keyholder on deployment-tin
* 20:11 hasharAway: Switch Jenkins slave deployment-mira.eqiad to deployment-tin.eqiad
* 20:09 hasharAway: deployment-tin: keyholder arm
* 20:08 hasharAway: deployment-tin for instance in `grep deployment /etc/dsh/group/mediawiki-installation`; do ssh-keyscan `dig +short $instance` >> /etc/ssh/ssh_known_hosts; done;
* 19:49 hasharAway: Dropping deployment-tin02 , replacing it with deployment-tin which has been rebuild to Jessie T144006
* 12:44 hashar: Cant finish up the switch to deployment-tin,  puppet still does not pass due to weird clone issues ...
* 11:48 hashar: Deleting deployment-tin Trusty instance and recreate one with same hostname as Jessie; Meant to replace deployment-tin02  T144006
* 10:44 hashar: CI updating all mwext-Wikibase* jenkins jobs for https://gerrit.wikimedia.org/r/#/c/313056/  T142158
* 10:43 hashar: Updating slave scripts for "Disable garbage collection for mw-phpunit.sh"  https://gerrit.wikimedia.org/r/313051  T142158
* 08:31 hashar: Reloading Zuul to deploy dc2ada37
== 2016-09-27 ==
* 20:11 hashar: Reloading Zuul to deploy 3c3289aa1a  for T143938 and T146783
* 16:29 anomie: Cherry-picked https://gerrit.wikimedia.org/r/#/c/313035/ on deployment-puppetmaster
== 2016-09-26 ==
* 23:58 bd808: Started udp2log-mw on deployment-fluorine02 for T146723
* 11:35 hashar: deployment-salt02 : autoremoving a bunch of java related packages
* 11:31 hashar: rebooting deployment-salt02  has a kernel soft lock while hitting the disk
* 11:24 hashar: beta: mass upgrading all debian packages on all instances
* 10:32 hashar: beta: on deployment-pdf01 rm -fR /home/cscott/tmp/npm*
* 10:29 hashar:  deployment-pdf01 apt-get upgrade / cleaning files left over etc
* 10:28 hashar: beta: on deployment-pdf01 rm -fR /home/cscott/.npm/ T145343
== 2016-09-24 ==
* 20:08 hashar: deployment-tin is shutdown. Replaced by Jessie deployment-tin02
* 20:02 hashar: deployment-mira: ssh-keyscan deployment-tin02.deployment-prep.eqiad.wmflabs >> /etc/ssh/ssh_known_hosts
* 20:00 hashar: beta: dropping deployment-tin (ubuntu) replaced by deployment-tin02 (jessie). Primary is still deployment-mira (https://gerrit.wikimedia.org/r/#/c/312654/  T144578 )
== 2016-09-23 ==
* 20:21 hashar: integration:  salt -v '*trusty*' cmd.run 'service mysql start'
* 20:00 hashar: rebooting all CI permanent slaves.  Making sure nothing is left on /mnt (which is no more mounted)
* 19:53 hashar: added a 30 minutes build timeout to https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/
* 15:02 hashar: rebooting integration-slave-jessie-1001
* 14:04 hashar: remove the /mnt based tmpfs for T146381 /  https://gerrit.wikimedia.org/r/#/c/312518/ via: salt -v '*' cmd.run 'umount /mnt/home/jenkins-deploy/tmpfs'
* 13:41 hashar: Switching tmpfs from /mnt to /srv https://gerrit.wikimedia.org/r/#/c/312330/  and running fab deploy_slave_scripts
== 2016-09-22 ==
* 19:29 hasharAway: switching Jenkins slaves workspace from /mnt/jenkins-workspace to /srv/jenkins-workspace  (actually the same dir/inode on the filesystem)
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/312158
== 2016-09-21 ==
* 18:22 yuvipanda: shutting down integration-puppetmaster
* 17:26 yuvipanda: cherry-pick https://gerrit.wikimedia.org/r/#/c/312044/ on deployment-puppetmaser
* 16:41 hashar: deployment-tin02 initiale provisioning is complete. Gotta add it as a deployment server via a puppet.git patch
* 16:01 hashar: deployment-tin02 applied puppet classes beta::autoupdater, beta::deployaccess, role::deployment::server, role::labs::lvm::srv
* 15:32 hashar: spawned deployment-tin02
* 14:55 hashar: removed the CI puppet class from deployment-sca01 and deployment-sca02 .  Stopped services using /srv  ,  unmounted /srv, removed it from /etc/fstab
* 14:27 hashar: deployment-sca01 and deployment-sca02 are now broken.  The CI puppet class mount /srv which ends up being only 500 MBytes
* 14:08 hashar: deployment-mira adding puppet class beta::autoupdater
* 14:06 hashar: Enabling Jenkins slave deployment-mira
* 14:05 hashar: deployment-mira seems ready for action and is the primary deployment server.  Enabling jenkins to it
* 11:25 hashar: removing Jenkins slave deployment-tin , deployment-mira is the new deployment master  T144578
* 10:58 hashar: Changing Jenkins slaves home dir for deployment-sca01 and deployment-sca02  from /mnt/home/jenkins-deploy to /srv/jenkins/home/jenkins-deploy
* 10:57 hashar: Changing Jenkins slaves home dir for deployment-tin and deployment-mira from /mnt/home/jenkins-deploy to /srv/jenkins/home/jenkins-deploy
* 10:10 hashar: deployment-mira removing "role::labs::lvm::srv"  duplicate with role::ci::slave::labs::common
* 10:07 hashar: Making deployment-mira a Jenkins slave by applying puppet class role::ci::slave::labs::common  T144578
* 10:05 hashar: Arming keyholder on deployment-mira
* 09:43 hashar: beta: switching master deployment server from deployment-tin to deployment-mira
* 09:34 hashar: From [[Hiera:deployment-prep]] remove bit already in puppet:  "scap::deployment_server": deployment-tin.deployment-prep.eqiad.wmflabs
* 08:55 moritzm: remove mira from deployment-prep (replaced by deployment-mira)
* 08:37 hashar: beta: manually rebased puppetmaster
* 08:11 elukey: terminated jobrunner01 and removed from deployment-prep's sacp dsh list
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/311927
== 2016-09-20 ==
* 21:49 hashar: Deleting deployment-mira02 /srv was too small. Replaced by deployment-mira
* 20:54 hashar: from deployment-tin for T144578, accept ssh host key of deployment-mira :  sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mira.deployment-prep.eqiad.wmflabs
* 20:47 hashar: Creating deployment-mira instance with flavor c8.m8.s60 (8 cpu, 8G RAM and 60G disk) T144578
* 19:00 thcipriani: cherry-picked https://gerrit.wikimedia.org/r/#/c/311760/ to deployment-puppetmaster to fix failing beta-scap-eqiad job, had to manually start rsync, puppet failed to start
* 18:38 hashar: on tin: `sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mira02.deployment-prep.eqiad.wmflabs`  - T144006
* 18:33 hashar: on deployment-mira02  ran `sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs` per T144006
* 18:01 marxarelli: deployed mediawiki-config changes on beta cluster. back in read/write mode using new database instances
* 17:37 marxarelli: deployment-db04 restored from backup and replication started
* 16:54 marxarelli: upgraded package and data to mariadb 10 on deployment-db03
* 16:31 marxarelli: cherry picking operations/puppet patches (T138778) to deployment-puppetmaster
* 16:30 moritzm: rebooting deployment-mira02
* 16:23 marxarelli: applied innodb transaction logs to deployment-db1 backup and successfully restored on deployment-db03
* 15:47 marxarelli: completed innobackupex on deployment-db1. copying backup to deployment-db03 for restoration
* 14:54 hashar: beta: cherry picking fix up for the jobrunner logging https://gerrit.wikimedia.org/r/#/c/311702/ and  https://gerrit.wikimedia.org/r/311719 T146040
* 14:44 marxarelli: entering read-only mode on beta cluster
* 14:27 elukey: stopped puppet, jobrunner and jobchron on deployment-jobrunner01
* 14:20 marxarelli: disabling beta cluster jenkins jobs in preparation for data migration (T138778)
* 13:07 godog: add deployment-prometheus01 instance T53497
* 11:20 elukey: applied beta::deployaccess, role::labs::lvm::srv, role::mediawiki::jobrunner to jobrunner02
* 10:45 elukey: created deployment-jobrunner02 in deployment-prep
== 2016-09-19 ==
* 22:01 legoktm: shutdown integration-puppetmaster
* 21:29 yuvipanda: regenerated client certs only on integration-puppetmaster01, seems ok now
* 20:46 yuvipanda: re-enable puppet everywhere
* 20:43 yuvipanda: enable puppet and run on integration-slave-trusty-1003.eqiad.wmflabs
* 20:41 yuvipanda: accidentally deleted /var/lib/puppet/ssl on integration-puppetmaster01 as well, causing it to lose keys. Reprovision by pointing to labs puppetmaster
* 20:34 yuvipanda: rm -rf /var/lib/puppet/ssl on all integration nodes
* 20:34 yuvipanda: copied /etc/puppet/puppet.conf from integration-trusty-slave-1001 to all integration
* 20:25 yuvipanda: delete /etc/puppet/puppet.conf.d/10-self.conf and /var/lib/puppet/ssl on integration-slave-trusty-1001
* 20:20 yuvipanda: re-enabled puppet on integration-slave-trusty-1001
* 20:08 yuvipanda: reset puppetmaster of integration-puppetmaster01 to be labs puppetmaster
* 20:03 yuvipanda: disable puppet across integration project, moving puppetmasters
* 19:49 legoktm: creating T144951 enabled role::puppetmaster::standalone role on integration-puppetmaster01
* 19:33 legoktm: creating T144951 integration-puppetmaster01 instance using m1.small and debian jessie
* 15:11 hashar: beta: updating jobrunner service 0dc341f..a0e8216
== 2016-09-17 ==
* 07:11 legoktm: deploying https://gerrit.wikimedia.org/r/311024
== 2016-09-16 ==
* 21:03 hashar: deployment-tin  did a git gc on /srv/deployment/ores  That freed up disk space and cleared an alarm on co master mira02
* 21:00 hashar: deleted deployment-parsoid05
* 20:52 hashar: fixed puppet on deployment-parsoid05 . Temporary instance will delete it later to clear out shinken.wmflabs.org
* 20:27 hashar: beta:  force running puppet in batches of 4 instances:  salt --batch 4 -v 'deployment-*' cmd.run 'puppet agent -tv'
* 20:13 hashar: beta: restarted puppetmaster
* 20:07 hashar: beta: salt -v '*' cmd.run 'rm -fR /var/lib/puppet/client/ssl/'
* 20:07 hashar: beta: stopping puppetmaster,  rm -f /var/lib/puppet/server/ssl/ca/signed/*
* 19:53 hashar: beta created instance "deployment-parsoid05" Should be deleted later, that is merely to purge the hostname from Shinken ( http://shinken.wmflabs.org/host/deployment-parsoid05 )
* 11:42 hashar: beta: apt-get upgrade on deployment-jobrunner01
* 11:36 hashar: apt-get upgrade on deployment-tin , bring in a new hhvm version and others
== 2016-09-15 ==
* 22:29 legoktm: sudo salt '*precise*' cmd.run 'service mysql start', all mysql's are down
* 16:45 godog: install xenial kernel on deployment-zotero01 and reboot T145793
* 16:18 hashar: prometheus enabled on all beta cluster instance.  Does not support Precise hence puppet will fail on the last two Precise instances deployment-db1 and deployment-db2  until they are migrated to Jessie  T138778
* 15:53 godog: add role::prometheus::node_exporter to classes in hiera:deployment-prep T144502
* 15:10 hashar: beta: Applying puppet class role::prometheus::node_exporter to mira02 just like mira.  That is for godog
* 15:08 hashar: T144006 Disabled Jenkins job  beta-scap-eqiad.  On mira02  rm -fR /srv/*  .  Applying puppet for role::labs::lvm::srv
* 15:05 hashar: T144006  Applying class role::labs::lvm::srv to mira02  (it is out of disk space :D )
* 14:45 hashar: T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@mira02.deployment-prep.eqiad.wmflabs
* 14:44 hashar: T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki05.deployment-prep.eqiad.wmflabs
* 12:33 elukey: added base::firewall, beta::deployaccess, mediawiki::conftool, role::mediawiki::appserver to mediawiki05
* 12:20 elukey: terminate mediawiki02 to create mediawiki05
* 10:48 hashar: beta: cherry picking moritzm patch https://gerrit.wikimedia.org/r/#/c/310793/ "Also handle systemd in keyholder script" T144578
* 09:33 hashar: T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki06.deployment-prep.eqiad.wmflabs
* 09:10 elukey: executed git pull and then git rebase -i on deployment puppet master
* 08:52 elukey: terminated mediawiki03 and created mediawiki06
* 08:45 elukey: removed mediawiki03 from puppet with https://gerrit.wikimedia.org/r/#/c/310749/
* 02:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/310701
== 2016-09-14 ==
* 21:37 hashar: integration: setting "ulimit -c 2097152" on all slaves due to Zend PHP segfaulting  T142158
* 14:31 hashar: Added otto to integration labs project
* 13:28 gehel: upgrading deployment-logstash2 to elasticsearch 2.3.5 - T145404
* 09:27 hashar: Deleting deployment-mediawiki01 , replaced by deployment-mediawiki04  T144006
* 07:19 legoktm: sudo salt '*trusty*' cmd.run 'service mysql start', it was down on all trusty salves
* 07:17 legoktm: mysql just died on a bunch of slaves (trusty-1013, 1012, 1001)
== 2016-09-13 ==
* 17:02 marxarelli: re-enabling beta cluster jenkins jobs following maintenance window
* 16:59 marxarelli: aborting beta cluster db migration due to time constraints and ops outage. will reschedule
* 15:34 marxarelli: disabled beta jenkins builds while in maintenance mode
* 15:18 marxarelli: starting 2-hour read-only maintenance window for beta cluster migration
* 10:06 hashar: beta: manually updated  jobrunner install on deployment-jobrunner01 and deployment-tmh01 then reloaded the services with:  service jobchron reload
* 10:02 hashar: Trebuchet is broken for /srv/deployment/jobrunner/jobrunner  cant reach the deploy minions somehow.  Did the update manually
* 10:00 hashar: Upgrading beta cluster jobrunner to catch up with upstream b952a7c..0dc341f  merely picking up a trivial log change ( https://gerrit.wikimedia.org/r/#/c/297935/ )
* 09:40 hashar: Unpooled deployment-mediawiki01 from scap and varnish. Shutting down instance.  T144006
* 09:02 hashar: on deployment-tin, accepted mediawiki04 host key for jenkins-deploy user : sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs  T144006
* 08:26 hashar:  mwdeploy@deployment-mediawiki04  manually accepted ssh host key of deployment-tin  T144006
* 08:17 hashar: beta: manually accepted ssh host key for deployment-mediawiki04 as user mwdeploy on deployment-tin and mira T144006
* 07:46 gehel: upgrading elasticsearch to 2.3.5 on deployment-elastic0? - T145404
== 2016-09-12 ==
* 14:41 elukey: applied base::firewall, beta::deployaccess, mediawiki::conftool, role::mediawiki::appserver to deployment-mediawiki04.deployment-prep.eqiad.wmflabs (Debian jessie instance) - T144006
* 12:50 gehel: rolling back upgrading elasticsearch to 2.4.0 on deployment-elastic05 - T145058
* 12:03 gehel: upgrading elasticsearch to 2.4.0 on deployment-elastic0? - T145058
* 12:01 hashar: Gerrit: made analytics-wmde group to be owned by themselves
* 11:57 hashar: Gerrit: added ldap/wmde as an included group of the 'wikidata' group. Asked by and demoed to addshore
== 2016-09-11 ==
* 18:45 legoktm: deploying https://gerrit.wikimedia.org/r/309829
== 2016-09-09 ==
* 20:53 thcipriani: testing scap 3.2.5-1 on beta cluster
* 11:08 hashar: Added git tag for latest versions of mediawiki/selenium and mediawiki/ruby/api
* 09:30 legoktm: Image ci-jessie-wikimedia-1473412532 in wmflabs-eqiad is ready
* 08:53 legoktm: added phpflavor-php70 label to integration-slave-jessie-100[1-5]
* 08:49 legoktm: deploying https://gerrit.wikimedia.org/r/309048
== 2016-09-08 ==
* 21:33 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/309413  " Inject PHP_BIN=php5 for php53 jobs"
* 20:00 hashar: nova delete ci-jessie-wikimedia-369422  (was stuck in deleting state)
* 19:49 hashar: Nodepool, deleting instances that Nodepool lost track of (from nodepool alien-list)
* 19:47 hashar: nodepool cant delete: ci-jessie-wikimedia-369422 [ delete | 2.24  hours . Stuck in task_state=deleting  :(
* 19:46 hashar: Nodepool looping over some tasks since 17:45  ( https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=21&fullscreen  )
* 19:26 legoktm: repooled integration-slave-jessie-1005 now that php7 testing is done
* 19:19 hashar: integration: salt -v '*' cmd.run 'cd /srv/deployment/integration/slave-scripts; git pull' | https://gerrit.wikimedia.org/r/308931
* 19:12 hashar: integration:  salt -v '*' cmd.run 'cd /srv/deployment/integration/slave-scripts; git pull'  | https://gerrit.wikimedia.org/r/309272
* 17:08 legoktm: deleted integration-jessie-lego-test01
* 16:50 legoktm: deleted integration-aptly01
* 10:03 hashar: Delete Jenkins job https://integration.wikimedia.org/ci/job/mwext-VisualEditor-sync-gerrit/ that has been left behind. It is no more needed. T51846 T86659
* 10:02 hashar: Delete mwext-VisualEditor-sync-gerrit job, already got removed by ostriches in 139d17c8f1c4bcf2bb761e13a6501e4d85684066 . The issue in Gerrit (T51846) has been fixed. Poke T86659 , one less job on slaves.
== 2016-09-07 ==
* 20:44 matt_flaschen: Re-enabled beta-code-update-eqiad .
* 20:35 hashar: Updated security group for deployment-prep labs project.  Allow ssh port 22 from contint1001.wikimedia.org (matching rules for gallium). T137323
* 20:30 hashar: Updated security group for contintcloud and integration labs project.  Allow ssh port 22 from contint1001.wikimedia.org (matching rules for gallium). T137323
* 20:14 matt_flaschen: Temporarily disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ to test live revert of aa0f6ea
* 16:09 hashar: Nodepool back in action. Had to manually delete some instances in labs
* 15:58 hashar: Restarting Nodepool . Lost state when labnet got moved T144945
* 13:13 hashar: Image ci-jessie-wikimedia-1473253681 in wmflabs-eqiad is ready  , has php7 packages. T144872
* 11:53 hashar: Force refreshing Nodepool jessie snapshot to get PHP7 included T144872
* 11:03 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/308955/ "contint: prefer our bin/php alternative"  T144872
* 10:55 hashar: integration: dropped PHP7 cherry pick from puppet master. https://gerrit.wikimedia.org/r/#/c/308918/ has been merged.  Pushing it to the fleet of permanent Jessie slaves. T144872
* 10:37 hashar: beta: cleaning up salt-keys on deployment-salt02 . Bunch of instances got deleted
* 09:41 hashar: Moving rake jobs back to Nodepool ( T143938 ) with https://gerrit.wikimedia.org/r/#/c/306723/ and https://gerrit.wikimedia.org/r/#/c/306724/
* 05:57 legoktm: deploying https://gerrit.wikimedia.org/r/308932 https://gerrit.wikimedia.org/r/299697
* 05:26 legoktm: cherry-picked https://gerrit.wikimedia.org/r/#/c/308918/ onto integration-puppetmaster with a hack that has it only apply to integration-slave-jessie-1005
* 04:59 legoktm: added Krenair to integration project to help debug puppet stuff
* 04:35 legoktm: depooled integration-slave-jessie-1005 in jenkins so I can test puppet stuff on it
== 2016-09-06 ==
* 13:58 hashar: Qunit jobs should be all fine again now.  T144802
* 13:46 hashar: nodepool.SnapshotImageUpdater: Image ci-jessie-wikimedia-1473169259 in wmflabs-eqiad is ready  T144802
* 13:20 hashar: Rebuilding Nodepool Jessie image to hopefully include libapache-mod-php5 and restore qunit jobs behavior  T144802
* 10:37 hashar: gerrit: mark apps/android/commons hidden since it is now community maintained on GitHub. Will avoid confusion.  T127678
* 09:11 hashar: nodepool.SnapshotImageUpdater: Image ci-trusty-wikimedia-1473152801 in wmflabs-eqiad is ready
* 09:06 hashar: nodepool.SnapshotImageUpdater: Image ci-jessie-wikimedia-1473152393 in wmflabs-eqiad is ready
* 09:00 hashar: Trying to refresh Nodepool Jessie image .  Image properties have been dropped, should fix it
== 2016-09-05 ==
* 14:08 hashar: Refreshing Nodepool base images for Trusty and Jessie.  Managed to build new ones after T143769
== 2016-09-02 ==
* 20:36 legoktm: deploying https://gerrit.wikimedia.org/r/308227
* 15:17 hashar: Bringing tox jobs to Nodepool with https://gerrit.wikimedia.org/r/#/c/306725/
== 2016-09-01 ==
* 19:00 urandom: T130861: Restarting Cassandra on deployment-restbase0[1-2]
* 18:58 urandom: T130861: De-cherry-picking https://gerrit.wikimedia.org/r/#/c/282466/
* 18:34 urandom: T130861: Restarting Cassandra on deployment-restbase0[1-2]
* 18:32 urandom: T130861: Cherry picking https://gerrit.wikimedia.org/r/#/c/282466/ to deployment-puppetmaster
* 16:38 legoktm: deploying https://gerrit.wikimedia.org/r/307794
* 12:22 hashar: migrating deployment-tin keyholder to use base::service_unit for moritm https://gerrit.wikimedia.org/r/#/c/307510/ + reboot + keyholder arm
* 03:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/307909
== 2016-08-31 ==
* 23:40 bd808: forced puppet run on deployment-salt02. Had not run automatically for 8 hours
* 23:36 bd808: Deleted /data/scratch on integration-slave-trusty-1016 to fix puppet
* 23:32 bd808: Deleted /data/scratch on integration-slave-trusty-1013 to fix puppet
* 23:22 bd808: Deleted /data/scratch on integration-slave-trusty-1012 to fix puppet
* 23:19 bd808: Deleted /data/scratch on integration-slave-trusty-1011 to fix puppet
* 23:15 bd808: Deleted /data/scratch on integration-slave-precise-1012 to fix puppet
* 23:11 bd808: Deleted /data on integration-slave-precise-1011 to fix puppet
* 23:08 bd808: Deleted /data on integration-slave-jessie-1001 to fix puppet
* 23:04 bd808: Deleted empty /data, /data/project, and /data/scratch on integration-puppetmaster to fix puppet
* 22:59 bd808: Deleted empty /data, /data/project, and /data/scratch on integration-publisher to fix puppet
* 01:44 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/307670
== 2016-08-30 ==
* 23:31 yuvipanda: cherry-picking https://gerrit.wikimedia.org/r/#/c/307656/ fixed puppet on the elasticsearch machines!
* 22:29 yuvipanda: in lieu of blood sacrifice, restart puppetmaster on deployment-pupetmaster
* 21:44 yuvipanda: use clush to fix puppet.conf of all clients, realize also accidentally set a client's puppet.conf for the server, recover server's old conf file from a cat in shell history, restore, breathe sigh of relief
* 21:37 yuvipanda: sudo takes like 15s each time, is there no god?
* 21:36 yuvipanda: managed to get vim into a state where I can not quit it, probably recording a macro. I hate computers
* 21:16 yuvipanda: deployment-pdf01 fixed manually
* 21:15 yuvipanda: deployment-pdf02 has proper ssl certs mysteriously without me doing anything
* 21:06 yuvipanda: moved deployment-db[12], deployment-stream to not use role::puppet::self, attempting to semi-automate rest
* 20:52 yuvipanda: cherry-picked appropriate patch on deployment-puppetmaster for T120159, did https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep/host/deployment-puppetmaster&oldid=818847 to make sure the puppetmaster allows connections from elsewhere
* 19:48 legoktm: deploying https://gerrit.wikimedia.org/r/306710
* 19:13 bd808: Fixed puppet runs on deployment-sca0[12] with cherry-pick of https://gerrit.wikimedia.org/r/#/c/307561
* 18:57 bd808: Duplicate declaration: File[/srv/deployment] is already declared in file /etc/puppet/modules/contint/manifests/deployment_dir.pp:14; cannot redeclare at /etc/puppet/modules/service/manifests/deploy/common.pp:12 on node deployment-sca01.deployment-prep.eqiad.wmflabs
* 18:40 bd808: Puppet busted on deployment-aqs01 -- Could not find data item analytics_hadoop_hosts in any Hiera data file and no default supplied at /etc/puppet/manifests/role/aqs.pp:46
* 12:59 hashar: beta: revert master branch to origin.  Ran scap and enabled again beta-code-update-eqiad job.
* 12:55 hashar: Running scap on beta cluster via https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/117786/console  T143889
* 12:53 hashar: Cherry picking https://gerrit.wikimedia.org/r/#/c/307501/ on beta cluster for T143889
* 12:51 hashar: disabling https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/  to cherry pick a revert patch
== 2016-08-29 ==
* 07:56 hashar: hard rebooting integration-slave-trusty-1012 via horizon and restarting puppet manually
* 07:50 hashar: integration-slave-trusty-1013  puppet.conf  certname was set to 'undef' breaking puppet
== 2016-08-27 ==
* 20:51 hashar: integration: tweak sudo policy for jenkins-deploy running cowbuilder: env_keep+=DEB_BUILD_OPTIONS
* 20:24 hashar: Manually installing jenkins-debian-glue 0.17.0 on integration-slave-jessie-1004 and integration-slave-jessie-1005 ( T142891 ) .  That is to support PBUILDER_USENETWORK T141114
* 20:05 hashar: Jenkins added global env variable BUILD_TIMEOUT set to 30  for T144094
== 2016-08-26 ==
* 22:29 legoktm: deploying https://gerrit.wikimedia.org/r/307025
* 08:15 Amir1: restart uwsgi-ores and celery-ores-worker in deployment-sca03 (T143567)
* 08:11 hashar: beta-scap-eqiad job is back in operation.  Was blocked on logstash not being reachable. T143982
* 08:10 hashar: deployment-logstash2 is back after a hard reboot. T143982
* 08:07 hashar: rebooting deployment-logstash02 via Horizon. Kernel hang apparently T143982
* 08:00 hashar: beta-scap-eqiad failing investigating
* 07:54 Amir1: cherry-picked 306839/1 into deployment-puppetmaster
* 00:28 twentyafterfour: restarted puppetmaster service on deployment-puppetmaster
== 2016-08-25 ==
* 23:15 Amir1: cherry-picked 306839/1 into puppetmaster
* 20:10 hashar: Delete  integration-slave-trusty-1023 with label AndroidEmulator.  The Android job has been migrated to a new Jessie based instance via  T138506
* 19:05 hashar: hard rebooting integration-raita via Horizon
* 16:04 hashar: fixing puppet.conf on integration-slave-trusty-1013  it mysteriously considered itself as the puppetmaster
* 16:02 hashar: integration restarted puppetmaster service
* 08:28 hashar: beta update database fixed
* 08:28 hashar: beta cluster update database failed due to: "Your composer.lock file is up to date with current dependencies!"  Probably a race condition with ongoing scap.
== 2016-08-24 ==
* 15:14 halfak: deploying ores d00171
* 09:50 hashar: deployment-redis02 fixed AOF file /srv/redis/deployment-redis02-6379.aof and restarted the redis instance should fix T143655  and might help T142600
* 09:43 hashar: T143655 stopping redis 6379 on deployment-redis02 : initctl stop redis-instance-tcp_6379
* 09:38 hashar: deployment-redis02 initctl stop redis-instance-tcp_6379 && initctl start redis-instance-tcp_6379 | That did not fix it magically though  T143655
== 2016-08-23 ==
* 18:21 legoktm: deploying https://gerrit.wikimedia.org/r/306257
* 16:38 bd808: Fixed ops/puppet sync by removing stale cherry-pick of https://gerrit.wikimedia.org/r/#/c/305996/
* 08:22 hashar: running puppet on integration-slave-trusty-1014
* 08:18 hashar: reboot integration-slave-trusty-1014
* 08:16 hashar: disabled/enabled Jenkins Gearman client to remove deadlock with Throttle plugin
== 2016-08-22 ==
* 23:40 legoktm: updating slave_scripts on all slaves
== 2016-08-18 ==
* 22:03 bd808: deployment-fluorine02: Hack 'datasets:x:10003:997::/home/datasets:/bin/bash' into /etc/passwd for T117028
* 20:30 MaxSem: Restarted hhvm on appservers for wikidiff2 upgrades
* 19:03 MaxSem: Upgrading hhvm-wikidiff2 in beta cluster
* 16:53 legoktm: deploying https://gerrit.wikimedia.org/r/#/c/305532/
== 2016-08-17 ==
* 22:28 legoktm: deploying https://gerrit.wikimedia.org/r/305408
* 21:33 cscott: updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29
* 21:28 bd808: restarted salt-minion on deployment-pdf02
* 21:26 bd808: restarted salt-minion on deployment-pdf01
* 21:15 cscott: starting OCG deploy to beta
* 14:10 gehel: upgrading elasticsearch to 2.3.4 on deployment-logstash2.deployment-prep.eqiad.wmflabs
* 13:28 gehel: upgrading elasticsearch to 2.3.4 on deployment-elastic*.deployment-prep + JVM upgrade
== 2016-08-16 ==
* 23:10 thcipriani: max_servers at 6, seeing 6 allocated instances, still seeing 403 already used 10 of 10 instances :((
* 22:37 thcipriani: restarting nodepool, bumping max_servers to match up with what openstack seems willing to allocate (6)
* 09:06 Amir1: removing ores-related-cherry-picked commits from deployment-puppetmaster
== 2016-08-15 ==
* 21:30 thcipriani: update scap on beta to 3.2.3-1 bugfix release
* 02:30 bd808: Forced a zuul restart -- https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
* 02:23 bd808: Lots and lots of "AttributeError: 'NoneType' object has no attribute 'name'" errors in /var/log/zuul/zuul.log
* 02:21 bd808: nodepool delete 301068
* 02:20 bd808: nodepool delete 301291
* 02:20 bd808: nodepool delete 301282
* 02:19 bd808: nodepool delete 301144
* 02:11 bd808: nodepool delete 299641
* 02:11 bd808: nodepool delete 278848
* 02:08 bd808: Aug 15 02:07:48 labnodepool1001 nodepoold[24796]: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403)
== 2016-08-13 ==
* 23:16 Amir1: cherry-picking 304678/1 into the puppetmaster
* 00:08 legoktm: deploying https://gerrit.wikimedia.org/r/304588
* 00:06 legoktm: deploying https://gerrit.wikimedia.org/r/304068
== 2016-08-12 ==
* 23:57 legoktm: p
* 23:57 legoktm: deploying https://gerrit.wikimedia.org/r/304587, no-o
* 18:19 Amir1: deploying 2ef24f2 to ores-beta in sca03
== 2016-08-10 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/304149
* 23:47 thcipriani: stopping nodepool to clean up
* 23:41 legoktm: deploying https://gerrit.wikimedia.org/r/304131
* 21:59 thcipriani: restarted nodepool, no trusty instances were being used by jobs
* 01:58 legoktm: deploying https://gerrit.wikimedia.org/r/303218
== 2016-08-09 ==
* 23:21 Amir1: ladsgroup@deployment-sca03:~$ sudo service celery-ores-worker restart
* 15:24 thcipriani: due to https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Jenkins_execution_lock
* 15:20 thcipriani: beta site updates stuck for 15 hours :(
* 02:17 legoktm: deploying https://gerrit.wikimedia.org/r/303741
* 02:16 legoktm: manually updated slave-scripts on all slaves via `fab deploy_slave_scripts`
* 00:56 legoktm: deploying https://gerrit.wikimedia.org/r/303726
== 2016-08-08 ==
* 23:33 Tim: deleted instance deployment-depurate01
* 16:19 bd808: Manually cleaned up root@logstash02 cronjobs related to logstash03
* 14:39 Amir1: deploying d00159c for ores in sca03
* 10:14 Amir1: deploying 616707c into sca03 (for ores)
== 2016-08-07 ==
* 12:01 hashar: Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403)
* 12:01 hashar: nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>)
* 11:54 hashar: Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403)
* 11:54 hashar: nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>)
== 2016-08-06 ==
* 12:31 Amir1: restarting uwsgi-ores and celery-ores-worker in deployment-sca03
* 12:28 Amir1: cherry-picked 303356/1 into the puppetmaster
* 12:00 Amir1: restarting uwsgi-ores and celery-ores-worker in deployment-sca03
== 2016-08-05 ==
* 17:54 bd808: Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/3 for testing
* 17:50 bd808: Removed stale cherry-picks for https://gerrit.wikimedia.org/r/#/c/302303/ and https://gerrit.wikimedia.org/r/#/c/300458/ that were blocking git rebase
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/303113
* 00:31 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/300068
== 2016-08-04 ==
* 20:07 marxarelli: Running jenkins-jobs update config/ 'selenium-*' to deploy https://gerrit.wikimedia.org/r/#/c/302775/
* 17:03 legoktm: jstart -N qamorebots /usr/lib/adminbot/adminlogbot.py --config ./confs/qa-logbot.py
== 2016-08-01 ==
* 20:28 thcipriani: restarting deployment-ms-be01, not responding to ssh, mw-fe01 requests timing out
* 08:28 Amir1: deploying fedd675 to ores in sca03
== 2016-07-29 ==
* 23:27 bd808: Rebooting deployment-logstash2; Console showed hung task timeouts (P3606)
* 15:55 hasharAway: pooled Jenkins slave integration-slave-jessie-1003 [10.68.21.145]
* 14:02 hashar: deployment-prep / beta : added addshore to the project
* 13:24 hashar: created integration-slave-jessie-1003 m1.medium to help processing debian-glue jobs
* 13:01 hashar: Upgrading Zuul on jessie slaves using https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb
* 12:53 hashar: Upgrading Zuul on precise slaves using https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3/zuul_2.1.0-391-gbc58ea3-wmf2precise1_amd64.deb
* 09:38 hashar: Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1
== 2016-07-28 ==
* 21:46 hashar_: xintegration: change sudo policy for jenkins-deploy to help on T141538 : env_keep+=WORKSPACE
* 12:18 hashar: installed 2.1.0-391-gbc58ea3-wmf1jessie1 on zuul-dev-jessie.integration.eqiad.wmflabs T140894
* 12:18 hashar: installed 2.1.0-391-gbc58ea3-wmf1jessie1 on zuul-dev-jessie.integration.eqiad.wmflabs
* 09:46 hashar: Nodepool: Image ci-trusty-wikimedia-1469698821 in wmflabs-eqiad is ready
* 09:35 hashar: Regenerated Nodepool image for Trusty.  The snapshot failed while upgrading grub-pc for some reason. Noticed with thcipriani yesterday
== 2016-07-27 ==
* 16:13 hashar: salt -v '*slave-trusty*' cmd.run 'service mysql start'    ( was missing on integration-slave-trusty-1011.integration.eqiad.wmflabs )
* 14:03 hashar: upgraded zuul on gallium via dpkg -i /root/zuul_2.1.0-391-gbc58ea3-wmf1precise1_amd64.deb    (revert is zuul_2.1.0-151-g30a433b-wmf4precise1_amd64.deb )
* 12:43 hashar: restarted Jenkins for some trivial plugins updates
* 12:35 hashar: hard rebooting integration-slave-trusty-1011 from Horizon. ssh lost, no log in Horizon.
* 09:46 hashar: manually triggered debian-glue on all operations/debs repo that had no jenkins-bot vote. Via zuul enqueue on gallium and list fetched from "gerrit query --current-patch-set 'is:open NOT label:verified=2,jenkins-bot project:^operations/debs/.*'|egrep '(ref|project):'"
* 06:21 Tim: created instance deployment-depurate01 for testing of role::html5depurate
== 2016-07-26 ==
* 20:13 hashar: Zuul deployed https://gerrit.wikimedia.org/r/301093 which adds 'debian-glue' job on all of operations/debs/ repos
* 18:10 ostriches: zuul: reloading to pick up config change
* 12:49 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/300827/ on deployment-puppetmaster
* 11:59 legoktm: also pulled in I73f01f87b06b995bdd855628006225879a17fee5
* 11:59 legoktm: deploying https://gerrit.wikimedia.org/r/301109
* 11:37 hashar: rebased integration puppetmaster git repo
* 11:31 hashar: enable puppet agent on integration-puppetmaster . Had it disabled while hacking on https://gerrit.wikimedia.org/r/#/c/300830/
* 08:42 hashar: T141269 On integration-slave-trusty-1018 , deleting workspace that has a corrupt git: rm -fR /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm*
* 01:08 Amir1: deployed ores a291da1 in sca03, ores-beta.wmflabs.org works as expected
== 2016-07-25 ==
* 22:45 legoktm: restarting zuul due to depends-on lockup
* 14:24 godog: bounce puppetmaster on deployment-puppetmaster
* 13:17 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/300827/ on deployment-puppetmaster
== 2016-07-23 ==
* 20:06 bd808: Cleanup jobrunner01 logs via -- sudo logrotate --force /etc/logrotate.d/mediawiki_jobrunner
* 20:03 bd808: Deleted jobqueues in redis with no matching wikis: ptwikibooks, labswiki
* 19:20 bd808: jobrunner01 spamming /var/log/mediawiki with attempts to process jobs for wiki=labswiki
== 2016-07-22 ==
* 20:26 hashar: T141114 upgraded jenkins-debian-glue from v0.13.0 to v0.17.0  on integration-slave-jessie-1001 and integration-slave-jessie-1002
* 19:07 thcipriani: beta-cluster has successfully used a canary for mediawiki deployments
* 16:53 thcipriani: bumping scap to v.3.2.1 on deployment-tin to test canary deploys, again
* 16:46 thcipriani: rolling back scap version to v.3.2.0
* 16:38 thcipriani: bumping scap to v.3.2.1 on deployment-tin to test canary deploys
* 13:02 hashar: zuul rebased patch queue on tip of upstream branch and force pushed branch. c3d2810...4ddad4e HEAD -> patch-queue/debian/precise-wikimedia (forced update)
* 10:32 hashar: Jenkins restarted and it pooled both integration-slave-jessie-1002  and  integration-slave-trusty-1018
* 10:23 hashar: Jenkins has some random deadlock. Will probably reboot it
* 10:17 hashar: Jenkins can't ssh / add slaves integration-slave-jessie-1002 or  integration-slave-trusty-1018 . Apparently due to some Jenkins deadlock in the ssh slave plugin :-/  Lame way to solve it: restart Jenkins
* 10:10 hashar: rebooting integration-slave-jessie-1002 and integration-slave-trusty-1018 . Hang somehow
* 10:06 hashar: T141083 salt -v '*slave-trusty*' cmd.run 'service mysql start'
* 09:55 hashar: integration-slave-trusty-1001 service mysql start
== 2016-07-21 ==
* 16:11 hashar: Updated our JJB fork cherry picking f74501e781f by madhuvishy.  Was made to support the maven release plugin. Branch bump is 10f2bcd..6fcaf39
* 16:04 hashar: integration/zuul.git .Updated upstream branch:bc58ea34125f11eb353abc3e5b96ac1efad06141  finally caught up with upstream \O/
* 15:13 hashar: integration/zuul.git .Updated upstream branch:  06770a85fcff810fc3e1673120710100fc7b0601:upstream
* 14:03 hashar: integration/zuul.git bumping upstream branch:  git push d34e0b4:upstream
* 03:18 greg-g: had to do https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update twice, seems to be back
* 00:13 bd808: Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/ to deployment-puppetmaster so wdqs nginx log parsing can be tested
== 2016-07-20 ==
* 13:55 hashar: beta: switching job beta-scap-eqiad to use 'scap sync' per https://gerrit.wikimedia.org/r/#/c/287951/  (poke thcipriani )
* 12:47 hashar: integration: enabled unattended upgrade on all instances by adding contint::packages::apt to https://wikitech.wikimedia.org/wiki/Hiera:Integration
* 10:28 hashar: beta dropped salt-key on deployment-salt02 for the three instances: deployment-upload.deployment-prep.eqiad.wmflabs , deployment-logstash3.deployment-prep.eqiad.wmflabs and deployment-ores-web.deployment-prep.eqiad.wmflabs
* 10:26 hashar: beta: rebased puppetmaster git repo. "Parsoid: Move to service::node"  has weird conflict https://gerrit.wikimedia.org/r/#/c/298436/
* 10:15 hashar: beta: removing puppet cherry pick of https://gerrit.wikimedia.org/r/#/c/258979/ "mediawiki: add conftool-specifc credentials and scripts"  abandonned/superseeded and caused a conflict
* 08:17 hashar: deployment-fluorine : deleting a puppet lock file /var/lib/puppet/state/agent_catalog_run.lock  (created at 2016-07-18 19:58:46 UTC)
* 01:53 legoktm: deploying https://gerrit.wikimedia.org/r/299930
== 2016-07-18 ==
* 20:56 thcipriani: Deleted deployment-fluorine:/srv/mw-log/archive/*-201605* freed 30 GB
* 15:00 hashar: Upgraded Zuul on the Precise slaves to zuul_2.1.0-151-g30a433b-wmf4precise1
* 12:10 hashar: (restarted qa-morebots)
* 12:10 hashar: Enabling puppet again on integration-slave-precise-1002 , removing Zuul-server config and adding the slave back in Jenkins pool
== 2016-07-16 ==
* 23:19 paladox: testing morebots
== 2016-07-15 ==
* 08:34 hashar: Unpooling integration-slave-precise-1002  will use it as a zuul-server test instance temporarily
== 2016-07-14 ==
* 18:54 ebernhardson: deployment-prep manually edited elasticsearch.yml on deployment-elastic05 and restarted to get it listening on eth0. Still looking into why puppet wrote out wrong config file
* 09:05 Amir1: rebooting deployment-ores-redis
* 08:29 Amir1: deploying 0e9555f to ores-beta (sca03)
== 2016-07-13 ==
* 16:05 urandom: Installing Cassandra 2.2.6-wmf1 on deployment-restbase0[1-2].deployment-prep.eqiad.wmflabs : T126629
* 13:58 hashar:  T137525 reverted Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb  . It could not connect to Gerrit reliably
* 13:46 hashar: T137525 Stopped zuul that ran in a terminal (with -d). Started it with the init script.
* 11:37 hashar: apt-get upgrade on deployment-mediawiki02
* 08:33 hashar: removing deployment-parsoid05 from the Jenkins slaves T140218
== 2016-07-12 ==
* 20:29 hashar: integration: force running unattended upgrade on all instances:  salt --batch 4 -v '*' cmd.run 'unattended-upgrade'  . That upgrades diamond and hhvm among others.  imagemagick-common has a prompt though
* 20:22 hashar: CI force running puppet on all instances:  salt --batch 5 -v '*' puppet.run
* 20:04 hashar: Maybe fix unattended upgrade on the CI slaves via https://gerrit.wikimedia.org/r/298568
* 16:43 Amir1: deploying f472f65 to ores-beta
* 10:11 hashar: Github created repos operations-debs-contenttranslation-apertium-mk-en and operations-docker-images-toollabs-images        for Gerrit replication
== 2016-07-11 ==
* 14:24 hashar: Removing ZeroMQ config from the Jenkins jobs. It is now enabled globally. T139923
* 10:16 hashar: T136188: on Trusty slaves, upgrading Chromium from v49 to v51:  salt -v '*slave-trusty-*' cmd.run 'apt-get -y install chromium-browser chromium-chromedriver chromium-codecs-ffmpeg-extra'
* 10:13 hashar: T136188: salt -v '*slave-trusty*' cmd.run 'rm /etc/apt/preferences.d/chromium-*'
* 10:09 hashar: Unpinning Chromium v49 from the Trusty slaves and upgrading to v51 for T136188
* 09:34 zeljkof: Enabled ZMQ Event Publisher on all Jobs in Jenkins
== 2016-07-09 ==
* 18:57 legoktm: deploying https://gerrit.wikimedia.org/r/297731 and https://gerrit.wikimedia.org/r/298142
* 14:07 bd808: Testing logstash change https://gerrit.wikimedia.org/r/#/c/298115/ via cherry-pick
== 2016-07-08 ==
* 16:08 hashar: scandium: git -C /srv/ssd/zuul/git/mediawiki/services/graphoid remote set-head origin --auto
* 16:06 hashar: scandium: git -C /srv/ssd/zuul/git/mediawiki/services/graphoid init &&  git -C /srv/ssd/zuul/git/mediawiki/services/graphoid remote add origin ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/services/graphoid
* 14:59 hashar: nodepool: rebuild Trusty image from scratch Image ci-trusty-wikimedia-1467989709 in wmflabs-eqiad is ready
* 12:35 hashar: beta:  find /data/project/upload7/*/*/thumb -type f -atime +30 -delete
* 10:31 hashar: beta: mass delete http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload files T64835
* 10:26 hashar: beta: mass delete http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload files
== 2016-07-07 ==
* 21:41 MaxSem: Chowned php-master/vendor back to jenkins-deploy
* 13:10 hashar: deleting integration-slave-trusty-1024 and integration-slave-trusty-1025  to free up some RAM. We have enough permanent Trusty slaves. T139535
* 02:43 MaxSem: started redis-server on deployment-stream
* 01:14 bd808: Restarted logstash on deployment-logstash2
* 01:13 MaxSem: Leaving my hacks for the night to collect data, if needed revert with cd /srv/mediawiki-staging/php-master/vendor && sudo git reset --hard HEAD && sudo chown -hR jenkins-deploy:wikidev .
* 00:50 bd808: Rebooting deployment-logstash3.eqiad.wmflabs; console full of hung process messages from kernel
* 00:27 MaxSem: Initialized ORES on all wikis where it's enabled, was causing job failures
* 00:13 MaxSem: Debugging a fatal in betalabs, might cause syncs to fail
== 2016-07-06 ==
* 20:30 hashar: beta: restarted mysql on both db1 and db2 so it takes in account the --syslog setting  T119370
* 20:08 hashar: beta:  on db1 and db2  move the MariaDB 'syslog' setting under [mysqld_safe] section. Cherry picked https://gerrit.wikimedia.org/r/#/c/296713/3 and reloaded mysql on both instances. T119370
* 14:54 hashar: Image ci-jessie-wikimedia-1467816381 in wmflabs-eqiad is ready  T133779
* 14:47 hashar_: attempting to refresh ci-jessie-wikimedia image to get librdkafka-dev included for T133779
== 2016-07-05 ==
* 21:54 hasharAway: CI has drained the gate-and-submit queue
* 21:37 hasharAway: Nodepool: nodepool delete  a few instances that would never spawn / have been stuck for ~ 40 minutes
== 2016-07-04 ==
* 18:58 hashar: Upgrading arcanist on permanent CI slaves since xhpast was broken T137770 
* 12:50 yuvipanda: migrating deployment-tin to labvirt1011
== 2016-07-03 ==
* 13:10 paladox: phabricator Update phab-01 and phab-05 (phab-02) and phab-03 to fix a security bug in phabricator (Did the update last night but forgot to log it)
* 12:04 jzerebecki: reloading zuul for 7e6a2e2..13ea50f
== 2016-07-02 ==
* 13:38 jzerebecki: reloading zuul for 15127b2..7e6a2e2
== 2016-06-30 ==
* 10:31 hashar: Deleting integration-slave-trusty-1015 . Can not bring up mysql T138074  and the ssh slave connection would not hold anyway. Must be broken somehow
* 10:04 hashar: Attempting to refresh Nodepool image for Jessie ( ci-jessie-wikimedia ). Been stall for 284 hours (12 days)
* 09:36 hashar: Trusty is missing the package arcanist ... :(
* 09:35 hashar: Attempting to refresh Nodepool image for Trusty ( ci-trusty-wikimedia ). Been stall for 283 hours (12 days)
== 2016-06-28 ==
* 21:33 halfak: deploying ores beec291
* 21:15 halfak: deploying ores 6979a98
== 2016-06-27 ==
* 22:32 eberhardson: deployment-prep deployed gerrit.wikimedia.org/r/296279 to puppetmaster to test kibana4 role
* 19:41 bd808: Rebooting deployment-logstash3.eqiad.wmflabs via wikitech. Console log full of blocked kworker messages, ssh non-responsive, and blocking logstash records being recorded.
* 18:20 thcipriani: deployment-puppetmaster.deployment-prep:/var/lib/git/labs/private modules/secret/secrets/keyholder keys conflicts resolved
* 18:09 bd808: Git repo at deployment-puppetmaster.deployment-prep:/var/lib/git/labs/private is behind upstream due to multiple modules/secret/secrets/keyholder local files that would be overwritten by upstream changes.
== 2016-06-24 ==
* 15:04 hashar: switch apps-android-wikimedia-* jobs to Jessie T138506
* 14:07 James_F: Killed https://integration.wikimedia.org/ci/job/pywikibot-core-tox-nose-jessie/556/console (stuck for 90 minutes)
* 09:54 hashar: T138506 Adding a JDK installation "Debian - OpenJdk 8" in Jenkins global configuration with JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
== 2016-06-23 ==
* 13:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/295691
* 12:13 hashar: Deleting integration-saltmaster and recreating it with Jessie T136410
* 10:14 hashar: T137807 Upgrading Jenkins TAP Plugin
* 08:55 hashar: integration: rebased puppet master by dropping a conflicting/obsolete patch
* 08:28 hashar: fixing puppet cert on deployment-cache-text04
== 2016-06-17 ==
* 10:35 jzerebecki: offlined integration-slave-trusty-1015 T138074
* 10:06 hashar: Refreshed Nodepool Trusty image
* 10:02 hashar: Refreshed Nodepool Jessie image
== 2016-06-14 ==
* 14:22 hashar: T136971 on tin MediaWiki 1.28.0-wmf.6, from 1.28.0-wmf.6, successfully checked out.  Applying security patches
* 11:21 hashar: T137797 Created Gerrit repository operations/debs/geckodriver  to package https://github.com/mozilla/geckodriver
== 2016-06-13 ==
* 21:11 hashar: https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1015/ put offline. Jenkins cant ssh / pool it for some reason
* 20:07 hashar: beta: update.php / database update finally pass!
* 19:55 hashar: T137615 deployment-db2, **eswiki** > CREATE INDEX echo_notification_event ON echo_notification (notification_event);
* 19:22 hashar: T137615 deployment-db2, enwiki > CREATE INDEX echo_notification_event ON echo_notification (notification_event);
* 10:37 hashar: Restarted puppetmaster on integration-puppetmaster (memory leak / can not fork: no memory)
* 10:35 hashar: T137561  salt -v '*trusty*' cmd.run "cd /root/ && dpkg -i firefox_46.0.1+build1-0ubuntu0.14.04.3_amd64.deb"
* 10:23 hashar: Hard reboot integration-slave-trusty-1015
* 08:30 hashar: Beta: `mwscript extensions/Echo/maintenance/removeInvalidTargetPage.php --wiki=enwiki` for T137615
== 2016-06-10 ==
* 15:49 jzerebecki: reloading zuul for 8c048fb..272d1ec
* 15:29 jzerebecki: T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git reset --hard 1e1ff12b13b73b5c5e2015a72f51561f10b305d0
* 15:19 jzerebecki: T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && dpkg -i firefox_46.0.1+build1-0ubuntu0.14.04.3_amd64.deb"
* 15:18 jzerebecki: T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && wget 'https://ubuntu.wikimedia.org/ubuntu/pool/main/f/firefox/firefox_46.0.1%2bbuild1-0ubuntu0.14.04.3_amd64.deb'"
* 15:15 jzerebecki: T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/39/293739/1 && git cherry-pick FETCH_HEAD
== 2016-06-09 ==
* 18:49 hashar: restarting nutcracker on deployment-mediawiki02
* 16:53 hashar: rebuild Nodepool trusty image ci-trusty-wikimedia-1465490962
* 16:37 hashar: Manually deleting old zuul references on scandium.eqiad.wmnet . Running in a screen
* 16:32 hashar: rebuild Nodepool jessie image ci-jessie-wikimedia-1465489579
* 16:03 hashar: Restarting Nodepool
== 2016-06-08 ==
* 02:56 legoktm: / on gallium is read-only
* 02:47 legoktm: disabling/enabling gearman in jenkins because everything is stuck
== 2016-06-07 ==
* 19:28 hashar: Nodepool has troubles spawning instances probably due to on going (?) labs maintenance
* 14:56 hashar: Restarting Jenkins to upgrade Rebuilder plugin with https://github.com/jenkinsci/rebuild-plugin/pull/34  (sort out parameters not being reinjected)
* 09:02 hashar: Upgrading Jenkins IRC plugin 2.25..2.27 and instant messaging plugin 1.34..1.35  . The former should fix a deadlock on shutdowning Jenkins | T96183
== 2016-06-06 ==
* 19:26 hasharAway: Regenerating Nodepool snapshots for Trusty and Jessie
* 13:04 hashar: Migrated all qunit jobs to Nodepool T136301 has the related Gerrit changes
* 10:05 hashar: migrating mediawiki-core-qunit job to Nodepool instances https://gerrit.wikimedia.org/r/#/c/291322/  T136301
== 2016-06-04 ==
* 00:09 Krinkle: krinkle@integration-slave-trusty-1017:~$ sudo rm -rf /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/Babel (T86730)
== 2016-06-03 ==
* 19:18 hashar: Image ci-jessie-wikimedia-1464981111 in wmflabs-eqiad is ready  Zend 5.x for qunit | T136301
* 15:17 hashar: refreshed Nodepool Trusty image due to some imagemagick upgrade issue.  Image ci-trusty-wikimedia-1464966671 in wmflabs-eqiad is ready
* 10:40 hashar: scandium (zuul merger):  rm -fR /srv/ssd/zuul/git/mediawiki/extensions/Collection  T136930
== 2016-06-02 ==
* 12:10 hashar: Upgraded Zuul upstream code being 66c8e52..30a433b package is  2.1.0-151-g30a433b-wmf1precise1
== 2016-06-01 ==
* 17:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/292186
* 16:45 tgr: enabling AuthManager on beta cluster
* 15:20 legoktm: deploying https://gerrit.wikimedia.org/r/292153
* 14:44 twentyafterfour: jenkins restart completed
* 14:36 twentyafterfour: restarting jenkins to install "single use slave" plugin (jenkins will restart when all builds are finished)
* 13:49 hashar: Beta : clearing temporary files under /data/project/upload7  (mainly wikimedia/commons/temp )
* 10:29 hashar: Upgraded Linux kernel on deployment-salt02  T136411
* 10:14 hashar: beta: salt-key -d deployment-salt.deployment-prep.eqiad.wmflabs  T136411
* 09:16 hashar: Enabling puppet again on Trusty slaves. Chromium is now properly pinned to version 49 ( https://gerrit.wikimedia.org/r/#/c/291116/3 | T136188 )
* 08:55 hashar: integration slaves : salt -v '*' pkg.upgrade
== 2016-05-31 ==
* 20:24 bd808: Reloading zuul to pick up I58f878f3fd19dfa21a46a52464575cb06aacbb22
== 2016-05-30 ==
* 18:39 hashar: Upgraded our Jenkins Job Builder fork to 1.5.0 + a couple of cherry picks: cd63874...10f2bcd
* 12:53 hashar: Upgrading Zuul 1cc37f7..66c8e52 T128569
* 08:04 ori: zuul is back up but jobs which were enqueued are gone
* 07:50 ori: restarting jenkins on gallium, too
* 07:49 ori: restarted zuul-merger service on gallium
* 07:44 ori: Disconnecting and then reconnecting Gearman from Jenkins did not appear to do anything; going to depool / repool nodes.
* 07:42 ori: Temporarily disconnecting Gearman from Jenkins, per <https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Known_issues>
== 2016-05-28 ==
* 04:43 ori: depooling integration-slave-trusty-1015 to profile phpunit runs
== 2016-05-27 ==
* 19:29 hasharAway: Refreshed Nodepool images
* 18:13 thcipriani: restarting zuul for deadlock
* 18:00 thcipriani: Reloading Zuul to deploy I0c3aeacf92d430ad1272f5f00e7fb7182b8a05bf
* 02:55 bd808: Deleted deployment-fluorine:/srv/mw-log/archive/*-20160[34]* logs; freed 26G
== 2016-05-26 ==
* 22:23 hashar: salt -v '*trusty*' cmd.run 'puppet agent --disable "Chromium needs to be v49. See T136188"'
* 21:47 hashar: integration-slave-trusty-1015 still on Chromium 50 .. T136188
* 21:42 hashar: downgrading chromium-browser on integration-slave-1015  T136188
* 09:24 jzerebecki: reloading zuul for d38ad0a..6798539
* 07:48 gehel: deployment-prep upgrading elasticsearch to 2.3.3 and restarting (T133124)
* 07:36 dcausse: deployment-prep elastic: updating cirrussearch warmers (T133124)
* 07:31 gehel: deployment-prep deploying new elasticsearch plugins (T133124)
== 2016-05-25 ==
* 22:38 Amir1: running puppet agent manually on sca01
* 16:26 hashar: 2016-05-25 16:24:35,491 INFO nodepool.image.build.wmflabs-eqiad.ci-trusty-wikimedia: Notice: /Stage[main]/Main/Package[ruby-jsduck]/ensure: ensure changed 'purged' to 'present'  T109005
* 15:07 hashar: g++ added to Jessie and Trusty Nodepool instances | T119143
* 14:12 hashar: Regenerating Nodepool snapshot to include g++ which is required by some NodeJS native modules T119143
* 10:58 hashar: Updating Nodepool ci-jessie-wikimedia snapshot image to get netpbm package installed into it. T126992  https://gerrit.wikimedia.org/r/290651
* 09:30 hashar: Clearing git-sync-upstream script on integration-slave-trusty1013 and integration-slave-trusty-1017. That is only supposed to be on the puppetmaster
* 09:15 hashar: Fixed resolv.conf on integration-slave-trusty-1013 and force running puppet to catch up with change since May 16 19:52
* 09:11 hashar: restarting puppetmaster on integration-puppetmaster  ( memory leak / can not fork)
== 2016-05-24 ==
* 07:03 mobrovac: rebooting deployment-tin, can't log in
== 2016-05-23 ==
* 19:35 hashar: killed all mysqld process on Trusty CI slaves
* 15:49 thcipriani: beta code update not running, disconnect-reconnect dance resulted in: [05/23/16 15:48:39] [SSH] Authentication failed.
* 14:32 jzerebecki: offlined integration-slave-trusty-1004 because it can't connect to mysql T135997
* 13:32 hashar: Upgrading Jenkins git plugins and restarting Jenkins
* 11:01 hashar: Upgrading hhvm on Trusty slaves. Bring him hhvm compiled against libicu52 instead of libicu48
* 09:12 _joe_: deployment-prep: all hhvm hosts in beta upgraded to run on the newer libicu; now running updateCollation.php (T86096)
* 09:11 hashar: Image ci-jessie-wikimedia-1463994307 in wmflabs-eqiad is ready
* 09:01 hashar: Image ci-trusty-wikimedia-1463993508 in wmflabs-eqiad is ready
* 08:56 _joe_: deployment-prep: starting upgrade of HHVM to a version linked to libicu52, T86096
* 08:54 hashar: Regenerating Nodepool image manually. Broke over the week-end due to a hhvm/libicu transition.  Should get pip 8.1.x now
== 2016-05-20 ==
* 20:30 bd808: Killing https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/43608/ which has been running for 5 hours
== 2016-05-19 ==
* 16:47 thcipriani: deployment-tin jenkins worker seems to be back online after [https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update some prodding]
* 16:41 thcipriani: beta-code-update eqiad hung for past few hours
* 15:16 hashar: Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked
* 11:59 hashar: CI: salt -v '*' cmd.run 'pip install --upgrade pip==8.1.2'
* 11:54 hashar: Upgrading pip on CI slaves from 7.0.1 to 8.1.2  https://gerrit.wikimedia.org/r/#/c/289639/
* 10:15 hashar: puppet broken on deployment-tin :    ?[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter trusted_group on node deployment-tin.deployment-prep.eqiad.wmflabs?[0m
== 2016-05-18 ==
* 13:16 Amir1: deploying a05e830 to ores nodes (sca01 and ores-web)
* 12:46 urandom: (re)cherry-picking c/284078 to deployment-prep
* 11:36 hashar: Restarted qa-morebots
* 11:36 hashar: Marked mediawiki/core/vendor repository has hidden in Gerrit. It got moved to mediawiki/vendor including the whole history Settings page: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/core/vendor
== 2016-05-13 ==
* 14:39 thcipriani: remove shadow l10nupdate user from deployment-tin and mira in beta
* 10:20 hashar: Put integration-slave-trusty-1004 offline.  Ssh/passwd is borked  T135217
* 09:59 hashar: Deleting non nodepool mediawiki PHPUnit jobs for T135001 (mediawiki-phpunit-hhvm mediawiki-phpunit-parsertests-hhvm mediawiki-phpunit-parsertests-php55 mediawiki-phpunit-php55)
* 04:06 thcipriani|afk: changed ownership of mwdeploy public keys post shadow mwdeploy user removal is important
* 03:47 thcipriani|afk: ldap failure has created a shadow mwdeploy user on beta, deleted using vipw
== 2016-05-12 ==
* 22:53 bd808: Started dead mysql on integration-slave-precise-1011
== 2016-05-11 ==
* 21:05 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/288128  #T134946
* 20:26 hashar: rebooting integration-slave-trusty-1016  is back up
* 20:15 hashar: rebooting integration-slave-trusty-1016  unreachable somehow
* 16:43 hashar: Reduced number of executors on Trusty instances from 3 to 2. Memory get exhausted causing the tmpfs to drop files and thus MW jobs to fail randomly.
* 13:33 hashar: Added contint::packages::php to Nodepool images  T119139
* 12:59 hashar: Dropping texlive and its dependencies from gallium.
* 12:52 hashar: deleted integration-dev
* 12:51 hashar: creating  integration-dev instance to hopefully have Shinken clean itself
* 11:42 hashar: rebooting deployment-aqs01 via wikitech  T134981
* 10:46 hashar: beta/ci puppetmaster : deleting old tags in /var/lib/git/operations/puppet  and repacking the repos
* 08:49 hashar: Deleting instances deployment-memc02 and deployment-memc03 (Precise instances, migrated to Jessie)  #T134974
* 08:43 hashar: Beta: switching memcached to new Jessie servers by cherry picking https://gerrit.wikimedia.org/r/#/c/288156/ and running puppet on mw app servers  #T134974
* 08:20 hashar: Creating deployment-memc04 and deployment-memc05 to switch beta cluster memcached to Jessie.  m1.medium with security policy "cache" T13497
* 01:44 matt_flaschen: Created Flow-specific External Store tables (blobs_flow1) on all wiki databases on Beta Cluster: T128417
== 2016-05-10 ==
* 19:17 hashar: beta / CI  purging old Linux kernels:  salt -v '*' cmd.run 'dpkg -l|grep ^rc|awk "{ print \$2 }"|grep linux-image|xargs dpkg --purge'
* 17:34 cscott: updated OCG to version b0c57a1c6890e9fa1f2c3743fc14cb6a7f244fc3
* 16:44 bd808: Cleaned up 8.5G of pbuilder tmp output on integration-slave-jessie-1001 with `sudo find /mnt/pbuilder/build -maxdepth 1 -type d -mtime +1 -exec rm -r {} \+`
* 16:35 bd808: https://integration.wikimedia.org/ci/job/debian-glue failure on integration-slave-jessie-1001 due to /mnt being 100$ full
* 14:20 hashar: deployment-puppetmaster mass cleaned packages/service/users etc  T134881
* 13:54 moritzm: restarted zuul-merger on scandium for openssl update
* 13:52 moritzm: restarting zuul on gallium for openssl update
* 13:51 moritzm: restarted apache and zuul-merger on gallium for openssl update
* 13:48 hashar: deployment-puppetmaster : dropping role::ci::jenkins_access role::ci::slave::labs and role::ci::slave::labs::common  T134881
* 13:46 hashar: Deleting Jenkins slave deployment-puppetmaster T134881
* 13:45 hashar: Change https://integration.wikimedia.org/ci/job/beta-build-deb/ job to use label selector "DebianGlue && DebianJessie" instead of "BetaDebianRepo"  T134881
* 13:33 hashar: Migrating all debian glue jobs to Jessie permanent slaves T95545
* 13:30 hashar: Adding  integration-slave-jessie-1002 in Jenkins.  it is all puppet compliant
* 12:59 thcipriani|afk: triggering puppet run on scap targets in beta for https://gerrit.wikimedia.org/r/#/c/287918/ cherry pick
* 09:07 hashar: fixed puppet.conf on deployment-cache-text04
== 2016-05-09 ==
* 20:58 hashar: Unbroke puppet on integration-raita.integration.eqiad.wmflabs . Puppet was blocked because role::ci::raita was no more. Fixed by rebasing https://gerrit.wikimedia.org/r/#/c/208024 T115330 
* 20:13 hashar: beta: salt -v '*' cmd.run 'dpkg --purge libganglia1 ganglia-monitor; rm -fR /etc/ganglia'  # T134808
* 20:06 hashar: CI, removing ganglia configuration entirely via:  salt -v '*' cmd.run 'rm -fRv /etc/ganglia'  # T134808
* 20:04 hashar: CI, removing ganglia configuration entirely via:  salt -v '*' cmd.run 'dpkg --purge ganglia-monitor'  # T134808
* 16:32 jzerebecki: reloading zuul for 3e2ab56..d663fd0
* 15:39 andrewbogott: migrating deployment-flourine to labvirt1009
* 15:39 hashar: Adding label contintLabsSlave  to integration-slave-jessie1001 and  integration-slave-jessie1002
* 15:26 hashar: Creating integration-slave-jessie-1001 T95545
== 2016-05-06 ==
* 19:45 urandom: Restart cassandra-metrics-collector on deployment-restbase0[1-2]
* 19:41 urandom: Rebasing 02ae1757 on deployment-puppetmaster : T126629
== 2016-05-05 ==
* 22:09 MaxSem: Promoted Yurik and Jgirault to sysops on beta enwiki. Through shell because logging in is broken for me.
== 2016-05-04 ==
* 21:28 cscott: deployed puppet FQDN domain patch for OCG: https://gerrit.wikimedia.org/r/286068 and restarted ocg on deployment-pdf0[12]
* 15:03 hashar: beta-scap: deployment-tin.deployment-prep.eqiad.wmflabs Name or service not known
* 15:03 hashar: beta-scap: deployment-tin.deployment-prep.eqiad.wmflabs
* 12:24 hashar: deleting Jenkins job mediawiki-core-phpcs  , replaced by Nodepool version mediawiki-core-phpcs-trusty  T133976
* 12:11 hashar: beta: restarted nginx on varnish caches ( systemctl restart nginx.service ) since they were not listening on port 443 #T134362
* 11:07 hashar: restarted CI puppetmaster  (out of memory leak)
* 10:57 hashar: CI: mass upgrading deb packages
* 10:53 hashar: beta: clearing out leftover apt conf that points to unreachable web proxy : salt -v '*' cmd.run "find /etc/apt -name '*-proxy' -delete"
* 10:48 hashar: Manually fixing nginx upgrade on deployment-cache-text04 and deployment-cache-upload04  see T134362 for details
* 09:27 hashar: deployment-cache-text04 systemctl stop varnish-frontend.service  . To clear out all the stuck CLOSE_WAIT connections  T134346
* 08:33 hashar: fixed puppet on deployment-cache-text04 (race condition generating puppet.conf )
== 2016-05-03 ==
* 23:21 bd808: Changed "Maximum Number of Retries" for ssh agent launch in jenkins for deployment-tin from "0" to "10"
* 23:01 twentyafterfour: rebooting deployment-tin
* 23:00 bd808: Jenkins agent on deployment-tin not spawning; investigating
* 20:02 hashar: Restarting Jenkins
* 16:49 hashar: Notice: /Stage[main]/Contint::Packages::Python/Package[pypy]/ensure: ensure changed 'purged' to 'present'  | T134235