Difference between revisions of "Release Engineering/SAL"

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159)
imported>Stashbot
(James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change T292910)
(156 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== 2021-10-22 ==
* 21:19 James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change [[phab:T292910|T292910]]
* 18:06 dancy: Updating deb packages on deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud
* 00:31 James_F: Zuul: [DiscussionTools] Add EventLogging as dependency for Phan jobs
== 2021-10-21 ==
* 18:07 dduvall: replacing runner-1001 with new instance ([[phab:T293835|T293835]])
* 09:49 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test [[phab:T292728|T292728]]
== 2021-10-19 ==
* 22:24 brennen: gitlab-test: deleted unused gitlab-runner-test
* 22:20 brennen: adding Dzahn and Aokoth to gitlab-test and gitlab-runners projects, cleaning up contractor access
* 21:20 brennen: Updating dev-images docker-pkg files on primary contint for php 8 images ([[phab:T283426|T283426]])
* 18:00 James_F: Zuul: [mediawiki/services/function-schemata] Switch from pipeline to direct CI
* 11:32 hashar: Archiving integration/gear.git , no more used after Python gear 0.16.0 has been released # [[phab:T289512|T289512]]
== 2021-10-18 ==
* 16:26 Lucas_WMDE: killed one beta-scap-sync-world job that seemed stuck for a long time
* 14:43 James_F: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [[phab:T290274|T290274]]
* 09:41 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # [[phab:T248676|T248676]]
== 2021-10-16 ==
* 16:47 Lucas_WMDE: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T293070|T293070]], based on [[phab:T271808|T271808]]#6739578
* 01:17 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/730949
== 2021-10-15 ==
* 16:14 brennen: gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for [[phab:T292372|T292372]]
== 2021-10-14 ==
* 16:13 addshore: reloaded zuul for zuul: mw-ext-MailGun will need PHP 7.3+ [[phab:T236396|T236396]]
* 14:13 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/730207
== 2021-10-13 ==
* 23:12 bd808: Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors.
* 16:48 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T293277|T293277]]
* 11:56 hashar: integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes'
== 2021-10-12 ==
* 19:11 hashar: Building Docker images for https://gerrit.wikimedia.org/r/730279
* 17:28 James_F: Zuul: [mediawiki/extensions/THICC] Mark as archived for [[phab:T292948|T292948]]
== 2021-10-11 ==
* 15:35 hashar: gerrit: marked integration/composer read-only. We are no more using it and the few use cases left are tracked at https://phabricator.wikimedia.org/T249949
* 02:08 Krinkle: Browsers fail to connect with https://upload.wikimedia.beta.wmflabs.org/ (Certificate expired two days ago)
== 2021-10-08 ==
* 21:17 hashar: Purging Docker images on all CI agents
* 20:00 hashar: Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620
* 19:23 hashar: Building CI Docker images for Quibble 1.1.1
* 19:08 hashar: Tag quibble 1.1.1 @ {{Gerrit|b54af2aa60}}
* 19:07 brennen: gitlab1001: restarting to check correct session_duration value of 604800
* 11:01 hashar: Rebuilding CI Docker images based on Bullseye to include apt security updates
== 2021-10-07 ==
* 22:12 James_F: Zuul: [design/codex] Provide initial CI, node14-browser suite
* 17:07 James_F: Zuul: [mediawiki/extensions/ParsoidBatchAPI] Mark as archived [[phab:T252744|T252744]]
* 13:29 hashar: restarting CI Jenkins for git plugin update
* 13:14 hashar: Upgraded CI Jenkins on contint2001
* 13:10 hashar: Upgraded releases Jenkins from 2.289.2 to  2.303.2
* 13:04 hashar: Updating Jenkins jobs for Quibble 1.1.0 # [[phab:T291549|T291549]]
* 04:46 Krinkle: Temporarily change `mediawiki-fresnel-patch-docker` job change back to quibble-fresnel:1.0.1 ref [[phab:T292684|T292684]]
* 04:41 Krinkle: Reconfigure `mediawiki-fresnel-patch-docker` job to revert quibble change back to quibble-fresnel:1.0.1
* 01:06 Reedy: unstick beta ci jobs
== 2021-10-06 ==
* 11:19 hashar: Tag Quibble 1.1.0 @ {{Gerrit|3dabcab}}
== 2021-10-05 ==
* 21:47 brennen: updating dev-images docker-pkg files on primary contint for https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/7
* 12:03 majavah: root@deployment-cache-text06:~# systemctl restart sssd # [[phab:T286502|T286502]]
* 11:11 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/726576
== 2021-10-04 ==
* 21:09 brennen: finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 ([[phab:T292256|T292256]])
* 21:02 brennen: gitlab-runners: upgrading each shared runner to 14.3.2
* 18:12 James_F: Docker: Publish composer 2.x-based images for [[phab:T279857|T279857]]
* 16:46 James_F: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
* 15:54 James_F: Zuul: Add Lindsay Wardell personal e-mail to CI allow list
* 14:31 brennen: gitlab1001: upgrading to 14.3.2 ([[phab:T292256|T292256]])
== 2021-09-30 ==
* 21:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/683031
== 2021-09-29 ==
* 15:52 hashar: Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # [[phab:T277500|T277500]]
* 15:03 brennen: gitlab: pausing shared runners while working through access-level implications
== 2021-09-27 ==
* 20:52 James_F: Zuul: [integration/visualdiff] Add basic node12 CI job
* 20:15 brennen: gitlab1001: login and user rename as "brennen" checks out; applying user mappings for remaining ~80 users
* 19:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/724103/3
* 19:24 hashar: Updating Jenkins jobs using a Stretch related image to latest version of the image. https://gerrit.wikimedia.org/r/c/integration/config/+/724156/ # [[phab:T291425|T291425]]
* 17:23 hashar: Building java8-scala-spark image
* 07:11 hashar: Switching python based jobs to docker-registry.wikimedia.org/releng/tox-buster:0.5.0 # [[phab:T291292|T291292]]
== 2021-09-24 ==
* 15:41 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/coal/+/722948 and latest https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep.  Should only affect deployment-webperf11.
== 2021-09-23 ==
* 19:55 brennen: gitlab-ansible-test: sudo gitlab-ctl cleanse to drop test data and reset
* 18:41 James_F: Zuul: Create 'extension-apitests' template for simplicity
* 15:54 brennen: gitlab1001: brief downtime to apply [[gerrit:714382{{!}}gitlab cas: uid instead of CN; add nickname_key]] for [[phab:T288392|T288392]]
* 15:13 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/723217
* 14:46 hashar: Building releng/tox-buster:0.5.0 image
* 11:44 hashar: Reloaded zuul for "zuul: promote shellcheck job for integration/config" https://gerrit.wikimedia.org/r/c/integration/config/+/723160/1
* 10:57 hashar: Building tox images for improved shellcheck package https://gerrit.wikimedia.org/r/721895
== 2021-09-22 ==
* 15:47 Amir1: set row format for text table in simplewiki to compressed
* 15:22 Amir1: restart mysql on db07 to add performance schema
== 2021-09-21 ==
* 16:26 dduvall: deleted wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to a branch cut job gone awry. re-running job to cut wmf/1.38.0-wmf.1 ([[phab:T281165|T281165]])
* 16:04 dduvall: deleting wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to the branch cut job gone awry
* 15:49 James_F: Re-running the branch cut job for [[phab:T281165|T281165]]
* 15:48 James_F: Updated the releases-jenkins automatic branch cut job to make 1.38.0 branches (missed ahead of [[phab:T281165|T281165]])
== 2021-09-17 ==
* 19:05 hashar: Building Docker images for [tox-buster] Install shellcheck and cascade [integration/config] - https://gerrit.wikimedia.org/r/721881
* 18:08 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 18:07 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T28477|T28477]]
* 16:19 dpifke: Enabled TLS on Jumbo Kafka instances in deployment-prep.
== 2021-09-16 ==
* 22:17 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/721567 in deployment-prep, should only affect deployment-webperf11.
* 16:51 addshore: reload zuul for Promote branchdeploy for query-builder to test https://gerrit.wikimedia.org/r/721581 (https://phabricator.wikimedia.org/T278706)
* 13:27 Reedy: Actually reloading Zuul to deploy  https://gerrit.wikimedia.org/r/683589
* 13:01 addshore: reload zuul for https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) 1 experimental branchdeploy job
== 2021-09-15 ==
* 22:59 dpifke: Un-cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep for now; more work needed.
* 22:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/721044 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep; should only affect deployment-webperf11.
* 08:58 Amir1: update wb_changes_dispatch set chd_disabled = 1 where chd_site = 'enwiki'; ([[phab:T290985|T290985]])
== 2021-09-14 ==
* 17:17 Amir1: delete from wb_changes_dispatch where chd_db = 'enwiki'; ([[phab:T290985|T290985]])
== 2021-09-13 ==
* 09:33 hashar: Castor cache: nuked files that were last changed more than six months ago to free up disk space
== 2021-09-10 ==
* 21:52 James_F: Created experimental integration-agent-docker-1021 for [[phab:T252071|T252071]]
* 21:48 James_F: Deleting CI agent integration-agent-docker-1001 for [[phab:T252071|T252071]]
* 21:44 James_F: Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for [[phab:T252071|T252071]]
* 21:23 James_F: Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job
* 17:41 James_F: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI
* 02:18 James_F: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12
* 02:13 James_F: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job
* 02:10 James_F: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12
* 01:58 James_F: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental
* 01:48 James_F: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices
== 2021-09-09 ==
* 22:20 brennen: gitlab-ansible-test: resetting instance data
* 19:42 James_F: Docker: Building node<nowiki>{</nowiki>10,12<nowiki>}</nowiki>-test-browser-php80-composer for [[phab:T290651|T290651]]
* 10:56 hashar: Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17
* 10:37 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496
== 2021-09-08 ==
* 20:16 thcipriani: self +2 on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719500 to unbreak beta
* 18:47 brennen: contint1001 / contint2001: /srv/dev-images: git remote set-url origin 'https://gitlab.wikimedia.org/releng/dev-images.git'
* 18:28 dduvall: Running ./fab deploy_docker to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719350
* 17:05 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355
* 16:37 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719493
* 14:55 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/714328/
* 14:05 brennen: runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners ([[phab:T289802|T289802]])
== 2021-09-07 ==
* 20:49 James_F: Marked https://gerrit.wikimedia.org/g/mediawiki/tools/cli as read-only and pointed users to GitLab.
* 20:46 brennen: migrating dev-images to https://gitlab.wikimedia.org/releng/dev-images and deactivating on gerrit
* 18:16 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 18:11 dduvall: creating 2 new jenkins jobs for deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 16:25 James_F: Docker: Publishing node12-test-browser-php<nowiki>{</nowiki>72,80<nowiki>}</nowiki>-composer images
* 16:19 James_F: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add 4 dependencies
== 2021-09-03 ==
* 23:02 Krinkle: Creating integration-agent-qemu-1002 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 17:42 dduvall: deploying blubberoid:2021-09-03-160524-production to eqiad/codfw (https://gerrit.wikimedia.org/r/c/blubber/+/716519) ([[phab:T289367|T289367]])
* 17:36 dduvall: staging blubberoid to deploy https://gerrit.wikimedia.org/r/c/blubber/+/716519
== 2021-09-02 ==
* 15:17 brennen: gitlab-test: testing upgrade path to 14.x
== 2021-09-01 ==
* 21:22 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/716041 in beta.
* 16:55 urbanecm: deployment-prep: Unlock scap
* 15:50 urbanecm: deployment-prep: Lock scap again
* 15:40 urbanecm: deployment-prep: Lock scap to be able to test something
* 14:08 urbanecm: deployment-prep: Create foundationwiki ([[phab:T290164|T290164]])
* 14:07 urbanecm: urbanecm@deployment-mediawiki11:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]
* 13:58 urbanecm: urbanecm@deployment-cache-text06:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]
== 2021-08-31 ==
* 21:38 dduvall: deploying new blubberoid to eqiad/codfw following successful testing in staging
* 21:35 dduvall: staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276
* 14:29 hashar: Restarting CI Jenkins for plugins upgrade
== 2021-08-30 ==
* 19:53 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta
* 19:27 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 19:16 brennen: gitlab-test: powering off gitlab (former main test instance)
* 19:11 James_F: Zuul: [mediawiki/services/apple-search] Add postmerge publish
* 19:10 brennen: gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test
* 18:23 brennen: gitlab-test: associating floating IP to primary test box
* 16:36 James_F: Zuul: [mediawiki/services/apple-search] Remove composer-package
* 15:16 James_F: Zuul: [mediawiki/services/apple-search] Add pipeline CI for [[phab:T289224|T289224]]
* 12:20 Amir1: foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5
== 2021-08-27 ==
* 19:24 James_F: Docker: Publish initial node14 CI images for [[phab:T267888|T267888]]
== 2021-08-25 ==
* 17:48 twentyafterfour: updating zuul config after deploying https://gerrit.wikimedia.org/r/714538
== 2021-08-24 ==
* 23:15 James_F: Zuul: Configure the REL1_37 test and gate pipelines [[phab:T289587|T289587]]
* 22:18 thcipriani: phab1001:sudo /srv/phab/phabricator/bin/bulk make-silent --id 2822 (releng-logspam -> unstewarded production error)
* 20:02 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/712990
== 2021-08-23 ==
* 20:37 James_F: Docker: Publish php-ast with 1.0.14 ([[phab:T289429|T289429]]) and no longer support PHP 7.0 or 7.1 (last trace!)
== 2021-08-21 ==
* 15:03 majavah: fixing deployment-prep puppet merge conflicts re: swift
== 2021-08-20 ==
* 21:11 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging/private$ rm mwblocker.log # remove weird blank log file
* 18:56 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/713439
* 18:55 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 18:54 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do echo "test $i" {{!}} mwscript edit.php --wiki=<nowiki>{</nowiki>cswiki,enwiki<nowiki>}</nowiki> --user="Martin Urbanec (test $i)" --summary="test" Sandbox; done
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=enwiki
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 18:46 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki "Martin Urbanec (test $i)"; done
* 18:40 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript createAndPromote.php --wiki=cswiki "Martin Urbanec (test $i)" "$password"; done # to test a feature that needs a lot of different accounts
* 16:30 majavah: restart sssd on deployment-cache-text06, [[phab:T286502|T286502]]?
* 16:24 majavah: deployment-prep: configure wikifunctions.beta.wmflabs.org dns zones and add to acme-chief [[phab:T284162|T284162]]
== 2021-08-19 ==
* 22:58 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/713612/
== 2021-08-17 ==
* 21:23 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:709831{{!}}add buster-apache2, log errors to stdio]] ([[phab:T283416|T283416]])
== 2021-08-16 ==
* 23:08 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/711731
== 2021-08-15 ==
* 17:44 James_F: Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI
== 2021-08-13 ==
* 20:09 urbanecm: Manually start `beta-update-databases-eqiad` CI job
* 20:06 urbanecm: deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py
* 20:03 urbanecm: Kill beta-scap-sync-world job for the usual reason
* 13:13 majavah: `mwscript extensions/CentralAuth/maintenance/importMissingLocalNames.php --wiki metawiki` on the beta cluster
== 2021-08-11 ==
* 00:52 James_F: Zuul: Add Aca to the CI allow list
* 00:52 James_F: Zuul: [mediawiki/extensions/SimpleCalendar] Add basic quibble CI
== 2021-08-10 ==
* 16:57 James_F: Zuul: Update e-mail address for Zabe in the allow list
* 16:06 James_F: Ran `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Wikibase` on doc1001 for [[phab:T288396|T288396]]
== 2021-08-09 ==
* 21:02 urbanecm: Remove hanging beta-scap-sync-world job in CI to unblock beta auto-updates
== 2021-08-07 ==
* 00:48 James_F: Docker: Publish quibble-buster-php73-coverage 1.1.1 for [[phab:T287918|T287918]].
* 00:29 James_F: Zuul: Add skin-coverage jobs to all Wikimedia production skins [[phab:T287918|T287918]]
* 00:27 James_F: Zuul: Provide a skin-coverage template [[phab:T287918|T287918]]
== 2021-08-06 ==
* 23:53 James_F: Docker: Publishing quibble-buster-php73-coverage 1.1.0 for [[phab:T287918|T287918]]
* 23:47 James_F: Zuul: [mediawiki/skins/Mirage] Not a production skin; move to right section
== 2021-08-05 ==
* 22:26 brennen: gitlab: setting CI access on all repos to "Project members only", per https://www.mediawiki.org/wiki/GitLab/Policy#Permissions - may need revisited depending on effects
* 22:04 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/710108
* 15:10 hashar: integration: sudo cumin --force 'name:docker' 'docker container prune -f && docker image prune -f'
* 15:09 hashar: integration-agent-docker-1020: docker container prune && docker image prune
* 15:02 hashar: Building Docker image helm-linter for https://gerrit.wikimedia.org/r/c/integration/config/+/710276
== 2021-08-04 ==
* 22:53 bd808: Updated composer-github-oauthtoken in Jenkins config to us a newer personal access token from GitHub per notices about https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
* 17:49 brennen: gitlab-test: testing upgrade to 13.12.9
* 16:41 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/710066 # [[phab:T288111|T288111]]
* 15:13 thcipriani: puppet fixed on deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki>
* 15:08 thcipriani: rebase deployment-puppetmaster04:labs/private causing deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki> failure for...¯\_(ツ)_/¯
== 2021-08-03 ==
* 21:12 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:698880{{!}}add buster php images]]
== 2021-08-02 ==
* 23:27 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/709563
* 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709562
* 23:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709558
* 22:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709555 https://gerrit.wikimedia.org/r/709556
* 21:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709543
== 2021-07-30 ==
* 21:27 dduvall: "Total reclaimed space: 141.4GB" on releases1002 following docker prune
* 21:24 dduvall: running `docker system prune -af` on releases1002
== 2021-07-29 ==
* 22:12 jeena: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/708596
* 14:06 brennen: gerrit: added ldap/ops to ownership of operations/gitlab-ansible
* 13:21 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/708764 "Add michaelcochez to whitelist users"
== 2021-07-28 ==
* 17:53 andrewbogott: rebooting deployment-logstash03 as it's in an inconsistent config state
* 14:44 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/708536
== 2021-07-27 ==
* 16:55 dduvall: creating new gitlab runner instance runner-1002 for testing
* 16:32 hashar: cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches
== 2021-07-26 ==
* 20:45 brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation ([[phab:T287279|T287279]])
* 18:10 dduvall: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 18:00 dduvall: creating 2 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 08:24 hashar: jjb: update jobs to Quibble 1.0.1  # https://gerrit.wikimedia.org/r/c/integration/config/+/708038
== 2021-07-23 ==
* 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner ([[phab:T287279|T287279]])
* 14:08 hashar: Building Docker images for quibble 1.0.1
* 13:45 hashar: Tag Quibble 1.0.1 @ {{Gerrit|5a2548699a}} # [[phab:T287001|T287001]]
== 2021-07-21 ==
* 21:06 brennen: gitlab1001: running ansible for logging typo fix ([[phab:T274462|T274462]])
* 20:36 dancy: Newest scap deployed to beta cluster
* 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes ([[phab:T274462|T274462]], [[phab:T275170|T275170]])
* 16:46 dancy: restarting Gerrit to fix plugins
* 16:07 dancy: Updating plugins on releases-jenkins
* 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki
== 2021-07-20 ==
* 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar ([[phab:T281156|T281156]])
* 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47  # [[phab:T287001|T287001]]
== 2021-07-19 ==
* 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values
== 2021-07-18 ==
* 08:48 majavah: set shared_acme_certificates: <nowiki>{</nowiki><nowiki>}</nowiki> on deployment-prep shared hiera, [[phab:T276653|T276653]]
== 2021-07-16 ==
* 21:01 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/698774
* 17:44 brennen: Try #2: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 17:12 hashar: Tag Quibble 1.0.0 @ {{Gerrit|a13d133f7d1}} # [[phab:T286187|T286187]] [[phab:T280506|T280506]] [[phab:T90875|T90875]] [[phab:T218534|T218534]] [[phab:T227352|T227352]]
== 2021-07-14 ==
* 22:47 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 21:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in beta puppet again, should only affect deployment-webperf12.
== 2021-07-13 ==
* 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep.  Should only affect deployment-webperf12.
* 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for [[phab:T281430|T281430]]
* 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived [[phab:T281430|T281430]]
* 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet.  Should only affect deployment-webperf12.
* 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles  # [[phab:T278990|T278990]]
* 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
* 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # [[phab:T278990|T278990]]
* 07:46 hashar: Wiping all Docker images from contint2001l
== 2021-07-12 ==
* 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, [[phab:T286506|T286506]]
* 18:43 majavah: deployment-cache-text06 varnish not starting, [[phab:T286506|T286506]], causing an outage on text traffic on deployment-prep
* 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
* 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
* 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' ([[phab:T286479|T286479]])
* 14:44 majavah: fix merge conflict on deployment-puppetmaster04
* 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
* 13:19 James_F: Zuul: Add R4356th to the CI allow list
* 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [[phab:T286207|T286207]]
== 2021-07-09 ==
* 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days ([[phab:T286375|T286375]])
* 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days ([[phab:T286358|T286358]])
== 2021-07-08 ==
* 08:21 majavah: kick stuck puppet agent on deployment-logstash04
== 2021-07-07 ==
* 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance [[phab:T286298|T286298]]
== 2021-07-06 ==
* 01:16 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/703219
== 2021-07-05 ==
* 15:02 Amir1: deployed 703212 ([[phab:T286058|T286058]])
== 2021-07-02 ==
* 17:32 brennen: gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/701068 ([[phab:T274463|T274463]])
* 09:49 James_F: Zuul: [pywikibot/core] Add deeptest to gate-and-submit section
== 2021-06-30 ==
* 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
* 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
* 17:57 thcipriani: restart ci jenkins following upgrade
* 17:54 thcipriani: restart releases-jenkins following upgrade
* 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31
== 2021-06-29 ==
* 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # [[phab:T285811|T285811]]
== 2021-06-28 ==
* 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
* 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for [[phab:T285125|T285125]]
== 2021-06-24 ==
* 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)
== 2021-06-23 ==
* 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick
== 2021-06-22 ==
* 23:03 jeena: increasing releases-jenkins executors from 2 to 4
* 21:25 jeena: Updating dev-images docker-pkg files on primary contint [[phab:T273682|T273682]]
* 09:31 hashar: Removed DISPLAY=:94 being set from Zuul for most jobs https://gerrit.wikimedia.org/r/c/integration/config/+/700857/
== 2021-06-21 ==
* 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.
== 2021-06-19 ==
* 13:44 majavah: remove deployment-deploy02 [[phab:T278689|T278689]]
* 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host [[phab:T283013|T283013]]
== 2021-06-16 ==
* 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException [[phab:T284732|T284732]]
* 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert
== 2021-06-14 ==
* 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
* 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
* 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge [[phab:T236915|T236915]]
* 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing [[phab:T236915|T236915]]
* 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
* 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
* 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli
== 2021-06-13 ==
* 18:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699567
* 18:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699541
== 2021-06-11 ==
* 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
* 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
* 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497  # [[phab:T73480|T73480]]
* 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
* 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # [[phab:T100837|T100837]]
* 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for [[phab:T271011|T271011]].  Comment left on gerrit patch and task,  not reverting patch in master at this stage
== 2021-06-10 ==
* 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too [[phab:T281195|T281195]]
* 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency [[phab:T281195|T281195]]
== 2021-06-08 ==
* 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
* 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
* 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
* 17:40 brennen: gitlab1001: running k6 data generator
* 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 [[phab:T280491|T280491]]
== 2021-06-07 ==
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # [[phab:T263617|T263617]]
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #[[phab:T263617|T263617]]
* 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for [[phab:T284492|T284492]]
* 11:51 hashar: zuul enqueue  --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1  # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470
== 2021-06-05 ==
* 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium
== 2021-06-04 ==
* 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
* 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
* 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 [[phab:T284345|T284345]]
* 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [[phab:T284345|T284345]]
* 19:20 James_F: Docker: Publishing node12 CI images [[phab:T284343|T284343]]
== 2021-06-03 ==
* 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/*  # [[phab:T284125|T284125]]
* 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
* 00:24 James_F: Zuul: Add Anysite to CI allowlist
* 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline [[phab:T281132|T281132]]
== 2021-06-02 ==
* 19:21 urbanecm: deployment-prep: Enlarge mwlog's /srv partition to 5 GB (was 2 GB)
* 13:49 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project search/MjoLniR --change 685569,4  #  Rerun sonar analysis against https://gerrit.wikimedia.org/r/c/search/MjoLniR/+/685569
== 2021-05-28 ==
* 09:41 addshore: reload zuul for https://gerrit.wikimedia.org/r/697033
* 08:02 hashar: Successfully published image docker-registry.discovery.wmnet/releng/ci-bullseye:0.1.0 # [[phab:T283777|T283777]]
== 2021-05-27 ==
* 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
* 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
* 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
* 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
* 16:05 brennen: gitlab1001: re-running ansible and puppet per [[phab:T279545|T279545]]
* 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
* 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs
== 2021-05-26 ==
* 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` ([[phab:T279545|T279545]])
* 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org
== 2021-05-24 ==
* 06:19 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693639
== 2021-05-21 ==
* 18:30 James_F: Zuul: Add phan to all extensions and skins [[phab:T283097|T283097]]
* 17:59 James_F: Docker: Publishing mediawiki-phan* images where the job passes if .phan/config.php is absent [[phab:T283097|T283097]]
* 08:30 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693361 https://gerrit.wikimedia.org/r/691323
== 2021-05-20 ==
* 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
* 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
* 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
* 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
* 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs [[phab:T280669|T280669]]
== 2021-05-19 ==
* 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs [[phab:T280669|T280669]]
* 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps [[phab:T279275|T279275]]
* 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users [[phab:T279552|T279552]]
== 2021-05-17 ==
* 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive [[phab:T279221|T279221]]
== 2021-05-16 ==
* 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
* 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04
== 2021-05-15 ==
* 21:51 James_F: Zuul: [mediawiki/tools/cli] Make mw-cli-test experimental for now [[phab:T248779|T248779]]
* 21:40 James_F: Zuul: [mediawiki/tools/cli] Add new bespoke job [[phab:T248779|T248779]]
* 21:09 James_F: Zuul: [mediawiki/extensions/MediaWikiAuth] Mark repo as archived [[phab:T282955|T282955]]
* 18:38 James_F: Zuul: Temporarily remove TwoColConflict from gated extensions [[phab:T234002|T234002]] [[phab:T282935|T282935]].
* 09:30 Majavah: create deployment-logstash04 to install elk7
* 09:22 Majavah: beta: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315, remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/683837 [[phab:T277990|T277990]]
* 06:23 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/691494/ [[phab:T281986|T281986]]
== 2021-05-14 ==
* 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php),  [[phab:T198673|T198673]]
* 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, [[phab:T198673|T198673]],  https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
* 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, [[phab:T198673|T198673]], https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
* 20:51 Krinkle: I broke beta  `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
* 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
* 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
* 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job [[phab:T174018|T174018]] [[phab:T177260|T177260]] [[phab:T280901|T280901]]
* 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI [[phab:T281538|T281538]]
== 2021-05-13 ==
* 23:14 Krinkle: root@integration-agent-qemu-1001:/home/addshore# rm qmeutest/out.img (Reclaiming space to be able to make other changes)
* 22:48 longma: restarting zuul to try getting tests to run for https://gerrit.wikimedia.org/r/c/mediawiki/skins/Modern/+/684509
== 2021-05-12 ==
* 19:40 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/439774 in deployment-prep puppet.  Hasn't been touched in over a year and is preventing merges.
* 19:32 James_F: Zuul: Add PageImages, PageViewInfo, and Graph to MediaWiki gated extension set [[phab:T249674|T249674]]
* 19:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/683695 in deployment-prep puppet.  Should only affect beta logstash.
* 16:15 James_F: Docker: Publishing civicrm 0.2.1 for [[phab:T277500|T277500]]
* 16:00 James_F: Zuul: Add GrowthExperiments to the MediaWiki gated extension set [[phab:T247507|T247507]] [[phab:T249674|T249674]]
* 15:36 _joe_: updating helm-linter docker image
== 2021-05-11 ==
* 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set [[phab:T237538|T237538]] [[phab:T249674|T249674]]
* 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI
== 2021-05-10 ==
* 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
* 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default ([[phab:T271206|T271206]])
* 14:04 CFisch_WMDE: Forward renamed config name for improved template search features ([[phab:T277028|T277028]])
== 2021-05-07 ==
* 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue [[phab:T282018|T282018]]
== 2021-05-06 ==
* 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds [[phab:T264877|T264877]]
* 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
* 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job [[phab:T271562|T271562]]
== 2021-05-05 ==
* 23:52 James_F: Zuul: Ensure Parsoid's tests include the Disambiguator extension (take 2) [[phab:T271863|T271863]] [[phab:T237538|T237538]]
* 20:38 James_F: Docker: Publishing quibble-buster* 0.0.47 images
* 19:58 James_F: Docker: Publishing quibble-stretch* 0.0.47-s1 images, now with node
* 18:23 James_F: Docker: Publishing quibble-stretch* 0.0.47 images
* 16:54 hashar: Tag Quibble 0.0.47 @ {{Gerrit|8b200cfb0}} # [[phab:T271863|T271863]] [[phab:T199403|T199403]] [[phab:T281607|T281607]]
* 14:17 CFisch_WMDE: Disable ReferencePreviews beta mode on beta labs ([[phab:T271206|T271206]])
* 13:22 Majavah: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/684034 https://gerrit.wikimedia.org/r/c/operations/puppet/+/684088 - might cause beta to go read only for a bit - [[phab:T110115|T110115]]
== 2021-05-04 ==
* 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
* 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02
== 2021-05-03 ==
* 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [[phab:T281256|T281256]]
* 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
* 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
* 13:55 CFisch_WMDE: enable new search features for the template dialog ([[phab:T271802|T271802]])
== 2021-05-02 ==
* 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
* 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
* 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref [[phab:T198673|T198673]]
* 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
* 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117
== 2021-05-01 ==
* 19:19 James_F: Zuul: Add atagar to the CI allow list
* 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - [[phab:T278641|T278641]]
* 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004
== 2021-04-30 ==
* 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
* 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for [[phab:T257301|T257301]]
* 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
* 14:15 Majavah: revert above as it's not working, [[phab:T206158|T206158]]
* 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 [[phab:T206158|T206158]]
* 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
* 02:37 James_F: Docker: Publishing node10 images based on buster [[phab:T278203|T278203]] [[phab:T240955|T240955]]
== 2021-04-29 ==
* 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist [[phab:T281418|T281418]]
== 2021-04-28 ==
* 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs [[phab:T279180|T279180]]
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # [[phab:T249268|T249268]]
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
* 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip <nowiki>{</nowiki><nowiki>}</nowiki> \+  # [[phab:T249268|T249268]]
* 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.
== 2021-04-27 ==
* 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for [[phab:T277742|T277742]].
* 17:52 Majavah: delete deployment-sessionstore03 [[phab:T263617|T263617]] [[phab:T278641|T278641]]
* 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for [[phab:T281283|T281283]]
* 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/680476
* 12:44 hashar: Restarted CI Jenkins for plugins upgrade
* 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2  (with ldap plugin 1.26)
* 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
* 06:40 Majavah: installing deployment-sessionstore04 [[phab:T263617|T263617]]
* 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs [[phab:T281198|T281198]]
== 2021-04-26 ==
* 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list
== 2021-04-24 ==
* 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job
== 2021-04-23 ==
* 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
* 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
* 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts
== 2021-04-22 ==
* 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
* 02:53 Reedy: killed a few stuck beta ci jobs
* 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
* 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 [[phab:T280874|T280874]]
* 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental [[phab:T280874|T280874]]
* 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos [[phab:T280874|T280874]]
== 2021-04-21 ==
* 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
* 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs [[phab:T280874|T280874]]
* 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
* 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies
== 2021-04-20 ==
* 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog ([[phab:T273425|T273425]])
* 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor ([[phab:T271825|T271825]])
== 2021-04-19 ==
* 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental [[phab:T280491|T280491]]
* 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
* 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for [[phab:T280491|T280491]]
== 2021-04-17 ==
* 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for [[phab:T280420|T280420]]
== 2021-04-16 ==
* 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks [[phab:T234020|T234020]] [[phab:T280167|T280167]]
* 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
* 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch [[phab:T278203|T278203]]
* 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs [[phab:T252432|T252432]]
* 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref [[phab:T235142|T235142]]
* 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref [[phab:T198673|T198673]]
* 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref [[phab:T198673|T198673]]
* 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
* 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
* 17:02 dancy: Updating dev-images docker-pkg files on primary contint
* 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template [[phab:T252434|T252434]]
== 2021-04-15 ==
* 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
* 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts
== 2021-04-14 ==
* 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov [[phab:T234020|T234020]]
== 2021-04-13 ==
* 17:00 halfak: failed deploy to ORES (connection to host failed)
* 16:57 halfak: deploying ores {{Gerrit|f08a3cb}}
* 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
* 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 [[phab:T280004|T280004]]
* 07:46 awight: enable syntax highlighting line numbering on all namespaces ([[phab:T267911|T267911]])
== 2021-04-12 ==
* 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; [[phab:T279853|T279853]])
* 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # [[phab:T279853|T279853]]
* 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per [[phab:T106915|T106915]]#6279270 - [[phab:T135427|T135427]]
* 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
* 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)
== 2021-04-11 ==
* 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
* 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov [[phab:T234020|T234020]]
* 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3<nowiki>{</nowiki>5,6<nowiki>}</nowiki> [[phab:T274971|T274971]]
* 00:02 James_F: Docker: Publishing quibble-buster images [[phab:T252432|T252432]]
== 2021-04-10 ==
* 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list
== 2021-04-09 ==
* 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, [[phab:T233134|T233134]].
* 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
* 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 [[phab:T277750|T277750]] phan for wikibase release branches
* 09:51 Majavah: deleting deployment-jobrunner03 [[phab:T278664|T278664]]
* 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage [[phab:T234080|T234080]]
== 2021-04-08 ==
* 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
* 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563
== 2021-04-07 ==
* 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
* 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
* 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
* 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 [[phab:T278664|T278664]]
* 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
* 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382
== 2021-04-06 ==
* 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived [[phab:T277626|T277626]]
* 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs [[phab:T279260|T279260]]
* 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines [[phab:T279459|T279459]]
* 17:58 James_F: Zuul: [mediawiki/services/function-<nowiki>{</nowiki>orchestr,evalu<nowiki>}</nowiki>ator] Publish images
== 2021-04-05 ==
* 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
* 17:45 brennen: halting gitlab-test for resize
== 2021-04-02 ==
* 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
* 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12
== 2021-04-01 ==
* 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
* 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
* 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
* 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298
== 2021-03-30 ==
* 18:05 Majavah: remove <nowiki>{</nowiki>trysty,precise<nowiki>}</nowiki>-deployment-prep repos from deployment-deploy01 aptly
* 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
* 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those [[phab:T278689|T278689]]
* 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
* 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
* 13:35 Majavah: create and install deployment-deploy03 [[phab:T278689|T278689]]
* 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 07:26 Majavah: shutoff deployment-mediawiki-09 [[phab:T278664|T278664]]
* 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
* 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11
== 2021-03-29 ==
* 15:37 Majavah: hard reboot deployment-sessionstore03 [[phab:T263617|T263617]]
* 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) [[phab:T263617|T263617]]
* 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 ([[phab:T278664|T278664]]), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
* 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
* 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
* 09:29 Urbanecm: Manually run puppet on mediawiki10
* 09:28 Urbanecm: Re-enable puppet on mediawiki10
* 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
* 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - [[phab:T278664|T278664]]
* 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, [[phab:T278664|T278664]]
== 2021-03-26 ==
* 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
* 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 ([[phab:T218729|T218729]])
== 2021-03-25 ==
* 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
* 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
* 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749
== 2021-03-24 ==
* 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm [[phab:T238707|T238707]]
== 2021-03-23 ==
* 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per [[phab:T274015|T274015]]
* 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job [[phab:T207226|T207226]]
* 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
* 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade [[phab:T252434|T252434]]
* 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template [[phab:T252434|T252434]]
* 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use
== 2021-03-22 ==
* 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
* 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
* 12:07 Majavah: delete deployment-restbase[01-02], [[phab:T250574|T250574]]
* 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T276624|T276624]])
* 11:33 dcaro: Created subzone beta.wmcloud.org ([[phab:T276624|T276624]])
== 2021-03-19 ==
* 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
* 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
* 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, [[phab:T218729|T218729]]
* 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
* 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down [[phab:T250574|T250574]]
* 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for [[phab:T250574|T250574]]
== 2021-03-18 ==
* 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw ([[phab:T277109|T277109]])
* 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
* 18:27 brennen: updating gitlab-test to 13.9.4-ce
* 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 [[phab:T277750|T277750]]
* 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
* 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # [[phab:T269354|T269354]]
* 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 [[phab:T277750|T277750]] (apitests php versions)
* 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 [[phab:T277060|T277060]]
* 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11  # [[phab:T269354|T269354]]
* 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for [[phab:T276179|T276179]]
* 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
* 08:44 Majavah: delete now unused deployment-ircd [[phab:T277081|T277081]]
* 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now
== 2021-03-17 ==
* 20:30 hashar: Reloaded Zuul for {{Gerrit|I2368478e4c4ab8752581f55a7c5ab493fafdeb41}}
* 15:37 Majavah: shutdown deployment-restbase01 for [[phab:T250574|T250574]]
* 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # [[phab:T250574|T250574]]
* 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
* 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI
== 2021-03-16 ==
* 21:37 longma: Updating dev-images docker-pkg files on primary contint
* 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
* 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
* 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
* 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
* 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1  # [[phab:T277527|T277527]]
== 2021-03-15 ==
* 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
* 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118
== 2021-03-13 ==
* 17:10 twentyafterfour: restart apache on gerrit1001
== 2021-03-12 ==
* 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
* 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
* 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
* 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
* 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ [[phab:T276428|T276428]]
== 2021-03-11 ==
* 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job([[phab:T277236|T277236]])
* 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 [[phab:T276419|T276419]]
* 16:49 Majavah: delete deployment-etcd-01 [[phab:T276462|T276462]]
* 13:51 Majavah: shut down deployment-db06, now unused [[phab:T277070|T277070]]
* 13:48 Majavah: set deployment-db07 as r/w [[phab:T277070|T277070]]
* 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 [[phab:T277070|T277070]]
* 13:41 Majavah: stop slave on deployment-db06 [[phab:T276968|T276968]]
* 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 [[phab:T277070|T277070]]
* 13:34 Majavah: stop and reset slave on deployment-db07 [[phab:T277070|T277070]]
* 13:32 Majavah: set deployment-db06 as read only [[phab:T277070|T277070]]
* 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/
== 2021-03-10 ==
* 19:36 Majavah: shutdown deployment-ircd [[phab:T277081|T277081]]
* 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 [[phab:T277081|T277081]]
* 18:05 Majavah: create deployment-ircd02 for [[phab:T277081|T277081]]
* 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
* 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
* 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
* 17:09 Majavah: set beta cluster mediawiki as read write on mw config ([[phab:T276968|T276968]])
* 17:03 Majavah: make deployment-db06 read-write [[phab:T276968|T276968]]
* 16:50 Majavah: `reset slave;` on new master deployment-db06 [[phab:T276968|T276968]]
* 16:49 Majavah: add deployment-db07 as a replica of db06 for [[phab:T276968|T276968]]
* 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # [[phab:T276968|T276968]]
* 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; ([[phab:T276968|T276968]])
* 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . ([[phab:T276968|T276968]])
* 15:57 Majavah: set deployment-db06 as readonly from mysql side [[phab:T276968|T276968]]
* 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` ([[phab:T276968|T276968]])
* 15:54 Urbanecm: Start mariadb on db08 ([[phab:T276968|T276968]])
* 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 ([[phab:T276968|T276968]])
* 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
* 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
* 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade ([[phab:T276968|T276968]])
* 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) ([[phab:T276968|T276968]])
== 2021-03-09 ==
* 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
* 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user ([[phab:T276968|T276968]])
* 20:33 Majavah: install mariadb on deployment-db08 [[phab:T276968|T276968]]
* 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db ([[phab:T276968|T276968]])
* 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica ([[phab:T276968|T276968]])
* 19:50 marxarelli: restoring database dump on deployment-db07 ([[phab:T276968|T276968]])
* 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` ([[phab:T276968|T276968]])
* 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 [[phab:T276968|T276968]]
* 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 ([[phab:T276968|T276968]])
* 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster [[phab:T276968|T276968]]
* 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump ([[phab:T276968|T276968]])
* 18:09 Majavah: set deployment-db05 to read-only to avoid issues with [[phab:T276968|T276968]]
* 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances ([[phab:T276968|T276968]])
* 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
* 17:22 marxarelli: restarting deployment-db05 via horizon
* 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
* 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
* 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
* 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
* 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
* 13:32 arturo: hard-reboot deployment-db05 because issues related to [[phab:T276922|T276922]]
* 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other
== 2021-03-08 ==
== 2021-03-08 ==
* 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
* 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
Line 255: Line 1,189:
* 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master [[phab:T270976|T270976]]
* 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master [[phab:T270976|T270976]]
* 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting [[phab:T251967|T251967]]
* 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting [[phab:T251967|T251967]]
== 2020-12-31 ==
* 01:47 James_F: Docker: Building and publishing tox-homer:0.1.1
* 01:36 James_F: Docker: Building and publishing tox-eventlogging:0.4.1
* 01:19 James_F: Docker: Building and publishing tox-conftool:0.4.1
* 01:04 James_F: Docker: Building and publishing tox-censorshipmonitoring:0.1.1
== 2020-12-23 ==
* 20:40 legoktm: deploying https://gerrit.wikimedia.org/r/651819
* 19:48 James_F: Zuul: [mediawiki/tools/dependency-analysis] Add composer test CI
* 19:03 balloons: resized deployment-puppetdb03 to g2.cores2.ram4.disk40 ([[phab:T270420|T270420]])
== 2020-12-22 ==
* 22:40 James_F: Zuul: [integration/docroot] Only test PHP 7.3+ from now on
* 21:16 James_F: Docker: Building and publoshing tox-labs-striker:0.5.0
== 2020-12-21 ==
* 23:14 James_F: Docker: Building and publishing tox-pywikibot 0.5.0 [[phab:T270389|T270389]]
* 22:54 James_F: Docker: Building and publishing tox-buster
* 16:55 James_F: Zuul: Configure rust-test for integration/utils-rs repository
* 16:29 James_F: Zuul: No longer set PHP_BIN parameter [[phab:T225451|T225451]]
== 2020-12-19 ==
* 18:07 James_F: Zuul: [mediawiki/libs/LangConv] Enable node10 testing
* 17:59 James_F: Zuul: Enable php80 tests for IPSet, IPUtils, LangConv
* 17:03 James_F: Zuul: Re-add Echo as dependency for AbuseFilter
* 08:46 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/650675
== 2020-12-18 ==
* 14:52 hashar: Updating https://integration.wikimedia.org/ci/job/wikidata-query-flink-swift-plugin-maven-java8-docker-site-publish/  to run sonar with Java 11 (build remains on Java 8)  [[phab:T264873|T264873]]
* 07:37 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/649745
== 2020-12-17 ==
* 23:41 James_F: Zuul: Add Gerrit maintenance bot to whitelist [[phab:T253439|T253439]]
* 20:59 James_F: Zuul: [wikipeg] Run the composer-test-package suite [[phab:T269720|T269720]]
* 20:34 James_F: Zuul: Add experimental PHP 8.0 jobs for OOUI and Wikipeg
* 19:17 James_F: Zuul: [mediawiki/extensions/WikibaseStatementUpdater] Switch quibble to composer [[phab:T270369|T270369]]
* 18:51 James_F: Zuul: [mediawiki/extensions/WikibaseStatementUpdater] Install CI [[phab:T270369|T270369]]
* 18:32 dpifke: Restart puppetdb on deployment-puppetdb03 ([[phab:T248041|T248041]])
* 18:18 dpifke: Doing the same in beta puppet-private.
* 18:11 dpifke: Fixing some merge conflicts in beta puppet (so that upstream can sync again).
* 03:43 brennen: Updating dev-images docker-pkg files on primary contint for fundraising buster base and php7.2-dba inclusion
== 2020-12-16 ==
* 23:48 James_F: Zuul: Switch OOUI and Wikipeg over to versioned special jobs
* 23:40 James_F: Docker: Publishing node10-test-browser-php80-composer 0.0.1
* 22:00 mutante: adjusted 'puppet prefix' deployment-jobrunner to use "role::beta::mediawiki::jobrunner" instead of "role::mediawiki::jobrunner" - goes together with gerrit:649707 - no instance currently exists called 'deployment-jobrunner'
== 2020-12-15 ==
* 18:05 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/649441
* 18:01 marxarelli: deploying 2 new jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/649441
* 16:16 James_F: Zuul: Install CI for labs/tools/extjsonuploader [[phab:T250344|T250344]]
* 16:13 James_F: Zuul: Install CI for mediawiki/libs/RequestTimeout [[phab:T269326|T269326]]
* 12:25 Urbanecm: Create beta bnwiki ([[phab:T270165|T270165]])
== 2020-12-14 ==
* 19:46 hashar: integration: deleting faulty npm cache integration-castor03  /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node10-docker  # [[phab:T270100|T270100]]
* 08:34 hashar: deployment-prep restart puppetdb process on deployment-puppetdb03 # [[phab:T248041|T248041]]
== 2020-12-11 ==
* 21:52 marxarelli: rolling back blubberoid:2020-12-11-212149-production
* 21:32 marxarelli: deploying blubberoid:2020-12-11-212149-production (refs https://gerrit.wikimedia.org/r/c/blubber/+/647120 and [[phab:T263597|T263597]])
== 2020-12-09 ==
* 10:28 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/quibble-apache:0.0.1  [[phab:T225218|T225218]]
== 2020-12-08 ==
* 23:05 James_F: Zuul: [labs/tools/stewardbots] Disable PHP 8.0 tests, not passing
* 18:42 dpifke: Cleaning up some of my cherry-picks in beta puppet.
* 17:47 James_F: Zuul: [mediawiki/extensions/WatchSubpages] Run phan
* 02:43 James_F: Zuul: [mediawiki/php/wikidiff2] Make PHP 8.0 tests voting [[phab:T269052|T269052]]
== 2020-12-07 ==
* 15:53 James_F: Zuul: [mediawiki/extensions/LDAPAuthorization] Disable selenium tests
== 2020-12-05 ==
* 10:12 legoktm: deploying https://gerrit.wikimedia.org/r/645504
* 09:19 legoktm: deploying https://gerrit.wikimedia.org/r/645494
* 09:10 legoktm: deploying https://gerrit.wikimedia.org/r/645491
* 00:05 James_F: Docker: [quibble*] Switch from composer-php70 to use composer-scratch
== 2020-12-04 ==
* 23:12 James_F: Zuul: [mediawiki/extensions/IPInfo] Re-enable phan, move to prod section [[phab:T266136|T266136]]
* 21:24 James_F: Docker: Building composer-package images using composer-scratch, hopefully successfully this time.
* 21:08 James_F: Docker: Building composer-package images using composer-scratch
* 20:16 James_F: Docker: Fix composer-scratch images to use compose v1.10.17.
* 19:45 James_F: Docker: Pushing composer-phpN images that use composer-scratch
* 19:34 James_F: Zuul: Add experimental PHP 8.0 jobs for all the things. [[phab:T250298|T250298]]
* 18:46 James_F: Zuul: Added CI for Flink Swift Plugin [[phab:T262942|T262942]]
== 2020-12-02 ==
* 21:20 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/644888
* 18:23 James_F: Zuul: Install CI for mediawiki/servieces/function-evaluator [[phab:T269039|T269039]]
* 08:43 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/phpmetrics:2.4.4  and updating phpmetric job to it
== 2020-12-01 ==
* 16:12 James_F: Zuul: Make PHP74 voting for composer-test-php72-or-later
* 01:56 James_F: Docker: Providing composer-test-php74 and composer-test-php80
* 01:40 James_F: Zuul: Provide experimental PHP 8.0 job for libraries
* 01:18 James_F: Docker: Pushing correct version number of new quibble-strech-php80 image
== 2020-11-30 ==
* 23:47 legoktm: deploying https://gerrit.wikimedia.org/r/644361
* 23:12 legoktm: rebuilding docker iamges for https://gerrit.wikimedia.org/r/644355
* 23:07 legoktm: deploying https://gerrit.wikimedia.org/r/644354
* 22:57 James_F: jjb: Updated all 197 (!) jobs that referred to REL1_34 for [[phab:T268931|T268931]]
* 22:34 James_F: Docker: Bumping new PHP 8.0 images for leanness and AST-worthiness.
* 22:17 James_F: Zuul: [mediawiki/extensions/GlobalWatchlist] Move to Wikimedia-deployed section [[phab:T260862|T260862]]
* 21:27 legoktm: building php8.0 images https://gerrit.wikimedia.org/r/643777
* 19:11 James_F: Zuul: [mediawiki/extensions/WikimediaApiPortalOAuth] Mark as in-prod and enable coverage reports [[phab:T262495|T262495]]
* 16:30 James_F: Zuul: [mediawiki/services/similar-users] Install pipeline/publish CI [[phab:T268839|T268839]]
* 15:22 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/643742
== 2020-11-28 ==
* 18:23 James_F: Zuul: Install CI for mediawiki/extensions/EncryptedUploads
* 05:00 legoktm: reloading Zuul for https://gerrit.wikimedia.org/r/634626
== 2020-11-27 ==
* 16:08 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.11 for jayme / [[phab:T251305|T251305]]
== 2020-11-25 ==
* 21:02 legoktm: deploying https://gerrit.wikimedia.org/r/643549
* 20:47 legoktm: building mediawiki-phan-taint-check-demo image
== 2020-11-24 ==
* 17:11 James_F: Zuul: Install pipeline CI for rdf-streaming-updater [[phab:T265512|T265512]]
* 02:22 Krinkle: click "Disable publishing" for extensions-Scribunto ref [[phab:T267587|T267587]]#6624926, [[phab:T143162|T143162]]
* 01:04 James_F: Zuul: Archive mediawiki/services/graphoid/deploy
== 2020-11-23 ==
* 19:36 James_F: Zuul: Archive the BlueSpiceEditNotifyConnector extension [[phab:T268276|T268276]]
* 17:31 James_F: Zuul: [mediawiki/extensions/WikiSEO] Add Scribunto as phan dependency
== 2020-11-20 ==
* 16:06 James_F: Zuul: [labs/tools/book2scroll] Provide CI with tox-docker [[phab:T267488|T267488]]
* 02:36 DannyS712: dannys712@deployment-deploy01:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php metawiki DannyS712
== 2020-11-19 ==
* 20:30 Amir1: shut down deployment-sca04 ([[phab:T257118|T257118]])
* 20:30 Amir1: delete deployment-aqs* instances ([[phab:T257118|T257118]])
* 18:03 James_F: Zuul: [mediawiki/services/recommendation-api/deploy] Archive [[phab:T248355|T248355]]
* 17:51 James_F: Zuul:[mediawiki/services/recommendation-api] Drop npm-docker [[phab:T248355|T248355]]
* 09:52 hashar: Building newer Docker images for Bazel / Gerrit
* 00:25 Amir1: shutting off deployment-aqs* instances ([[phab:T257118|T257118]])
== 2020-11-18 ==
* 19:02 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:641739]]
== 2020-11-17 ==
* 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/633202 and https://gerrit.wikimedia.org/r/c/labs/private/+/635859 in beta.  Should only affect deployment-wpt01.
* 16:32 dpifke: Un-cherry-picked Python 3 patches in beta, to prevent merge conflicts when they're merged in prod later today.  Should only affect deployment-webperf1[12].
* 10:41 hashar: integration: removed /var/lib/dpkg/lock from integration-castor03
* 00:32 DannyS712: dannys712@deployment-deploy01:~$ mwscript changePassword.php metawiki --user="AGreen (WMF)" --password=<redacted>
== 2020-11-16 ==
* 17:03 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T245444|T245444]]
== 2020-11-12 ==
* 14:33 hashar: Beta cluster is available again.  The frontend cache had a mixup between varnish 5 running and config being for varnish 6.  The package got upgraded and it seems to serve traffic now.  # [[phab:T267561|T267561]]
* 11:19 hashar: Rename https://github.com/wikimedia/mediawiki-services-mathjax-attic back to https://github.com/wikimedia/mediawiki-services-mathjax and keeping it unarchived for Gerrit to replicate to it  # [[phab:T249908|T249908]]
* 11:14 hashar: Unarchiving https://github.com/wikimedia/mediawiki-services-mathjax  and renaming it to https://github.com/wikimedia/mediawiki-services-mathjax-attic  # [[phab:T249908|T249908]]
* 08:29 hashar: Building containers for composer update from 1.10.5 .. 1.10.17 :  https://gerrit.wikimedia.org/r/c/integration/config/+/640574
== 2020-11-11 ==
* 09:50 awight: metawiki: Promoting User:Jan Dittrich (WMDE) into centralnoticeadmin...
== 2020-11-10 ==
* 23:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/640226 and https://gerrit.wikimedia.org/r/c/performance/coal/+/640227 in beta; should only affect deployment-webperf11.
* 22:13 thcipriani: restarting php7.2-fpm on deployment-jobrunner03 and deployment-parsoid11
* 18:40 apergos: deployment-prep:  switched deployment-prep to libicu63, see [[phab:T264991|T264991]]
== 2020-11-09 ==
* 22:18 awight: metawiki: Promoting User:Jan Dittrich into centralnoticeadmin...
* 14:18 hashar: deployment-prep: removing role::beta::availability_collector  from deployment-cache06 since that broke puppet. The role no more exists # [[phab:T267006|T267006]]
== 2020-11-07 ==
* 00:16 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639885 and https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/639886 in beta.  Should only affect deployment-webperf12.
== 2020-11-05 ==
* 18:38 Reedy: beta has been stuck for ~22H. Killed backlog of jobs, seems to be running now
* 09:19 elukey: fixed kafka broker names (s/.eqiad.wmflabs/.eqiad1.wikimedia.cloud/g) in deployment-prep's hiera global config to unblock puppet
== 2020-11-04 ==
* 21:33 hashar: devtools upgraded gerrit da40615f726..9275b308ffd
* 20:33 hashar: devtools: fixed java version on gerrit-prod-1001.devtools.eqiad.wmflabs  by setting java::profile hieradata variable at https://horizon.wikimedia.org/project/instances/48c5ff1c-2885-410d-beeb-d5a57a0a91c7/
* 19:59 hashar: operations/software/gerrit : pushed upstream git tag, synced the master branch  # [[phab:T262241|T262241]]
* 18:50 dpifke: Re-cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta.
* 18:05 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/639223 in beta.
* 17:50 dpifke: Reverting previous cherry-pick for now; requires minor code change first.
* 17:43 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta.
* 16:12 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639197 in beta, should only affect deployment-webperf11.
== 2020-11-02 ==
* 22:18 hashar: Killed Pipeline job , stuck running `docker rmi --force {{Gerrit|ee88e6bf8898}} {{Gerrit|96f16ff7774f}} {{Gerrit|ceb93158475f}} 47e2530d59a6` [[phab:T267075|T267075]]
== 2020-10-30 ==
* 20:07 longma: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/631864
* 18:16 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.2-1 # [[phab:T234565|T234565]]
* 17:48 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.1-1 # [[phab:T234565|T234565]]
== 2020-10-29 ==
* 17:24 andrewbogott: signing pending puppet certs for deployment-mediawiki-07.deployment-prep.eqiad1.wikimedia.cloud and deployment-mediawiki-09.deployment-prep.eqiad1.wikimedia.cloud
* 17:23 andrewbogott: signing pending puppet certs for deployment-kafka* nodes
* 16:57 James_F: Zuul: Add CI for node-rdkafka-factory [[phab:T266058|T266058]]
* 16:17 andrewbogott: removing jkroll as a project member; the registered email is invalid so probably this user is no longer involved
* 14:26 hashar: restarted Jenkins on releases1002 (java upgrade)
== 2020-10-28 ==
* 22:52 hashar: integration: make sure  docker daemon no more has debug enabled by reloading it fleet wide:  sudo cumin --force 'name:docker' 'systemctl reload docker'
* 19:03 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Skip phan
* 18:40 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Use composer quibble
* 14:14 thcipriani: arm keyholder on beta cluster to fix beta-scap-eqiad
* 10:00 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.10 # [[phab:T266032|T266032]]
== 2020-10-27 ==
* 22:46 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/636759 in beta puppet.  Should only affect deployment-webperf12.
* 13:20 hashar: Enable Docker daemon debug mode on integration-agent-docker-1001 via Horizon https://horizon.wikimedia.org/project/instances/837fff63-ed52-4c2f-823d-936801056fd2/  # [[phab:T265615|T265615]]
* 13:19 hashar: Enabling puppet again on CI agent (that also causes Docker/containerd to be rolled back) # [[phab:T265615|T265615]]
== 2020-10-26 ==
* 22:21 James_F: Zuul: [mediawiki/extensions/MediaSearch] Add WikibaseCirrus dependency [[phab:T265939|T265939]]
* 12:00 hashar: Upgrading docker-ce to 19.03.13 and containerd to 1.3.7 on integration-agent-docker-1001, integration-agent-docker-1002, integration-agent-docker-1003 and integration-agent-docker-1004 # [[phab:T265615|T265615]]
* 11:12 hashar: Upgrading containerd.io from 1.2.10 to 1.2.13-2  on integration-agent-docker-1005, integration-agent-docker-1006, integration-agent-docker-1007, integration-agent-docker-1008 # [[phab:T265615|T265615]]
* 11:03 hashar: Bring back integration-agent-docker-1020 . It is not the only one affected by [[phab:T265615|T265615]]  which probably rules out Ceph as the slowness root cause ([[phab:T260916|T260916]])
* 10:57 hashar: Upgrading containerd.io from 1.2.10 to 1.2.13-2  on integration-agent-docker-1001, integration-agent-docker-1002, integration-agent-docker-1003, integration-agent-docker-1004 # [[phab:T265615|T265615]]
* 02:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/636229 [[phab:T266424|T266424]]
== 2020-10-21 ==
* 18:16 longma: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/635277
* 17:57 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/635398
* 17:31 marxarelli: generated 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/635398
== 2020-10-20 ==
* 22:15 James_F: Zuul: [mediawiki/extensions/MediaSearch] Install CI for this new prod repo [[phab:T265939|T265939]]
* 22:14 James_F: Zuul: [mediawiki/extensions/IPInfo] Disable phan for now
* 14:26 hashar: Switched CI jobs to rust 1.47 https://gerrit.wikimedia.org/r/c/integration/config/+/633693
* 09:49 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/sonar-scanner:1.0.1 # [[phab:T265913|T265913]]
* 09:39 hashar: Reloading Zuul for " layout: Add noop job for research/mwaddlink test and gate-and-submit - https://gerrit.wikimedia.org/r/635070" # [[phab:T261403|T261403]]
== 2020-10-19 ==
* 15:55 James_F: Zuul: [mediawiki/tools/train-dev] Add initial CI, non-voting tox
* 12:14 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.9 # [[phab:T263616|T263616]]
== 2020-10-18 ==
* 18:50 James_F: Zuul: Use phpunit-coverage-php72-docker-publish not unversioned one
* 18:05 James_F: Dockerfiles: [composer-test-php72,3] Stop depending on composer-test-php70
* 17:43 James_F: Zuul: Drop php-compile-php7[01]-or-later templates, no longer used
* 17:41 James_F: Zuul: [mediawiki/php/wmerrors] Only test PHP 7.2+ from now on
* 17:39 James_F: Zuul: [mediawiki/php/wikidiff2] Only test PHP 7.2+ from now on
* 17:36 James_F: Zuul: [mediawiki/php/luasandbox] Only test PHP 7.2+ from now on
* 17:33 James_F: Zuul: [mediawiki/php/excimer] Only test PHP 7.2+ from now on
* 17:31 James_F: Zuul: Drop `composer-test` (php70) template, no longer used
* 17:28 James_F: Zuul: [wikimedia/iegreview] Only test PHP 7.2+ from now on
* 17:26 James_F: Zuul: [operations/dumps/dcat] Only test PHP 7.2+ from now on
* 17:22 James_F: Zuul: [integration/docroot] Only test PHP 7.2+ from now on
* 17:17 James_F: Zuul: [wikimedia-cz/mediawiki-config] Only test PHP 7.2+ from now on
* 17:14 James_F: Zuul: [analytics/wmde/scripts] Only test PHP 7.2+ from now on
== 2020-10-16 ==
* 19:53 James_F: Zuul: Add Vlad Shapik to the CI allowlist
* 19:48 James_F: Zuul: [mediawiki/extensions/SimpleSecurity] Archive [[phab:T252722|T252722]]
* 19:48 James_F: Zuul: [labs/tools/wiki-importer] Add CI
* 19:48 James_F: Zuul: [labs/tools/search-filters] Add CI
* 19:34 James_F: Zuul: Remove selenium from 14(!) BlueSpice extensions [[phab:T264437|T264437]]
* 19:25 James_F: Zuul: [mediawiki/skins/Cosmos] Add soft dependencies to CI jobs [[phab:T265539|T265539]]
* 19:19 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add extension-codehealth [[phab:T224990|T224990]]
* 19:10 James_F: Zuul: Expand composer-test* templates for php73 now we have it
* 19:06 James_F: Zuul: [wikimedia/fundraising/SmashPig] Switch to composer-php73-docker
* 19:04 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=zhwiki autoconfirmed {{!}} [[phab:T265755|T265755]]
* 19:04 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=testwiki suppress {{!}} [[phab:T265755|T265755]]
* 19:03 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=testwiki editor {{!}} [[phab:T265755|T265755]]
* 19:03 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=testwiki autoreview {{!}} [[phab:T265755|T265755]]
* 19:02 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=fawiki autopatrol {{!}} [[phab:T265755|T265755]]
* 19:01 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=fawiki uploader {{!}} [[phab:T265755|T265755]]
* 19:00 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=enwiki suppress {{!}} [[phab:T265755|T265755]]
* 18:59 DannyS712: dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=enwiki autoreview {{!}} [[phab:T265755|T265755]]
* 18:45 James_F: Zuul: Switch various FR things from php70 to mwgate-composer-php73-docker
* 18:27 James_F: Zuul: [cloud/toolforge/buildpacks] Add CI
* 18:26 James_F: Zuul: Add in-wikimedia-fundraising-production and assert shape
* 18:15 legoktm: building new shellcheck docker image
* 17:26 James_F: [DonationInterface] Switch special FR job from REL1_31-php70 to REL1_35-php73
* 17:06 James_F: [FundraisingEmailUnsubscribe] Switch special FR job from REL1_31-php70 to REL1_35-php73
* 08:21 hashar: Nuking castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm
* 07:48 hashar: Disabling integration-agent-docker-1020 (the sole agent using Ceph): it is too slow # [[phab:T260916|T260916]] [[phab:T265615|T265615]]
* 07:44 hashar: arming keyholder on integration-cumin
== 2020-10-15 ==
* 19:23 James_F: Zuul: Add David Kamholz to CI allowlist
* 16:44 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/doxygen:0.9.0 # [[phab:T254465|T254465]]
== 2020-10-14 ==
* 18:12 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/634058 in beta
== 2020-10-12 ==
* 15:22 hashar: Updated non voting codehealth jobs to Java11 for [[phab:T259646|T259646]]: mwcore-codehealth-master-non-voting mwext-codehealth-master-non-voting
* 13:58 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/maven-java11:1.0.0 # [[phab:T259646|T259646]]
* 13:52 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/maven:3.5.2-1
* 13:45 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/maven-java8:0.7.0
* 13:35 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/java11:0.1
* 08:42 hashar: integration-castor: nuking large /srv/jenkins-workspace/caches/wikidata-query-rdf/master/wikidata-query-rdf-maven-java8-docker
* 08:39 hashar: integration-castor: nuking 17GB large /srv/jenkins-workspace/caches/wikidata-query-rdf/master/wikidata-query-rdf-maven-java8-docker-site-publish
== 2020-10-08 ==
* 16:22 hashar: CI Jenkins: upgraded a few plugins
* 16:19 hashar: Restarting CI Jenkins
* 10:06 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/operations-puppet:0.7.8
== 2020-10-07 ==
* 10:43 godog: move swift settings out of horizon and into puppet's hieradata
* 04:12 greg-g: deployment-prep: removing cwdent, demon, mobrovac, and smalyshev from access (former employees with very little chance of using access anymore)
== 2020-10-06 ==
* 23:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/632583 https://gerrit.wikimedia.org/r/632584
* 16:17 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/632510
* 15:04 addshore: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/632482
* 12:27 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.8 for jayme  # [[phab:T264157|T264157]]
== 2020-10-05 ==
* 22:31 Amir1: deleted deployment-mailman01 ([[phab:T257118|T257118]])
* 22:29 Amir1: deleted deployment-imagescaler01 and deployment-imagescaler02 ([[phab:T257118|T257118]])
* 21:00 Urbanecm: Run keyholder arm at deployment-cumin
* 21:00 Urbanecm: Run puppet at deployment-mwmaint01 and deployment-mediawiki-07
* 20:45 hauskatze: Fixed puppet for deployment-mediawiki-07: s/memcache/memcached/
* 15:50 thcipriani: start deployment-mediawiki-07
* 15:43 brennen: restarting gitlab-test after disabling prometheus metrics
* 11:39 hashar: deployment-prep sudo chown jenkins-deploy:wikidev /srv/mediawiki-staging/php-master/.git/index
* 11:38 hashar: deployment-prep: sudo chown -R jenkins-deploy:wikidev  /srv/mediawiki-staging/php-master/extensions
* 11:37 hashar: deployment-prep: sudo chown -R jenkins-deploy:wikidev  /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler
* 11:31 Urbanecm: Restart build of beta-scap-eqiad to sync beta with master branches
* 11:30 Urbanecm: Run cd /srv/mediawiki-staging/php-master/extensions/$1 && git reset HEAD * && git checkout -- * for all extensions at beta ([[phab:T264563|T264563]])
== 2020-10-03 ==
* 16:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/631931
== 2020-10-02 ==
* 14:39 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/operations-puppet:0.7.7 # [[phab:T263728|T263728]]
* 14:09 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.7 for jayme # [[phab:T264157|T264157]]
* 13:05 hashar: Successfully tagged docker-registry.wikimedia.org/releng/helm-linter:0.2.7  [[phab:T264157|T264157]]
== 2020-09-30 ==
* 20:47 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T261051|T261051]]
* 19:19 longma: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/630961
== 2020-09-29 ==
* 16:29 James_F: Zuul: Drop Echo and Thanks extensions as dependencies for AbuseFilter
== 2020-09-28 ==
* 19:09 James_F: Zuul: PipelineLib config for WVUI project
* 15:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/630607
* 13:47 hashar: deployment-snapshot01: freed from texlive packages. No more installed by puppet after https://gerrit.wikimedia.org/r/c/operations/puppet/+/540154
* 13:39 hashar: deployment-snapshot01 removing large texlive packages. Superseded by mathoid
== 2020-09-25 ==
* 12:56 hashar: Updating phan jobs for --long-progress-bar https://gerrit.wikimedia.org/r/c/integration/config/+/630163
* 12:03 hashar: Updated phan jobs to use PHAN_COLOR_SCHEME=light  # [[phab:T259890|T259890]]
* 11:34 hashar: Building docker images for https://gerrit.wikimedia.org/r/630094
* 08:59 hashar: Updating phan jobs to use text output instead of checkstyle. https://gerrit.wikimedia.org/r/c/integration/config/+/630068/ {{!}} [[phab:T259890|T259890]]
== 2020-09-24 ==
* 18:49 Urbanecm: Remove dereckson from wmf-deployment Gerrit group - user is no longer a deployer
* 18:21 paladox: created openstack/horizon/barbican-ui per request from andrewbogott in #wikimedia-cloud
* 15:55 hashar: Building docker images
== 2020-09-23 ==
* 19:22 hashar: operations/software/gerrit changing HEAD from deploy/wmf/stable-2.15 to deploy/wmf/stable-3.2  # [[phab:T254158|T254158]]
* 19:11 hashar: gerrit: flushing "project_list" cache after deletion of " mediawiki/extensions/WikiLambda":  gerrit flush-caches --cache project_list  # [[phab:T263682|T263682]] [[phab:T257775|T257775]]
* 17:08 hashar: Gerrit: removed "/srv/gerrit/git/ mediawiki/extensions/WikiLambda" (note the extra space in front of 'mediawiki')  # [[phab:T257775|T257775]]
* 13:33 James_F: Kicked the Beta Cluster node on Jenkins to make it come back again.
* 13:20 James_F: Restarted CI Jenkins for [[phab:T263289|T263289]]
* 13:20 James_F: Restarted Releases Jenkins for [[phab:T263289|T263289]]
== 2020-09-22 ==
* 14:18 hashar: Building Docker image releng/mediawiki-phan-testrun:0.2.0
== 2020-09-21 ==
* 22:05 mutante: maintenance with possible short downtime for releases.wikimedia.org while switching it to an active-active service with multi-dc backends
* 21:25 James_F: Zuul: Install CI for mediawiki/libs/Shellbox [[phab:T263295|T263295]]
* 19:12 hashar: Adding Razzi (new SRE) to Gerrit Analytics group # [[phab:T261443|T261443]]
* 18:56 dancy: Restarting Jenkins agent on deployment-deploy01
* 09:45 James_F: Beta Cluster: Deleted a bunch of old log files on deployment-snapshot01 so that scap stops failing
* 07:53 hashar: Upgrading all Jenkins jobs to Quibble 0.0.45
* 07:37 hashar: integration: purged old containers and images via integration-cumin:  sudo cumin --force 'name:docker' 'docker image prune -f' && sudo cumin --force 'name:docker' 'docker container prune -f'
* 07:34 hashar: Upgrading https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/ to Quibble 0.0.45
== 2020-09-18 ==
* 13:32 hashar: Building Quibble Docker images on contint1001 (cause it is faster than the primary contint2001 for some reason)
* 13:30 hashar: Tag Quibble 0.0.45 @ {{Gerrit|70a3cfe8c8045582c1b1f49797af30a4deb15def}}
* 13:25 hashar: contint1001 pruning dangling docker images
* 12:04 hashar: Disabled  Submit on Gerrit integration/* projects
* 09:30 hashar: deployment-snapshot01 : deleted /srv/mediawiki/php-master/cache/l10n/ and ran scap pull
* 09:01 hashar: deployment-snapshot01: deleted a couple of .~tmp~ directories under /srv/mediawiki/php-master/cache/l10n
== 2020-09-17 ==
* 13:18 James_F: Zuul: Add Arthur P. Smith to the CI allowlist
* 00:11 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/626241 in beta.
== 2020-09-16 ==
* 23:26 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/626241 in beta.
* 19:04 thcipriani: deployment-snapshot01 removed existing .~tmp~ directories
* 08:18 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.6.2-s4  # [[phab:T256942|T256942]]
== 2020-09-15 ==
* 21:45 brennen: Updating dev-images docker-pkg files on primary contint [[phab:T261051|T261051]]
* 17:49 Krinkle: Logstash in beta cluster appears broken. No messages from Scap or MediaWiki in several days. ref [[phab:T233134|T233134]]
* 17:41 Krinkle: Logstash in beta cluster appears broken. No messages from Scap or MediaWiki in several days.
== 2020-09-13 ==
* 18:30 James_F: layout: [mediawiki/extensions/PageMagic] Install quibble CI
== 2020-09-11 ==
* 22:10 dpifke: Testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/626779 in beta, should only affect deployment-webperf12.
* 20:39 brennen: gitlab-test.wmcloud.org back up, now with more SSH (hopefully)
* 19:46 ebernhardson: deployment-prep delete elastic indices dated prior to 2020.06 from deployment-logstash03
* 19:28 brennen: gitlab-test may experience some downtime as a result of zone transfer for [[phab:T262516|T262516]], [[phab:T261900|T261900]]
* 10:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/626605
* 10:19 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/mediawiki-phan-php74:0.1.0
* 10:14 hashar: Building docker image releng/mediawiki-phan74
* 10:14 hashar: Reloading Zuul for "Add PHP74 jobs to REL1_35 pipelines where missing" https://gerrit.wikimedia.org/r/c/integration/config/+/626628/
== 2020-09-09 ==
* 09:05 hashar: deployment-prep upgrading git from 2.11 to 2.20 on all Stretch based hosts # [[phab:T262244|T262244]]
* 09:03 hashar: deployment-puppetdb03 set `profile::puppetdb::jvm_opts: -Xmx256m` via Horizon and restarted Puppetdb # [[phab:T248041|T248041]]
* 08:43 hashar: Restarted puppetdb on deployment-puppetdb03 ([[phab:T248041|T248041]])
* 08:34 hashar: deployment-prep: cherry picking "upgrade git on stretch to 2.20" # https://gerrit.wikimedia.org/r/c/operations/puppet/+/625847
== 2020-09-08 ==
* 12:25 hashar: Updated releng Gerrit dashboard to list up to 3 of one patches being in merge conflict ( https://gerrit.wikimedia.org/r/p/releng/+/dashboard/main:default  )
* 11:51 hashar: Building Docker images for https://gerrit.wikimedia.org/r/625744 # [[phab:T262236|T262236]]
* 06:58 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/625137
== 2020-09-07 ==
* 19:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/625718 and https://gerrit.wikimedia.org/r/c/integration/config/+/625719
* 18:58 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/625715
* 18:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/624847
== 2020-09-04 ==
* 18:05 dpifke: Rebasing and re-cherry-picking some patches in beta puppet to fix merge conflicts.
* 17:52 dpifke: Cherry-picking newest patch set from https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 in beta.
* 17:28 James_F: Zuul: [ParserFunctions,cldr] Drop special FR REL1_31-php70 jobs
* 16:19 James_F: Zuul: Voting FR jobs for ParserFunctions and cldr.
* 15:32 hashar: Updated doc.wikimedia.org docroot for https://gerrit.wikimedia.org/r/c/integration/docroot/+/624714
== 2020-09-03 ==
* 13:29 hashar: Restarted CI Jenkins for Blue Ocean plugins upgrade
* 12:53 hashar: CI Jenkins: updating Blue Ocean plugins from 1.19.0 to 1.23.2
* 10:25 hashar: Unarchived https://github.com/wikimedia/mediawiki-services-mathjax-node/ , the canonical repository is in Gerrit now # [[phab:T249908|T249908]]
* 07:24 hashar: contint2001: restarting CI Jenkins for  plugins upgrade
* 07:07 hashar: contint2001: upgraded most plugins
* 07:07 hashar: releases-jenkins: upgraded all plugins
== 2020-09-02 ==
* 19:00 brennen: upgrading gitlab-ce install on gitlab-test
* 16:06 hashar: Update Quibble jobs to latest Docker images # https://gerrit.wikimedia.org/r/c/integration/config/+/623763/
* 15:58 hashar: Updating Jenkins jobs to latest Docker images # https://gerrit.wikimedia.org/r/c/integration/config/+/623762/
* 12:47 James_F: Zuul: Drop PHP 7.2 testing for REL1_35 branches [[phab:T257879|T257879]]
* 07:36 hashar: Updated doc.wikimedia.org document root ( 28f8ce4..a990b52 ): ssh doc1001.eqiad.wmnet sudo -H -u doc-uploader git -C /srv/docroot pull
== 2020-09-01 ==
* 11:04 hashar: Manually triggering doc generation for operations/software/pywmflib: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/pywmflib --change 617403,6 # [[phab:T257905|T257905]]
* 09:53 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/622997
== 2020-08-31 ==
* 17:54 James_F: Zuul: Install CI for mediawiki/extensions/IPInfo [[phab:T260609|T260609]]
* 17:36 James_F: Zuul: Install CI for new wikimedia/security repositories [[phab:T246392|T246392]]
* 15:36 James_F: Zuul: Add Wikibase as WikibaseManifest dependency [[phab:T260975|T260975]]
== 2020-08-29 ==
* 12:01 James_F: dockerfiles: [mediawiki-phan-php73] Publishing 0.1.1
* 11:43 James_F: layout: Migrate REL1_35 branches to use PHP 7.3 by default
* 11:24 James_F: layout: Migrate use of selenium-only jobs to explicit versions
== 2020-08-28 ==
* 22:35 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta puppet; should only affect deployment-webperf12.
* 06:04 legoktm: rebuilding Rust images for 1.46.0
* 00:21 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 in beta.
== 2020-08-27 ==
* 23:19 dpifke: Restarted puppetdb on deployment-puppetdb03 ([[phab:T248041|T248041]])
* 23:17 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/622904 in beta puppet, should only affect deployment-webperf12.
* 20:35 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/gerrit:0.1.0-s10
* 16:58 hashar: Docker CI images are still being build on contint2001. It takes age for some reason
== 2020-08-26 ==
* 16:29 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add code health job
* 10:50 hauskatze: GitHub: Netlify GitHub app install request approved for the wikimedia/wvui.git repository (only)
* 10:15 hashar: Updating Jenkins jobs  https://gerrit.wikimedia.org/r/622536
* 10:03 hashar: Rebuilding Docker images for  https://gerrit.wikimedia.org/r/622534 "stop cd /src in ci-src-setup.sh"
* 09:40 hashar: Updating Jenkins jobs for ci-src-setup-simple:0.4.1
* 09:39 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/ci-src-setup-simple:0.4.1
== 2020-08-25 ==
* 18:03 dpifke: Restarted puppetdb on deployment-puppetdb03 ([[phab:T248041|T248041]])
* 16:58 hashar: deployment-deploy01: upgraded scap (3.15.0-1+0~20200825155225.104~1.gbp6cd96d) over (3.15.0-1+0~20200825151423.100~1.gbpec9bb3)
* 16:54 hashar: Build Docker images to use a common ci-src-setup.sh script instead of copy pasted shells # https://gerrit.wikimedia.org/r/c/integration/config/+/622374
* 16:26 James_F: tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ 'beta-scap-eqiad'
* 08:16 legoktm: rebuilding helm-linter image for https://gerrit.wikimedia.org/r/622307
* 07:59 legoktm: rebuilding rust-coverage image for https://gerrit.wikimedia.org/r/c/integration/config/+/622305
* 01:49 dpifke: Deployed https://gerrit.wikimedia.org/r/c/performance/debs/xhgui/+/621855 on deployment-xhgui03.
== 2020-08-24 ==
* 15:44 addshore: Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/622166
* 11:53 James_F: Zuul: [mediawiki/extensions/LiveStreaming & LiveChat] Install basic quibble CI
* 11:52 James_F: Zuul: [mediawiki/extensions/OnionsPorFavor] Install basic quibble CI
* 11:46 James_F: Zuul: [mediawiki/extensions/RelMicroformat] Install basic quibble CI
* 07:40 hashar: contint2001: removed checkout of integration/docroot.git  which was at /srv # [[phab:T149924|T149924]]
== 2020-08-22 ==
* 02:10 dpifke: Manually installed new XHGui package built from https://gerrit.wikimedia.org/r/c/performance/debs/xhgui/+/621855 on deployment-xhgui03.
== 2020-08-21 ==
* 02:08 thcipriani: depool integration-agent-docker-1020
== 2020-08-20 ==
* 22:17 dpifke: Deleted deployment-xhgui01.deployment-prep - no longer need MongoDB test instance ([[phab:T180761|T180761]])
* 18:44 hashar: Pooling integration-agent-docker-1020 # [[phab:T260930|T260930]] / [[phab:T260916|T260916]]
* 16:07 hashar: Depooled integration-agent-docker-1020  to Jenkins cant connnect to /var/run/docker.sock # [[phab:T260916|T260916]]
* 16:02 hashar: Added integration-agent-docker-1020  to Jenkins # [[phab:T260916|T260916]]
* 15:40 hashar: Created dummy instance integration-agent-docker-1020  using a Ceph backed hypervisor # [[phab:T260916|T260916]]
* 12:11 hashar: Gracefully stopping Zuul
== 2020-08-19 ==
* 14:56 hashar: Updated Quibble jobs to git 2.20.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/621287
* 10:33 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/620666
* 10:17 legoktm: rebuilding rust image for https://gerrit.wikimedia.org/r/621213
* 09:37 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/621116
* 03:24 legoktm: rebuilding rust image for https://gerrit.wikimedia.org/r/621115
== 2020-08-18 ==
* 20:54 dpifke: Restarted puppetdb on deployment-puppetdb03
* 16:47 hashar: Rebuilding CI containers for Quibble to update GitPython [[phab:T256844|T256844]]#6393265 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/620990/
* 14:47 hashar: Refreshing all JJB jobs to ensure rollback change  https://gerrit.wikimedia.org/r/620966 is properly applied
* 11:39 hashar: Updating all CI jobs based on ci-stretch for git protocol v2 https://gerrit.wikimedia.org/r/c/integration/config/+/620913/ {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/620914/
* 09:26 hashar: gerrit: made operations-software-wmfmariadbpy  self owned, added kormat to the group. # [[phab:T260342|T260342]]
* 09:05 hashar: Restarting CI Jenkins
* 08:41 hashar: Restarted releases-jenkins for 'Email Extension' plugin update
* 06:03 legoktm: rebuilding rust and rust-coverage images (https://gerrit.wikimedia.org/r/620823, https://gerrit.wikimedia.org/r/620824)
* 05:15 legoktm: rebuilding rust-coverage image https://gerrit.wikimedia.org/r/620819
* 05:08 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/620817
* 00:38 legoktm: rebuilding bazel & gerrit images (try #2)
* 00:21 legoktm: rebuilding bazel & gerrit images (with https://gerrit.wikimedia.org/r/620807 fix)
== 2020-08-17 ==
* 23:58 legoktm: building rust-coverage image (https://gerrit.wikimedia.org/r/620804)
* 17:05 hashar: docker-pkg still running on contint2001, busy rebuilding ci-stretch based images :-\
* 15:43 thcipriani: restarting releases-jenkins after plugin update
* 15:17 hashar: Rebuild CI image releng/typos due to a build failure in previous change.  https://gerrit.wikimedia.org/r/c/integration/config/+/620715
* 15:16 hashar: Build new CI images for https://gerrit.wikimedia.org/r/c/integration/config/+/620696 (git v2 [[phab:T256844|T256844]] / drop stretch-backports [[phab:T257327|T257327]] )
== 2020-08-15 ==
* 07:41 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/620426
* 07:28 legoktm: rebuilding rust image https://gerrit.wikimedia.org/r/620404
* 06:19 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/620398 https://gerrit.wikimedia.org/r/620399
* 05:47 legoktm: building rust:1.45.2-2 image (https://gerrit.wikimedia.org/r/620396)
* 05:37 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/620395
== 2020-08-14 ==
* 13:06 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/rust:1.45.2-1 # [[phab:T256827|T256827]]
== 2020-08-13 ==
* 18:09 greg-g: deleting Phab user @Forza1000 cf: [[phab:T260380|T260380]]
* 09:33 James_F: Zuul: Archive mediawiki/extensions/AuthenticatedKeyValueStore [[phab:T252864|T252864]]
* 09:31 James_F: Docker: Publishing mediawiki-phan-php73
== 2020-08-12 ==
* 17:29 James_F: Docker: Publishing civicrm 0.2.0 image
* 17:00 James_F: Run CiviCRM tests on PHP 7.3 only
* 17:00 James_F: [pywikibot/core] Add fasttest-py37; drop fasttest-py27 [[phab:T213287|T213287]]
== 2020-08-11 ==
* 08:09 hashar: Gerrit: marked operations/software/prometheus_jmx_exporter read-only/archive since we now use a Debian package.  [[phab:T181728|T181728]]
== 2020-08-10 ==
* 23:01 dpifke: puppetdb on deployment-puppetdb03 died Aug  8 22:12:28 due to out-of-memory; restarted.
* 23:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/619391
* 22:56 dpifke: Cleaned up beta puppet to remove previously cherry-picked patches where are now merged.
* 10:14 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/composer-test-php73:0.1.0
* 09:10 hashar: Rebuilding images opereations-dns-lint and operations-puppet (  https://gerrit.wikimedia.org/r/619261 )
* 08:36 hashar: Building docker images for  https://gerrit.wikimedia.org/r/618962  "update most ENTRYPOINT to exec form"
* 08:31 hashar: Reloaded Zuul for Wikispeech services  https://gerrit.wikimedia.org/r/c/integration/config/+/619038
== 2020-08-07 ==
* 21:21 dpifke: Cherry picking latest version of https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429, and enabling swift-object-expirer on deployment-ms-be05 via horizon hieradata.
* 15:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/618801
== 2020-08-06 ==
* 23:46 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429 in beta.
* 16:53 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add extension-coverage
* 15:29 bd808: Restarted puppetdb on deployment-puppetdb03.deployment-prep.eqiad.wmflabs. Dead from OOM.
== 2020-08-05 ==
* 21:26 dpifke: Creating performance_arclamp Swift account and related container for testing on deployment-ms-fe03.
* 17:13 hashar: BAH, updated again  https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/
* 17:11 hashar: Reverted  https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ back
* 17:07 hashar: Updated https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ so that it no more fails when being skipped due to not being triggered from postmerge / gate-and-submit
== 2020-08-04 ==
* 22:00 mdholloway: deleted deployment-chromium01
== 2020-08-03 ==
* 20:08 hashar: Updating various jobs to fix a cache pollution caused by Chromium. https://gerrit.wikimedia.org/r/618144
* 19:11 hashar: Reloaded Zuul for {{Gerrit|I215ee6238932be041bff6fa6cc453dc4cfa9512f}}
* 15:46 bd808: `service puppetdb start` on deployment-puppetdb03.deployment-prep.eqiad.wmflabs. Looks like it died from OOM
* 08:07 _joe_: nevermind - I don't have the necessary permissions :)
* 08:06 _joe_: updating helm-lint job
* 07:09 _joe_: upgrade helm-linter image
== 2020-07-31 ==
* 21:21 thcipriani: deployment-snapshot01:sudo find /srv/mediawiki -name '.~tmp' -delete
== 2020-07-30 ==
* 23:07 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/616177
* 23:02 marxarelli: created 6 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/616177
* 20:45 dpifke: Fixed merge conflicts in beta puppetmaster.
* 16:19 bd808: Marked integration-agent-docker-1005 as online after reboot
* 16:12 bd808: Rebooting integration-agent-docker-1005
* 15:53 bd808: Marked integration-agent-docker-1005 offline in Jenkins to poke at a Puppet failure related to Docker upgrade there
* 02:00 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/617275 in beta, should only affect deployment-webperf12.
== 2020-07-29 ==
* 22:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/617276
* 13:09 James_F: Zuul: Installing CI for wikimedia/security/landing-page [[phab:T257837|T257837]]
== 2020-07-28 ==
* 15:48 hashar: integration/gear : git tag 0.15.1+wmf1 {{Gerrit|5fd9c37}}    for [[phab:T258630|T258630]]
* 10:06 James_F: Zuul: [mediawiki/extensions/GlobalWatchlist] Add extension-javascript-documentation
* 09:41 hashar: Removed bunch of chrome lighthouse cache files on castor instance # [[phab:T258972|T258972]]
== 2020-07-27 ==
* 10:11 James_F: Zuul: [mediawiki/services/mobileapps/deploy] Archive [[phab:T247012|T247012]]
* 10:08 James_F: Zuul: [mediawiki/services/mobileapps] Drop npm-docker job [[phab:T247012|T247012]] [[phab:T218733|T218733]]
== 2020-07-25 ==
* 18:10 James_F: Zuul: [mediawiki/extensions/Flow] Drop rake job, hurrah.
== 2020-07-24 ==
* 23:22 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T257044|T257044]]
* 18:49 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/616138
* 18:13 brennen: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/616133
* 03:21 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/615914
* 03:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/615912
* 03:08 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/615910
== 2020-07-23 ==
* 15:18 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/615763
== 2020-07-22 ==
* 20:48 brennen: restarted php7.2-fpm on deployment-mediawiki-<nowiki>{</nowiki>07,09<nowiki>}</nowiki> for [[phab:T258628|T258628]]
== 2020-07-21 ==
* 21:16 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615269 in beta puppet; should only affect deployment-webperf11.
* 18:15 brennen: iterating on test of maintenance-disconnect-full-disks for [[phab:T258515|T258515]]
* 15:47 thcipriani: intergration-agent-docker-1001 back online
== 2020-07-20 ==
* 22:17 brennen: offlining integration-agent-docker-1003 for maintenance-disconnect-full-disks testing
* 22:02 legoktm: onlined integration-agent-docker-1011
* 22:01 legoktm: onlined integration-agent-docker-1010
* 22:00 legoktm: onlined integration-agent-docker-1002
* 20:45 brennen: being testing changes to maintenance-disconnect-full-disks
* 19:35 halfak: deploying ores {{Gerrit|f3c44be}}
* 16:48 dancy: integration-agent-docker-1007 back online
* 14:27 thcipriani: unlock mediawiki/core for zuul merger
* 14:23 thcipriani: bring integration-agent-docker-101<nowiki>{</nowiki>1,3<nowiki>}</nowiki> back online, disk space looks fine
* 14:23 thcipriani: locking mediawiki/core for zuul merger to prevent a 16-patch chain from keeping CI occupied forever
== 2020-07-17 ==
* 22:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/613740/ into beta ArcLamp.
* 17:03 thcipriani: restarting apache2 on deployment-mediawiki-07
* 14:28 thcipriani: integration-agent-docker-1011:/srv @ 20%, bringing node back online
* 09:04 addshore: reload zuul for [[gerrit:612562]] [[phab:T257549|T257549]] layout: [mediawiki/extensions/Wikibase] make apitests voting
* 00:18 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/613359 into beta puppet.
== 2020-07-16 ==
* 23:49 Reedy: unstuck beta ci jobs
* 23:46 Krinkle: Beta Jenkins has been stuck for 5h+ (beta-mediawiki-config-update-eqiad  on https://integration.wikimedia.org/zuul/ )
* 17:15 andrewbogott: added "profile::java::egd_source: /dev/random" to project-wide hiera since lack of that setting seems to be breaking puppet in a lot of places
* 17:10 thcipriani: integration-agent-docker-1007 back online, large job using disk space, /srv was at 95%, now at 12%
* 11:56 James_F: Zuul: Add Thanks and Echo as dependencies of AF
* 11:55 James_F: Zuul: Adjust CI allowlist to for Art Tsymbar
== 2020-07-15 ==
* 21:47 brennen: bringing integration-agent-docker-1012 back online
* 20:33 brennen: taking integration-agent-docker-1012 offline to experiment with maintenance-disconnect-full-disks
* 20:17 brennen: deploying test version of maintenance-disconnect-full-disks (which might be wrong)
* 19:48 brennen: bringing integration-docker-1012 back online in jenkins
* 19:47 brennen: brennen@integration-agent-docker-1012:/srv/jenkins/workspace/workspace$ sudo rm -rf *
* 19:43 brennen: bringing integration-agent-docker-1010 back online in jenkins
* 19:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612284
* 18:41 brennen: finished jenkins upgrade & restart on all hosts
* 18:10 brennen: preparing to upgrade jenkins on contint1001, contint1002
* 15:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612673
== 2020-07-14 ==
* 19:56 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612430
* 19:43 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/612429
* 16:50 dancy: Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/611457 into deployment-puppetmaster04:/var/lib/git/operations/puppet
* 15:38 James_F: Docker: Publishing tox-pyspark 0.6.0
* 15:28 bd808: Silenced prometheus alerts for 7d
* 09:34 James_F: Running `./branch.py REL1_35 --bundle '*' --branchpoint master --core-version 1.35.0` for [[phab:T257892|T257892]]
* 05:44 James_F: Created WMF-deployed REL1_35 branches with `./branch.py --branchpoint {{Gerrit|7d04152db4f8ea9a459511bed8117101d9bb4602}} --bundle base --core-version 1.35.0-beta --task [[phab:T256376|T256376]] REL1_35`
* 05:17 James_F: Zuul: parameter_functions: VisualEditor and TemplateData are now tarballed (no-op)
== 2020-07-13 ==
* 20:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612364
* 15:44 Reedy: jjb deployed wikimedia-portals-build to bring in https://gerrit.wikimedia.org/r/612371
* 10:32 James_F: Kicked Jenkins deployment-deploy01 executor for the usual reason.
* 08:36 James_F: Zuul: Add Tks4Fish to CI allowlist
== 2020-07-10 ==
* 23:43 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/611465 into beta, should only affect deployment-webperf12.
* 21:54 dancy: Running `puppet agent --test` on deployment-deploy01.deployment-prep.eqiad.wmflab
* 13:00 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/tox-acme-chief:0.5.0 # [[phab:T257456|T257456]]
* 10:15 James_F: Zuul: [mediawiki/services/chromium-render/deploy] Archive [[phab:T217114|T217114]]
== 2020-07-09 ==
* 19:15 hashar: Reloading Zuul to have messages to gerrit tagged with 'autogenerated:ci' {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/610361 {{!}} [[phab:T48148|T48148]]
* 17:30 thcipriani: unstick beta-scap-eqiad by cancelling stuck job
* 10:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/610713 [[phab:T257549|T257549]]
* 09:25 hashar: contint2001: manually pulled all Docker images referenced in jjb.
* 01:12 dpifke: Cherry picking CL 608973 into beta puppet, should only affect webperf hosts.
== 2020-07-08 ==
* 16:32 Reedy: unstuck beta auto deploy jobs
== 2020-07-07 ==
* 03:56 DannyS712: beta cluster enwiki: ran `DELETE FROM pagetriage_page WHERE ptrp_page_id = 194182 LIMIT 1;` ([[phab:T234577|T234577]])
== 2020-07-04 ==
* 23:51 Amir1: deleted deployment-sentry01 ([[phab:T106915|T106915]])
* 08:56 hashar: Fixed Jenkins collapsible section parsing for Quibble. A logger changed from quibble.cmd to quibble.commands. # [[phab:T220586|T220586]]
== 2020-07-03 ==
* 05:03 legoktm: deploying https://gerrit.wikimedia.org/r/609279
== 2020-07-02 ==
* 19:26 James_F: Zuul: [mediawiki/extensions/CheckUser] Add EventLogging to phan deps
* 16:00 joal: rerun mediawiki-history-denormalize-wf-2020-06 after oozie sharelib bump through manual restart
== 2020-07-01 ==
* 21:48 dpifke: Un-cherry-picking change 603550 (now merged), cherry-picking 608962 in beta.
* 21:31 Krinkle: delete performance-beta.wmflabs.org web proxy (handled by performance.wikimedia.beta.wmflabs.org instead via puppetised handling)
* 20:17 longma: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/608500
* 19:14 hashar: Pooling back integration-agent-docker-1003 , got disconnected by maintenance-disconnect-full-disks # 191095
* 17:57 James_F: Zuul: Add CI for mediawiki/extensions/WikiLambda
* 16:39 James_F: Manually re-set zuul's checkout of labs/libraryupgrader's git repo on contint2001 so we can merge things again
* 08:47 James_F: Zuul: Configure the REL1_35 test and gate pipelines [[phab:T256377|T256377]]
* 08:38 James_F: Zuul: Stop defining the REL1_33 pipelines at all [[phab:T256087|T256087]]
* 08:37 James_F: Zuul: Stop ascribing stuff to REL1_33 pipelines [[phab:T256087|T256087]]
== 2020-06-30 ==
* 23:20 Reedy: Clean up bogus empty captchas from beta presumably caused by testing of [[phab:T230245|T230245]]
* 14:15 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/608626
* 07:32 hashar: contint2001: rm -fR /srv/deployment/integration/slave-scripts # [[phab:T252955|T252955]]
== 2020-06-29 ==
* 12:48 hashar: Updating Jenkins jobs for releng/ci-src-setup-simple:0.3.0  # https://gerrit.wikimedia.org/r/c/integration/config/+/608069 {{!}} [[phab:T256462|T256462]]
== 2020-06-28 ==
* 10:32 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/608154
== 2020-06-27 ==
* 18:52 hashar: Triggering again https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/
== 2020-06-26 ==
* 15:18 hashar: Building new ci-* Docker images # https://gerrit.wikimedia.org/r/#/c/integration/config/+/608013/
* 12:20 Daimona: Adding debug logging on beta-deploymentwiki for [[phab:T253213|T253213]]
== 2020-06-25 ==
* 22:02 dpifke: Testing changes 607369 and 607370 in beta, should only affect deployment-webperf12.
* 15:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/607818
* 12:52 hashar: for volans: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 603434,15
== 2020-06-24 ==
* 18:56 halfak: deploying ORES {{Gerrit|1b87365}}
* 16:32 dpifke: Cherry-picking new version of change 603550 into beta puppet for testing, and pointing beta XHGui at new instance.
* 08:40 James_F: Zuul: Add proc to CI allowlist
== 2020-06-23 ==
* 17:15 bstorm_: to fix puppet on several hosts, setting profile::java::hardened_tls: false in project puppet on horizon
* 17:09 bstorm_: restarted postgresql on deployment-puppetdb03
* 17:05 bstorm_: restarted puppetdb.service on deployment-puppetdb03
== 2020-06-22 ==
* 13:03 hashar: Deleted integration-cumin-01 , replaced by integration-cumin
* 12:40 James_F: Docker: Publishing images using npm 6.14.5
* 12:38 hashar: integration: created and provisioned a new cumin master based on Buster:  integration-cumin.integration.eqiad.wmflabs
* 12:38 hashar: deployment-prep: created and provisioned a new cumin master based on Buster:  deployment-cumin.deployment-prep.eqiad.wmflabs
== 2020-06-19 ==
* 10:31 hashar: Triggering documentation generation for transferpy ([[phab:T253219|T253219]]): contint2001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/transferpy --change 602719,9
== 2020-06-18 ==
* 18:28 hashar: integration-castor03: remove labstore::traffic_shaping::egress: 100mbps  in horizon. It is now applied project wide via puppet.git # [[phab:T232644|T232644]] [[phab:T255371|T255371]]
* 17:39 James_F: Zuul: [mediawiki/services/push-notifications] Enable new node codehealth job [[phab:T240989|T240989]] [[phab:T251441|T251441]]
* 16:56 hnowlan: disabled non-docker cpjobqueue instance in deployment-prep
* 16:46 James_F: Docker: Publishing java8-sonar-scanner:0.6.1 for [[phab:T240989|T240989]]
== 2020-06-17 ==
* 19:46 brennen: Updating dev-images docker-pkg files on primary contint for jobrunner changes ([[phab:T246935|T246935]])
* 15:22 hashar: Reloading zuul for gerrit-test.wikimedia.org  {{Gerrit|Iaaa7e8797d779578a6a42f735bb7c809e29f415a}}
* 15:05 James_F: Private config for MediaModeration extension added on Beta Cluster for [[phab:T247943|T247943]]
* 14:53 hashar: Reloading Zuul for {{Gerrit|I5f10c7438a1514d705d1b0b8569c325305982366}}
== 2020-06-16 ==
* 21:03 Platonides: testing a [[phab:T230245|T230245]] fix
* 20:29 Platonides: done!
* 20:14 Platonides: livehacking MWExceptionHandler on deploy01
* 19:07 Krinkle: Done fiddling with deploy01
* 18:48 hashar: Reloading Zuul for {{Gerrit|Ibb8b8655f1837ced28783ef44f2b7b1e4a15e9ea}}
* 17:25 Krinkle: Fiddling on deploy01 with Task 230245
* 10:23 hashar__: Reloading Zuul to deploy {{Gerrit|I31d8c714e2b5c76d0cd823d22fac2f7bbbf583e5}} and {{Gerrit|I95ea11a5224fc52cde4b03e0adf4673d683332f8}}
* 02:55 Krinkle: It appears logstash-beta.wmflabs.org is down/unresponsive
== 2020-06-15 ==
* 10:32 James_F: Zuul: Rename CI allow list [[phab:T255378|T255378]]
== 2020-06-14 ==
* 19:22 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/605417
== 2020-06-13 ==
* 21:03 Krinkle: Done fiddling
* 20:43 Krinkle: Fiddling on deployment-deploy01 to investigate Task 230245
== 2020-06-12 ==
* 23:01 Krinkle: Done fiddling on deployment-deploy01
* 19:04 Krinkle: Fiddling on deployment-deploy01 to investigate [[phab:T230245|T230245]]
* 16:47 addshore: deployed a bunch of phpmetric jobs to jenkins
* 16:23 James_F: Zuul: Factor out generic-node10-rundoc-docker into node10-rundoc-docker template
* 15:57 James_F: Docker: Publishing new mediawiki-phan helper image version
* 15:13 James_F: Zuul: Update trusted user list per zeljkof
* 08:15 awight: Granted dewiki-beta sysop to Adamw via commandline
* 07:57 awight: Update QuickSurveys config
== 2020-06-11 ==
* 23:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604892
* 23:39 Reedy: deployed quibble-donationinterface-REL1_31-php70-docker with https://gerrit.wikimedia.org/r/604892
* 23:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604719 [[phab:T255096|T255096]]
* 14:44 Reedy: rm -rf doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-libs-PasswordBlacklist [[phab:T254799|T254799]]
* 14:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604707
* 13:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604706 [[phab:T254799|T254799]]
* 10:37 Urbanecm: Run `update page set page_content_model="json" where page_content_model = "CollaborationListContent" OR page_content_model = "CollaborationHubContent";` at beta enwiki ([[phab:T255107|T255107]])
* 02:36 dpifke: Cherry-picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/603550/ and https://gerrit.wikimedia.org/r/#/c/labs/private/+/604498/ into deployment-prep puppet to beta-test new XHGui ([[phab:T180761|T180761]]).
== 2020-06-10 ==
* 15:18 James_F: Zuul: [mediawiki/extensions/QuickSurveys] Stop running ruby jobs [[phab:T177261|T177261]]
* 14:37 James_F: Docker: Publishing php-ast:1.0.6 and related images [[phab:T255004|T255004]]
* 06:23 James_F: Docker: Publishing mediawiki-phan:0.6.0
* 06:08 James_F: Zuul: Drop extension-seccheck template, unused
* 06:06 James_F: Zuul: [mediawiki/extensions/Petition] Replace seccheck with phan
* 06:05 James_F: Zuul: [mediawiki/extensions/TEI] Replace seccheck with phan
* 06:01 James_F: Zuul: [mediawiki/extensions/Sudo] Replace seccheck with phan
* 05:59 James_F: Zuul: [mediawiki/extensions/ChessBrowser] Replace seccheck with phan
== 2020-06-09 ==
* 17:16 Krinkle: Messing with deploy01 in Beta to debug [[phab:T230245|T230245]]
* 15:56 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604058
* 14:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604018
== 2020-06-08 ==
* 20:47 James_F: Zuul: [mediawiki/extensions/MassEditRegex] Replace seccheck with phan
* 20:46 James_F: Zuul: [mediawiki/extensions/DisableAccount] Replace seccheck with phan
* 20:42 James_F: Zuul: [mediawiki/extensions/RelatedSites] Replace seccheck with phan
* 20:33 James_F: Zuul: [mediawiki/extensions/PageNotice] Replace seccheck with phan
* 18:38 hauskatze: Rebooting deployment-logstash0[2,3]
* 18:29 James_F: Zuul: [mediawiki/extensions/AdminLinks] Add phan
* 18:27 hauskatze: Deployment-prep: Rebooting deployment-logstash2, looks hung and cannot be reached
* 17:55 James_F: Docker: Publishing operations-puppet image, now with upgraded shellcheck [[phab:T254480|T254480]]
* 15:26 James_F: Zuul: Add Adham Khatean to CI allow list
== 2020-06-06 ==
* 15:19 Amir1: created deployment-mailman01 in beta cluster for testing upgrade to mailmain 3.1 ([[phab:T52864|T52864]] and [[phab:T130554|T130554]])
== 2020-06-05 ==
* 15:43 hashar: Building image docker-registry.discovery.wmnet/releng/operations-puppet:0.7.2 https://gerrit.wikimedia.org/r/#/c/integration/config/+/602710/
* 07:45 hashar: jenkins: upgrading ansicolor plugin 0.6.3 > 0.7.0  # [[phab:T254408|T254408]]
== 2020-06-04 ==
* 22:53 James_F: Docker: Publishing Quibble 44 images
* 20:31 James_F: Zuul: [mediawiki] Drop seccheck experimental, done in phan proper
* 17:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/602435
* 16:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/602424 https://gerrit.wikimedia.org/r/602425
* 11:21 hashar: Built docker-registry.discovery.wmnet/releng/doxygen:0.8.0 # [[phab:T253793|T253793]]
== 2020-06-03 ==
* 19:47 James_F: Zuul: [mediawiki/extensions/MachineVision] Add codehealth.
* 18:57 hauskatze: GitHub: Mirror deleted: mediawiki-extensions-AzharAuth {{!}} refs. [[phab:T198601|T198601]]
* 16:20 James_F: Docker: Publishing docker-registry.discovery.wmnet/releng/quibble-buster-php72
* 16:18 brennen: Updating dev-images docker-pkg files on primary contint
== 2020-06-02 ==
* 21:16 Reedy: ran `mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikimedia apiportalwiki api.wikimedia.beta.wmflabs.org` [[phab:T254185|T254185]]
* 21:11 Reedy: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/601808/ onto deployment-puppetmaster04 [[phab:T254185|T254185]]
* 13:35 hashar: Built releng/helm-linter:0.2.3 https://gerrit.wikimedia.org/r/#/c/integration/config/+/601707/
* 10:31 Lucas_WMDE: wikidata/query/gui-deploy: recreated production branch (commit {{Gerrit|9cab76d1281a1198766b485c815bc753d4171e94}}), pointed HEAD to production, deleted master branch, cf. [[phab:T254027|T254027]]
* 10:26 Lucas_WMDE: but also, correctly I think, deleted deploy and production branches on wikidata/query/gui ([[phab:T254027|T254027]])
* 10:25 Lucas_WMDE: CORRECTION accidentally deleted deploy and production branches on wikidata/query/gui-deploy, production should not have been deleted there ([[phab:T254027|T254027]])
* 10:20 Lucas_WMDE: wikidata/query/gui: deleted deploy and production branches, cf. [[phab:T254027|T254027]]
* 08:27 legoktm: reload zuul config for https://gerrit.wikimedia.org/r/600403
== 2020-06-01 ==
* 20:49 James_F: Zuul: Disabled new mediawiki-quibble-apitests-vendor-docker for Math as it doesn't pass [[phab:T254031|T254031]]
* 20:31 James_F: Zuul: EventBus now depends on EventStreamConfig for non-phan too [[phab:T229863|T229863]]
* 17:47 James_F: Zuul: EventBus now depends on EventStreamConfig [[phab:T229863|T229863]]
* 17:02 James_F: layout: Add CI for operations/debs/kubeyaml using debian-glue
== 2020-05-30 ==
* 16:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/599963
* 13:18 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki/php-master$ mwscript maintenance/createAndPromote.php --wiki=fawiki --bureaucrat --force --interface-admin --sysop Ladsgroup (Part of [[phab:T253291|T253291]])
== 2020-05-29 ==
* 22:46 James_F: Zuul: Archive mediawiki/extensions/PopupPages [[phab:T251000|T251000]]
* 18:42 hauskatze: gerrit: replication start mediawiki/extensions/WikiShare --wait refs. [[phab:T250400|T250400]]
* 18:17 hauskatze: GitHub: Deleted mirror wikimedia/mediawiki-extensions-PopupPages refs. [[phab:T251000|T251000]]
* 14:36 hashar: Built  Successfully tagged docker-registry.discovery.wmnet/releng/java8:0.6.4
* 09:09 hashar: Updating all Jenkins jobs: tox -e jenkins-jobs -- --flush-cache --conf jenkins_jobs.ini update --workers 2 ./jjb
* 07:18 hashar: Deleted last php5.6 job hurrah and thanks James ! https://integration.wikimedia.org/ci/job/composer-php56-docker/ # [[phab:T224906|T224906]]
* 07:18 hashar: Deleted last php5.6 job hurrah and thanks James ! https://integration.wikimedia.org/ci/job/composer-php56-docker/
== 2020-05-28 ==
* 17:52 James_F: Zuul: [integration/docroot] Switch from php56 to php70
* 16:10 James_F: Zuul: Move MediaModeration to production section and enforce [[phab:T247943|T247943]]
* 13:31 hashar: Rolling back Xunit Jenkins plugin  # [[phab:T194096|T194096]]
== 2020-05-27 ==
* 21:55 James_F: Nicely restarting Jenkins for xunit plugin upgrade.
* 15:33 James_F: Zuul: Re-pulling config forwards ~2 weeks on contint2001; forgotten in [[phab:T224591|T224591]]?
* 13:29 ottomata: start failed puppetdb on deployment-puppetdb03
== 2020-05-26 ==
* 12:26 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.1 # [[phab:T251869|T251869]]
== 2020-05-25 ==
* 23:08 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/598547
* 14:21 hashar: Built image releng/doxygen:0.7.0 for Doxygen 1.8.17 # [[phab:T242155|T242155]]
== 2020-05-21 ==
* 16:42 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597825
* 16:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597820
* 16:19 James_F: Zuul: [mediawiki/extensions/Bootstrap] Switch down to quibble-composer for now.
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597810
== 2020-05-20 ==
* 22:13 James_F: Docker: Publishing new zuul-cloner image with zuul-clonemap.yaml in it [[phab:T252955|T252955]]
* 21:18 James_F: Docker: Publishing new npm-test images with npm-install-dev.py in them [[phab:T252955|T252955]]
== 2020-05-19 ==
* 16:33 James_F: Docker: Publishing helm-linter:0.2.2 [[phab:T252428|T252428]]
== 2020-05-18 ==
* 21:36 ottomata: started  puppetdb on deployment-puppetdb03.  it was not running
== 2020-05-17 ==
* 01:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/596848
== 2020-05-15 ==
* 19:30 James_F: Zuul: Collapse extension-quibble-only-selenium users back into extension-quibble now it's split
* 19:27 James_F: Zuul: Always split out the selenium run into their own jobs
* 19:23 James_F: Zuul: Add CI for mediawiki/extensions/FlexiSkin
* 18:21 James_F: Zuul: Add TwoColConflict to the gate [[phab:T251276|T251276]]
* 17:48 James_F: Zuul: Add BetaFeatures to the gate [[phab:T249674|T249674]]
* 17:21 James_F: Zuul: Add CI for mediawiki/extensions/PageExchange [[phab:T252473|T252473]]
== 2020-05-13 ==
* 18:43 hashar: Building releng/helm-linter:0.2.1 for helm upgrade # [[phab:T252428|T252428]]
* 18:35 James_F: Zuul: [labs/toollabs] Stop running the glue job for Jessie [[phab:T247496|T247496]]
* 16:01 James_F: Zuul: Mark mediawiki/debian as archived in CI [[phab:T240521|T240521]]
== 2020-05-12 ==
* 22:23 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/594823
* 22:14 marxarelli: created 4 jenkins jobs with jjb for https://gerrit.wikimedia.org/r/c/integration/config/+/594823
* 20:10 cdanis: reverting to standard php7.2-fpm on deployment-mediawiki-07
* 19:34 cdanis: installing patched version of php7.2-fpm on deployment-mediawiki-07
* 18:44 brennen: cloning all (public) wmf repos from gerrit replica to a local system for number crunching
* 16:02 James_F: Zuul: Manually running fabric against contint1001 to add Privacybatm to the CI allow list [[phab:T248256|T248256]]
* 15:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/595968 [[phab:T248256|T248256]]
== 2020-05-11 ==
* 19:17 ottomata: set profile::trafficserver::tls::inbound_tls_settings.prioritize_chacha: 1, profile::cache::varnish::frontend::fe_vcl_config.large_objects_cutoff: 262144 in prefixpuppet deployment-cache-text horizon hiera to fix puppet on deployment-cache-text06
* 19:09 ottomata: started dead puppetdb service on deployment-puppetdb03
== 2020-05-10 ==
* 16:23 paladox: ssh -p 29418 paladox@gerrit.wikimedia.org gerrit create-project labs/tools/gdrive-to-commons.git --owner {{Gerrit|204871d01b8ce076f6559767264c7af2f8b737c7}} --empty-commit - per https://www.mediawiki.org/w/index.php?title=Gerrit%2FNew_repositories%2FRequests%2FEntries&type=revision&diff=3844303&oldid=3844168
== 2020-05-07 ==
* 19:52 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/595026
* 16:20 James_F: Deleted integration-cumin.integration.eqiad.wmflabs for last part of [[phab:T236576|T236576]]
* 16:17 James_F: Shut down integration-cumin-02, non-functional buster cumin host.
* 16:15 James_F: Shutting off integration-cumin for [[phab:T236576|T236576]]
* 16:14 James_F: Dropped integration-cumin (172.16.1.103) from WMCS integration cumin master list
* 16:11 James_F: Shutting down integration-agent-stretch-1001, unused.
* 16:07 hashar: integration-agent-stretch-1001 : clearing out /var/cache/pbuilder/build
* 15:43 James_F: Added integration-cumin-01 (172.16.1.185) to WMCS integration cumin master list
* 15:38 Daimona: foreachwiki extensions/AbuseFilter/maintenance/updateVarDumps.php --print-orphaned-records {{!}} [[phab:T246539|T246539]]
* 15:12 hashar: Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/ # [[phab:T236576|T236576]]
* 15:12 hashar: Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/
* 15:11 Daimona: mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=enwiki --print-orphaned-records {{!}} [[phab:T246539|T246539]]
* 12:34 mutante: removing role::labs::lvm::srv from deployment servers since this is now included in role:deployment_server and should neve have been a role in the first place
* 12:07 mutante: - puppet still broken on deployment_servers due to unrelated pre-existing issues, also no alerts about it in shinken
* 12:04 mutante: - puppet broken on deployment_servers - fix deployed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/594932
== 2020-05-06 ==
* 20:54 James_F: Created integration-cumin-01 instance in WMCS based on stretch for final part of [[phab:T236576|T236576]]
* 20:20 hashar: Running jjb for all Jenkins jobs to drop ansicolor definition (now globally enabled) https://gerrit.wikimedia.org/r/#/c/integration/config/+/594716/ # [[phab:T233688|T233688]]
* 17:01 James_F: Deleting integration-agent-puppet-docker-1001 from WMC for final stage of [[phab:T250502|T250502]].
* 13:50 hashar: Jenkins: enable xterm ansi color map for all builds via https://integration.wikimedia.org/ci/configure # [[phab:T233688|T233688]]
* 13:29 hashar: Updating Jenkins ansicolor from 0.6.2 to 0.6.3 # [[phab:T233688|T233688]]
* 01:03 thcipriani: integration-cumin:sudo cumin -p 50 'name:docker' 'docker pull docker-registry.wikimedia.org/wikimedia-stretch:latest'
== 2020-05-05 ==
* 22:48 James_F: jjb: Updating all jobs to master.
* 20:33 James_F: jjb: Bumped wmf-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test.
* 20:23 James_F: Bumped mediawiki-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test.
* 19:05 James_F: Docker: Publishing quibble 0.0.43 images.
* 15:02 James_F: Zuul: Change pipeline trigger for train-deploy-notes job
* 14:36 James_F: Zuul: Install service pipeline for mediawiki/services/push-notifications [[phab:T251901|T251901]]
== 2020-05-04 ==
* 18:31 Krinkle: Un-stall beta cluster puppetmaster (100 patches behind due to merge conflict)
* 17:43 James_F: Shutting off integration-agent-puppet-docker-1001 in WMCS for [[phab:T250502|T250502]]
* 17:41 James_F: Dropping integration-agent-puppet-docker-1001 from jenkins for [[phab:T250502|T250502]]
* 17:27 James_F: Depooling integration-agent-puppet-docker-1001 from running puppet CI [[phab:T250502|T250502]]
* 17:10 James_F: Pooling integration-agent-puppet-docker-1002 to run puppet CI [[phab:T250502|T250502]]
* 16:36 James_F: Docker: [composer-php70] Drop composer-install-dev-only, unused
* 16:13 hashar: Cleaning git repositories on deployment-deploy01
== 2020-05-01 ==
* 19:44 James_F: Zuul: [operations/puppet] Add operations-puppet-tests-buster-docker-test as experimental [[phab:T250502|T250502]]
* 19:40 James_F: jjb: Dropping operations-puppet-tests-stretch-docker, not triggered for over a month
* 19:20 James_F: Added self to integration sudoers policy
* 18:56 James_F: Created integration-agent-puppet-docker-1002 based on stretch for [[phab:T250502|T250502]]
* 18:43 James_F: Deleting integration-trigger-01 from WMCS for [[phab:T247109|T247109]]
* 18:41 James_F: Deleting integration-slave-jessie-1002 and integration-slave-jessie-1004 instances from WMCS for [[phab:T236576|T236576]]
== 2020-04-30 ==
* 19:49 hashar: deployment-logstash2  emptied /var/log/demon.log
* 18:32 James_F: Zuul: Install CI for mediawiki/services/texvcinfo [[phab:T249908|T249908]]
== 2020-04-29 ==
* 23:59 James_F: Zuul: [mediawiki/extensions/Sentry] Move out of deployed section [[phab:T91649|T91649]]
* 16:14 James_F: Zuul: CI for wikimedia/meet-accountmanager [[phab:T251425|T251425]]
* 16:03 James_F: Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for [[phab:T236576|T236576]]
* 16:03 James_F: Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for [[phab:T236576|T236576]]
== 2020-04-28 ==
* 18:20 James_F: Zuul: [mediawiki/extensions/reCaptcha] Mark as archived [[phab:T250641|T250641]]
* 10:00 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2020-04-27 ==
* 23:33 James_F: Zuul: Migrate WDQS jobs to Docker [[phab:T247123|T247123]]
* 21:26 James_F: Pushed Jeena's new version of wikimedia-portals-build based on Docker for [[phab:T242909|T242909]]
* 19:45 James_F: Zuul: Run quibble jobs for mediawiki/extensions/GroupWhitelist
* 16:58 Krinkle: krinkle@contint1001$ Replay zuul event for luasandbox@3.0.3 tag to publish docs at https://doc.wikimedia.org/mediawiki-php-luasandbox/3.0.3/  - [[phab:T186302|T186302]]
* 16:57 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/592706 (
== 2020-04-21 ==
* 15:05 Krinkle: install 'qemu-system-x86' package on integration-agent-qemu-1001
* 14:48 Krinkle: Creating integration-agent-qemu-1001 to experiment with VM-based CI jobs – [[phab:T250808|T250808]]
== 2020-04-20 ==
* 19:47 halfak: Deploying ORES {{Gerrit|514f94a}}
* 19:16 halfak: Deploying ORES {{Gerrit|ac2eb2f}}
* 18:52 halfak: Deploying ORES {{Gerrit|a5a5bce}}
* 17:07 James_F: Zuul: Unarchive the WikiShare extension [[phab:T250400|T250400]]
* 16:55 James_F: Zuul: Archive the TemplateInfo extension [[phab:T250591|T250591]]
* 15:41 halfak: Deploying ORES {{Gerrit|5d977f4}}
== 2020-04-19 ==
* 23:23 Krinkle: Change Jenkins Console Section pattern for quibble/Start to exclude fresnel user command
* 23:23 Krinkle: Fix Jenkins Console Section pattern for "setup quibble-fresnel"
== 2020-04-17 ==
* 21:21 hashar: Building Docker image  releng/node10-portals:0.1.2 for longma
* 21:11 James_F: Zuul: Add Scribunto to the gate after only five years of talking about it [[phab:T125050|T125050]]
* 19:59 James_F: Docker: Rebuild the quibble world for 0.0.42
* 19:01 longma: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/c/integration/config/+/589414
* 17:39 James_F: Docker: Rebuilding the PHP world to have native zip and unzip [[phab:T250496|T250496]]
* 15:49 brennen: Updating dev-images docker-pkg files on contint2001 for [[phab:T231864|T231864]]
== 2020-04-16 ==
* 20:01 hashar: gerrit: operations/debs/doxygen : renamed branch debian/buster-backports to debian/buster-wikimedia
* 17:43 James_F: Zuul: Add in-wikimedia-production to production items and assert shape
* 17:01 James_F: Zuul: Add Priyanshu varshney in CI whitelist
== 2020-04-15 ==
* 23:52 thcipriani: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/589159
* 19:59 dpifke: navtiming on deployment-webperf11 reverted to master.
* 19:46 dpifke: Testing patch 587775 to navtiming in beta cluster; will revert when done.
* 19:01 James_F: Zuul: Marking the ScanSet extension as archived for CI [[phab:T250240|T250240]]
== 2020-04-14 ==
* 17:00 James_F: Zuul: Archiving MagicNumberedHeadings [[phab:T248758|T248758]]; WikipediaExtracts [[phab:T248760|T248760]]; OracleTextSearch [[phab:T249656|T249656]].
* 16:57 James_F: Zuul: Adding Lex Nasser to the CI whitelist
== 2020-04-13 ==
* 21:20 James_F: Zuul: Archive the MSSQLBackCompat extension [[phab:T248965|T248965]]
* 21:03 James_F: Zuul: [mediawiki/extensions/CirrusSearch] Support standalone
* 18:13 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/588444
* 18:04 James_F: Restarting deployment-mediawiki-09 too.
* 18:03 James_F: Re-enabling puppet agent on deployment-mediawiki-09; disabled for 7 weeks with no comment.
* 17:50 James_F: Zuul: Add CI for labs/tools/github-pr-closer
* 17:39 James_F: Restarting deployment-mediawiki-07 to see if it comes back.
== 2020-04-11 ==
* 15:07 Krenair: Migrated from deployment-cache-text05 (stretch) to deployment-cache-text06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance - [[phab:T250006|T250006]]
* 14:52 Krenair: Migrated from deployment-cache-upload05 (stretch) to deployment-cache-upload06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance which coincidentally would turn one year old tomorrow
== 2020-04-10 ==
* 23:01 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/588054/
* 19:28 longma: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/c/integration/config/+/588029
* 18:40 James_F: Docker: Publishing composer-php70:0.2.0 and cascade.
* 17:12 twentyafterfour: deploying docker image on contint2001 with `tox -e fabric -- deploy_docker`
* 17:08 James_F: Docker: Publishing release-notes:0.0.5 on contint2001
* 16:46 hashar: contint1001: deleting docker-pkg maintained images. They are in the registry anyway.  # [[phab:T224591|T224591]]
* 07:53 Amir1: ladsgroup@deployment-deploy01:~$ mwscript createAndPromote.php --wiki=enwiki --custom-groups=checkuser --force Ladsgroup
* 00:03 Krinkle: Fix Jenkins Console Section for "setup quibble" to not consume output of 'mwext-php72-phan-seccheck-docker' jobs
== 2020-04-09 ==
* 19:50 James_F: Zuul: Adjust parsoid CI config and parsoid/deploy CI config
* 17:17 James_F: Zuul: Move last production extensions to use vendor
* 16:55 James_F: Zuul: Split Wikibase-dependent repos into non-selenium & selenium jobs
* 16:49 James_F: Zuul: parameter_functions: Drop dependencies for three archived extensions
* 16:32 James_F: Zuul: [mediawiki/extensions/TemplateStyles] Move to vendor
* 10:07 hashar: gerrit: created empty repository wikimedia/fundraising/crm/drush/vendor owned by fundraising. # [[phab:T249547|T249547]]
== 2020-04-08 ==
* 21:28 James_F: layout: [mediawiki/extensions/Scribunto] Standalone, always [[phab:T225068|T225068]]
* 21:18 James_F: layout: [mediawiki/extensions/Scribunto] Standalone, experimentally [[phab:T225068|T225068]]
* 20:25 James_F: Docker: Publishing quibble 0.0.41 images
* 19:44 hashar: Tag Quibble 0.0.41 {{Gerrit|cc2b408}} # [[phab:T234738|T234738]] [[phab:T225068|T225068]]
* 16:04 James_F: Updating all docker JJB jobs with seccomp over-ride for [[phab:T236675|T236675]]
* 16:01 James_F: Updating composer-package-php72-docker with seccomp over-ride for final [[phab:T236675|T236675]] testing
* 16:00 James_F: Updating wikipeg-npm-php-docker with seccomp over-ride for final [[phab:T236675|T236675]] testing
* 15:31 hashar: zuul-merger stopped on contint2001. It is being reimaged to Buster!  # [[phab:T224591|T224591]]
* 14:53 mutante: contint2001 down for reimaging
== 2020-04-07 ==
* 10:11 Lucas_WMDE: manually killed a Jenkins job for a gate-and-submit of a non-latest patch set
* 03:46 James_F: Zuul: [FundraisingEmailUnsubscribe] Disable MW-master testing
* 01:19 James_F: Zuul: FR voting for ContributionTracking & FundraisingEmailUnsubscribe
* 00:57 James_F: Zuul: Add experimental jobs for Fundraising extensions using PHP70/REL1_31
== 2020-04-06 ==
* 21:23 James_F: Zuul: Run CI for mediawiki/extensions/S5SlideShow
* 21:23 James_F: Zuul: Run CI for labs/tools/articles-needing-links
* 21:23 James_F: Zuul: Run CI for MathSearch [[phab:T117659|T117659]]
* 16:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586394
* 16:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586388
* 16:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586387
== 2020-04-04 ==
* 15:27 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/585884
== 2020-04-03 ==
* 20:30 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246953|T246953]]
* 19:02 longma: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/584701
== 2020-04-02 ==
* 22:53 James_F: Updated ooui-npm-special-node10-plus-composer-docker with COMPOSER_GITHUB_OAUTHTOKEN injected [[phab:T248387|T248387]]
* 22:00 James_F: Docker: [composer-php70] Inject GitHub OAuth token into composer runtime [[phab:T106452|T106452]] [[phab:T248387|T248387]]
* 21:46 hashar: deployment-deploy02: apt-get autoremove --purge
* 17:07 liw: testing upcoming scap on beta
* 16:44 James_F: Deleted integration-agent-jessie-docker-1001 from WMCS [[phab:T249226|T249226]]
* 16:43 James_F: Deleted integration-agent-jessie-docker-1001 from jenkins [[phab:T249226|T249226]]
* 16:43 James_F: Shut off integration-agent-jessie-docker-1001 ahead of removal [[phab:T249226|T249226]]
* 16:15 James_F: Zuul: Stop running Math tests with every change in Wikibase [[phab:T249235|T249235]]
* 15:52 hashar: deployment-deploy01:  sudo chown jenkins-deploy:wikidev /srv/packages/db/*
* 15:20 hashar: Updated https://integration.wikimedia.org/ci/job/scap-beta-deb # [[phab:T249225|T249225]]
* 13:49 hashar: Add Jenkins bot wmf-insecte to #wikimedia-analytics # [[phab:T210271|T210271]]
== 2020-04-01 ==
* 21:40 hashar: gerrit: deleting old sandbox branches in mediawiki/core that were used for testing purpose # [[phab:T227159|T227159]]
* 17:07 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/585275 # [[phab:T249076|T249076]]
== 2020-03-31 ==
* 15:36 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T223643|T223643]]
* 14:45 James_F: Zuul: Add pipeline CI job for mediawiki/tools/cli [[phab:T248412|T248412]]
== 2020-03-30 ==
* 20:36 James_F: Reverting [[phab:T246358|T246358]] work on JJB to fix [[phab:T248891|T248891]].
* 19:18 andrewbogott: restarting puppetdb on deployment-puppetdb03 and restarted apache2 on deployment-puppetmaster04 but puppet runs still fail everywhere
* 17:48 James_F: Zuul: [operations/software/censorship-monitoring] Switch to custom job [[phab:T248002|T248002]]
* 17:41 James_F: Docker: Publishing tox-censorshipmonitoring:0.1.0 [[phab:T248002|T248002]]
* 17:31 James_F: jjb: Deployed 'assert-no-mediawiki-errors' to all quibble jobs [[phab:T246358|T246358]]
* 17:12 James_F: jjb: Add 'assert-no-mediawiki-errors' to mediawiki-quibble-api-testing-vendor-*-php72-docker as a test [[phab:T246358|T246358]]
* 17:04 James_F: jjb: Add 'wikibase-client-docker' to wikibase-*-docker as a test [[phab:T246358|T246358]]
* 16:49 James_F: zuul: [operations/puppet] Use buster instead of stretch job
* 16:37 James_F: zuul: [operations/puppet] Add buster as experimental
* 16:37 James_F: https://gerrit.wikimedia.org/r/#/c/584628/
* 16:27 James_F: jjb: Created operations-puppet-tests-buster-docker
* 16:25 James_F: jjb: Sweitch operations-puppet-wmf-style-guide over to buster
* 16:17 James_F: Docker: Publish operations-puppet:0.7.0
* 16:16 James_F: [[phab:T248759|T248759]] layout: [mediawiki/extensions/ExtTab] Archive
* 16:07 James_F: [[phab:T242191|T242191]] layout: [labs/tools/qrcode-generator] Setup gate-and-submit-l10n
== 2020-03-29 ==
* 21:34 hashar: Restoring job mediawiki-core-doxygen-docker-publish deleted by Krinkle at 17:41
== 2020-03-25 ==
* 22:51 James_F: Zuul: Install CI for mediawiki/extensions/UserProtect
* 17:44 greg-g: Added Shay_Nowick to deployment-prep
== 2020-03-24 ==
* 22:56 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246921|T246921]]
* 16:16 James_F: Docker: Publishing new PHP images with GD installed [[phab:T236261|T236261]]
* 15:48 greg-g: added dkinzler@wikimedia.org to wmf-deployments (cf: [[phab:T248324|T248324]])
== 2020-03-23 ==
* 21:44 James_F: Zuul: Run all Kartographer tests alongside each patch to JsonConfig [[phab:T163274|T163274]]
* 18:43 James_F: jjb: Updated npm-node-6-docker for [[phab:T248326|T248326]] as an initial test
* 18:39 James_F: Zuul: [mediawiki/extensions/AControlImageLink] Run quibble
* 18:09 brennen: attempting to update docker-pkg files on contint1001, take 2
* 18:06 brennen: Updating docker-pkg files on contint1001 per James_F
* 17:58 James_F: Docker: Upgrade npm to v3.10.10 and cascade
* 15:43 James_F: Zuul: [mediawiki/extensions/ConfigManager] Add initial CI and dependency on BlueSpiceFoundation
* 15:38 James_F: Zuul: [labs/tools/bookreader] Add l10n bot pipeline [[phab:T241595|T241595]]
== 2020-03-20 ==
* 22:43 Krinkle: Click "Disable Publishing" for extension-TitleBlacklist, [[phab:T243175|T243175]], [[phab:T143162|T143162]]
* 22:22 James_F: Zuul: Unanchor the train-deploy-notes job regex filter 582111 [[phab:T243330|T243330]]
* 18:07 James_F: jjb: Manually disabled wikimedia-portals-build cronjob. [[phab:T213806|T213806]]
* 18:06 James_F: jjb: Manually disabled DockerJessie boxes for [[phab:T239981|T239981]]
* 18:03 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T248178|T248178]]
* 00:57 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581810
== 2020-03-19 ==
* 17:46 mutante: releases/releases-jenkins - closed firewall hole to port 80 for caching servers - kept it open just for envoy from the backends - ATS speaks https to them meanwhile
* 14:30 paladox: made labs-striker self owned
* 14:09 James_F: Zuul: Added QEDK to CI whitelist
== 2020-03-18 ==
* 19:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581047
* 18:41 James_F: Zuul: Add EventStreamConfig to gated extensions [[phab:T244521|T244521]]
* 18:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581005
* 16:50 James_F: Zuul: [mediawiki/services/parsoid] Drop pipeline
* 16:50 James_F: jjb: Dropped parsoid-pipeline-publish and trigger-parsoid-pipeline-publish
* 16:47 James_F: Zuul: Run Math tests for every change in Wikibase [[phab:T201496|T201496]]
* 03:31 Amir1: manually creating empty wb_terms
* 03:16 Amir1: dropping wb_terms table from wikidatawiki in beta cluster ([[phab:T219123|T219123]] [[phab:T219175|T219175]] [[phab:T208425|T208425]])
* 00:37 James_F: jjb: Updated parsoidsvc-parsertests-docker for JUnit compression.
== 2020-03-17 ==
* 19:49 James_F: Deleted deployment-dumps-puppetmaster02 for [[phab:T241719|T241719]]
* 19:48 James_F: Deleted deployment-mediawiki-parsoid10, no longer used.
* 19:45 James_F: Restarting deployment-prep's deployment-mediawiki-07 and deployment-mediawiki-09
* 18:59 thcipriani: re-enabling puppet agent on deployment-mediawiki-07 The last Puppet run was at Mon Feb 24 18:26:55 UTC 2020 (31711 minutes ago). Puppet is disabled. reason not specified -- nobody on the machine
* 16:31 James_F: Zuul: Run PHP74 for CirrusSearch again
* 16:18 James_F: Zuul: Run PHP74 for nonselenium jobs
== 2020-03-13 ==
* 19:38 James_F: jjb: Speculatively pushing mediawiki-fresnel-patch-docker with libgtk-3.0 for manual testing of [[phab:T247599|T247599]]
* 18:46 James_F: jjb: Make mwext-doxygen-publish publish to DOC_PROJECT not DOC_BASENAME [[phab:T246042|T246042]]
* 18:18 James_F: Docker: Pushing quibble-stretch 0.0.40-s1 and cascade.
== 2020-03-12 ==
* 17:21 James_F: Zuul: Make PHP74 tests voting for almost everything [[phab:T233012|T233012]]
* 16:59 James_F: Zuul: Make PHP74 tests voting for extensions [[phab:T233012|T233012]]
* 16:53 James_F: Zuul: [MediaWiki] Switch jsduck jobs over to node10 versions [[phab:T247536|T247536]]
* 16:43 James_F: Zuul: [MediaWiki] Run mediawiki-core-javascript-docker experimentally
* 13:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/579263
* 09:38 hashar: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ broken cause it can not ssh to  deployment-parsoid11.deployment-prep.eqiad.wmflabs # [[phab:T246854|T246854]]
* 08:17 hashar: integration: removing logstash Jenkins plugin from https://integration.wikimedia.org/ci/  {{!}} [[phab:T247036|T247036]] [[phab:T78705|T78705]]
* 00:34 James_F: Shutting down deployment-mediawiki-parsoid10 ahead of its removal [[phab:T246854|T246854]]
== 2020-03-11 ==
* 22:23 James_F: Zuul: [MediaWiki] Add PHP74 tests as voting for master [[phab:T233012|T233012]]
* 18:47 James_F: Zuul: Re-enabling PHP74 for all skins [[phab:T247214|T247214]]
* 17:30 James_F: Soft-restarting Jenkins to drop unused plugin.
* 16:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/578983
* 00:04 James_F: Restarting release jenkins for upgrade.
== 2020-03-10 ==
* 23:52 James_F: Restarting jenkins for upgrade.
* 14:58 James_F: Add CI for operations/software/atskafka [[phab:T237993|T237993]]
== 2020-03-09 ==
* 13:39 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2020-03-05 ==
* 17:08 Krinkle: Remove obsolete filters from "scap canary" dashboard, including obsolete "hhvm" query
* 15:25 James_F: Zuul: Add mediawiki-i18n-check template to mediawiki/services/mobileapps for [[phab:T246979|T246979]]
* 01:28 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246930|T246930]]
* 00:27 greg-g: removed cmcmahon, Coren, khorn, manybubbles, mobrovac, laner from users of integration
* 00:26 greg-g: added Jforrester as user and projectadmin to integration
* 00:23 greg-g: made Krenair a projectadmin in integration
== 2020-03-07 ==
* 13:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/577776
== 2020-03-06 ==
* 22:42 James_F: Raised executor count on contint1001 from 3 to 12 for [[phab:T247109|T247109]]
* 19:40 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/577653
* 19:33 marxarelli: creating 2 jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/577653
* 19:05 James_F: Manually pushing trigger-helloworldoid-pipeline-test over to contint1001 to test [[phab:T247109|T247109]]
* 18:56 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/577626
* 18:52 marxarelli: creating 2 jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/577626
* 15:42 hashar: contint1001: docker image prune -f
* 15:14 Reedy: unstuck beta auto deploy jobs
* 14:37 Urbanecm: Live debugging [[phab:T247078|T247078]] on deployment-mediawiki-07 ended
* 13:50 Urbanecm: Live debugging [[phab:T247078|T247078]] on deployment-mediawiki-07
== 2020-03-05 ==
* 23:11 Krenair: [[phab:T236576|T236576]] Shut down integration-puppetmaster01 in favour of integration-puppetmaster-02. /root and /home archives can be found in /root on the new system. All integration instances moved to use the new master. Old master to be deleted on 12th March in the absence of objections
== 2020-03-04 ==
* 22:15 James_F: Zuul: Archive CI for LiveTranslate [[phab:T196452|T196452]]
* 22:12 James_F: Zuul: Add CI for mediawiki/extensions/PushNotifications
* 20:59 hashar: deployment-puppetmaster04: /var/log/syslog /var/log/debug and /var/log/daemon.log are 5GB  filling /
* 20:59 James_F: Deleting deployment-parsoid09 for [[phab:T246395|T246395]]
* 19:53 hauskatze: GitHub: extension-LiveTranslate mirror deleted refs. [[phab:T196452|T196452]]
* 19:34 James_F: Shutting off deployment-parsoid09 for [[phab:T246395|T246395]]
* 19:23 James_F: Rebooting deployment-puppetmaster04
* 15:12 thcipriani: restarting zuul
* 01:34 James_F: Beta Cluster: Created deployment-parsoid11 in Horizon [[phab:T246854|T246854]] to test 576493.
* 01:23 Krenair: Made James a deployment-prep projectadmin
== 2020-03-03 ==
* 22:51 James_F: Zuul: Add gabrielchihonglee to CI allow list
* 22:43 James_F: Docker: Publish mediawiki-phan-testrun 0.1.4
* 22:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/575575
* 18:22 Daimona: Manually fixing eswikibooks DB rows with P10600 refs [[phab:T246539|T246539]]
* 15:23 Daimona: mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=eswikibooks {{!}} [[phab:T246539|T246539]]
== 2020-03-02 ==
* 21:14 Krenair: restarted php7.2-fpm on deployment-mediawiki-07
* 18:22 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246202|T246202]]
== 2020-02-29 ==
* 14:34 Reedy: !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/575689
* 11:02 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/575563
== 2020-02-28 ==
* 20:00 brennen: finished deleting wmf/1.34.0-wmf.* branches for [[phab:T244368|T244368]]
* 19:37 brennen: in process of deleting many old branches for [[phab:T244368|T244368]]
* 12:40 hauskatze: Migrated https://github.com/wikimedia/mediawiki-extensions-NearbyPages to Gerrit; GitHub repo now watches the Gerrit one
* 00:18 James_F: jjb: Deleting wikidata-query-gui-build
== 2020-02-27 ==
* 18:11 James_F: Reset mediawiki-quibble-composer-mysql-php72-docker to nominal [[phab:T246358|T246358]]
* 17:59 James_F: Updating mediawiki-quibble-composer-mysql-php72-docker to test probably-breaking requirement that error logs are empty [[phab:T246358|T246358]]
* 15:53 James_F: Docker: Publish mediawiki-phan-testrun:0.1.3 for [[phab:T226117|T226117]]
* 00:56 Krenair: Added edtadros to deployment-prep per request on IRC and https://en.wikipedia.org/w/index.php?title=User:ETadros_(WMF)/sandbox&diff=942815687&oldid=901073254 - wants access to eventlog05 in particular
== 2020-02-26 ==
* 21:45 James_F: Docker: Publishing mediawiki-phan-testrun:0.1.0 for [[phab:T226117|T226117]]
* 17:12 James_F: Zuul: Set up CI for mediawiki/extensions/MediaModeration [[phab:T246243|T246243]]
* 00:17 James_F: Zuul: Add a job to test new phan changes on MW core [[phab:T226117|T226117]]
== 2020-02-25 ==
* 23:18 Krenair: Added mayakpwiki to deployment-prep, user is already in the nda group and is working for WMF Product Analytics. Wants access to eventlog05
* 23:04 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/574890
* 21:00 hauskatze: gerrit: replication start mediawiki/extensions/MediaModeration --wait # New repo
* 20:43 James_F: Running a manual sync for Beta Cluster.
* 20:17 James_F: Beta jobs on Jenkins backed up; doing the restart dance.
* 19:07 James_F: Ran `sudo -u doc-uploader git -C /srv/docroot pull` on doc1001.
* 16:52 James_F: Zuul: [mediawiki/extensions/Popups] Publish PHP documentation [[phab:T242779|T242779]]
* 13:34 godog: set number of replicas for es indices to 0 on deployment-logstash03
== 2020-02-24 ==
* 20:59 James_F: doc1001: `sudo -u doc-uploader rm /srv/docroot/org/wikimedia/doc/MinervaNeue/master` except PHP bits from Doxygen; everything else is at mediawiki-skins-MinervaNeue/ now
* 20:45 James_F: Zuul: [mediawiki/skins/Vector] Publish PHP documentation [[phab:T242779|T242779]]
* 18:12 brennen: restarted jenkins on releases1001 and releases2001
* 15:23 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/574485
* 13:50 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 13:44 Urbanecm: deployment-prep: Run scap sync manually, it doesn't seem it was invoked by jenkins for some reason
== 2020-02-21 ==
* 23:27 James_F: Zuul: Allow l10n on analytics/wikistats2 for [[phab:T245805|T245805]]
* 19:38 thcipriani: remove a bunch of extensions from beta that were on disk but not in https://gerrit.wikimedia.org/rg/mediawiki/extensions/
* 01:02 James_F: Zuul: [operations/mediawiki-config] Run tox-docker always
* 00:59 James_F: Zuul: [operations/mediawiki-config] Run tox-docker as experimental
== 2020-02-20 ==
* 22:08 thcipriani: killing a whole bunch of backed-up beta-scap-eqiad jobs
* 21:50 James_F: deployment-deploy01 back online; Jenkins backlog clearing for beta cluster jobs.
* 21:47 James_F: Taking deployment-deploy01 temporarily offline
* 19:44 hauskatze: GitHub: RENAMED https://github.com/wikimedia/operations-software-atskafka to https://github.com/wikimedia/atskafka # [[phab:T245731|T245731]]
* 19:43 hauskatze: GitHub: Created https://github.com/wikimedia/operations-software-atskafka # [[phab:T245731|T245731]]
* 19:42 hauskatze: replication start operations/software/atskafka --wait
* 10:04 Reedy: postmerge jobs for mediawiki-config look stuck
* 10:03 Reedy: postmerge jobs look stuck
== 2020-02-19 ==
* 18:58 thcipriani: unblocking beta-scap-eqiad job (stuck for 18 hrs)
* 16:45 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/572488
== 2020-02-18 ==
* 22:01 brennen: adding dpifke to Gerrit Managers
* 21:23 James_F: Zuul: [mediawiki/extensions/Teahouse] Archive repo [[phab:T176349|T176349]]
== 2020-02-17 ==
* 12:34 hauskatze: extension-Teahouse@phab: Mirror deactivated # [[phab:T176349|T176349]]
* 12:34 hauskatze: extension-Teahouse@github: Mirror deleted # [[phab:T176349|T176349]]
== 2020-02-16 ==
* 21:24 hauskatze: AbuseFilterBypass@github: Deleted 'dev' branch {{!}} [[phab:T245363|T245363]]
* 21:24 hauskatze: AbuseFilterBypass@github: Switch default branch of mirror to 'master' {{!}} [[phab:T245363|T245363]]
== 2020-02-15 ==
* 23:52 legoktm: rebuilding tox docker image for https://gerrit.wikimedia.org/r/572453
* 22:21 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/572423
* 21:33 hauskatze: GitHub: Activated Travis CI at https://travis-ci.org/wikimedia/texvcjs for https://github.com/wikimedia/texvcjs per https://github.com/wikimedia/texvcjs/pull/34 and [[phab:T245344|T245344]]
* 20:25 Krenair: [[phab:T243226|T243226]] Shut off deployment-puppetdb02 and deployment-puppetmaster03 - will leave for a week before deletion
== 2020-02-14 ==
* 21:53 andrewbogott: moving deployment-puppetdb02 and deployment-puppetmaster03 off of cloudvirt1014 (which will be drained next week anyway)
* 20:33 brennen: Updating dev-images docker-pkg files on contint1001 to attempt a rebuild for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828
* 20:05 thcipriani: deployment-mwmaint01:rm lingering .~tmp~ directories from failed rsync's taking up space
* 19:31 thcipriani: deployment-snapshot01:sudo apt-get clean cleared 1.2G of space
* 17:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/571849
* 17:23 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/571849
* 17:20 thcipriani: updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/571849
== 2020-02-13 ==
* 22:19 James_F: Zuul: Add *.microsoft.com in email_whitelist
* 17:10 thcipriani: truncate deployment-puppetmaster04:/var/log/daemon.log -> 5G, 100% disk usage
* 15:58 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828
== 2020-02-12 ==
* 21:36 thcipriani: reloading zuul in an attempt to troubleshoot [[phab:T244690|T244690]]
* 20:22 James_F: Depooling integration-agent-docker-1001; it's being odd [[phab:T245054|T245054]]
* 17:11 brennen: restarting jenkins for updates
* 01:24 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T228910|T228910]]
== 2020-02-11 ==
* UKInEU: ssh -p 29418 paladox@gerrit.wikimedia.org replication start wikibase/vuejs-components --wait
* 21:27 hauskatze: github: Created https://github.com/wikimedia/wikibase-vuejs-components {{!}} [[phab:T240224|T240224]]
* 18:24 marxarelli: 1.35.0-wmf.19 was branched at {{Gerrit|3ac2e9ff045dc081317c9184dc83cf863fd518be}} for [[phab:T233867|T233867]]
* 16:44 James_F: Zuul: [mediawiki/extensions/Echo] Add experimental Postgres job
* 16:17 James_F: Terminated two multi-day running "parsoid-pipeline-publish" jobs
* 09:46 Daimona: foreachwiki extensions/AbuseFilter/maintenance/fixOldLogEntries.php {{!}} [[phab:T228655|T228655]]
* 09:43 Daimona: mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=enwiki {{!}} [[phab:T228655|T228655]]
== 2020-02-10 ==
* 23:42 James_F: Zuul: [parsoid/deploy] Make PHP tests voting [[phab:T239642|T239642]]
* 23:28 legoktm: restarting zuul
* 22:24 James_F: Zuul: Add tox for operations/homer/mock-private [[phab:T244690|T244690]]
* 19:48 James_F: Add mahveotm to CI whitelist
* 16:20 Daimona: mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=dewiki {{!}} [[phab:T228655|T228655]]
* 09:42 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571254
* 09:39 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/542071
* 09:25 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/571251
* 09:11 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/571247
* 09:02 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571242
* 08:44 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/539724
* 08:30 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571236
* 08:17 legoktm: updating tox-mysqld docker-pkg image https://gerrit.wikimedia.org/r/571234
* 07:43 legoktm: updating tox docker-pkg image for gettext https://gerrit.wikimedia.org/r/571019
* 07:20 legoktm: updating tox docker-pkg image (try #2) https://gerrit.wikimedia.org/r/571127
* 07:00 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/571017
== 2020-02-07 ==
* 23:20 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T244382|T244382]]
* 23:20 dpifke: puppetdb on deployment-puppetdb03 was killed by kernel OOM at Feb  7 09:50:29, per syslog.  I just ran `systemctl start puppetdb` on that host, to fix puppet issues in beta.
* 19:10 James_F: Zuul: [labs/tools/video-cut-tool-back-end] Add node10 CI [[phab:T244079|T244079]]
* 19:01 James_F: Zuul: [labs/tools/VideoCutTool] Switch from tox to node [[phab:T244079|T244079]]
* 16:57 halfak: deploying ores {{Gerrit|a6f4f14}}
== 2020-02-06 ==
* 20:27 James_F: Zuul: Run WikibaseCirrusSearch tests for every change in CirrusSearch [[phab:T221135|T221135]]
== 2020-02-05 ==
* 23:39 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/568140
* 16:56 thcipriani: Updating docker-pkg files on contint100 to deploy  https://gerrit.wikimedia.org/r/566915
* 16:40 thcipriani: restarting zuul
== 2020-02-04 ==
* 23:28 Krinkle: Puppet agent has not been able to run on deployment-mediawiki-07 for 2.5 days since 2 Feb 18:27
* 01:56 James_F: Zuul: [wikimedia/portals] Add service-pipeline configuration [[phab:T238747|T238747]]
* 01:42 James_F: Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker always [[phab:T243975|T243975]]
== 2020-02-03 ==
* 22:53 James_F: Zuul: Add CI for labs/tools/VideoCutTool [[phab:T244079|T244079]]
* 22:36 James_F: Zuul: New email address for Ricordisamoa
* 21:44 James_F: Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker experimentally [[phab:T243975|T243975]]
* 16:43 hauskatze: Gerrit: replication start mediawiki/extensions/AControlImageLink --wait # new repo
* 16:28 James_F: Zuul: Publish `npm run doc` for Vector [[phab:T242674|T242674]]
== 2020-02-02 ==
* afk: deployment-deploy01$ while read host; do ssh-keyscan -H "$host"; done < <(grep -vP '^#' /etc/dsh/group/mediawiki-installation {{!}} grep -vP '^$' ) {{!}} sudo -u jenkins-deploy tee /mnt/home/jenkins-deploy/.ssh/known_hosts
== 2020-01-31 ==
* 11:47 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php enwiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat  # make myself bureaucrat to make colleagues sysops
* 00:48 James_F: Restarting Jenkins.
== 2020-01-29 ==
* 23:55 James_F: zuul: [data-values/value-view] Migrate to node10, now that that passes [[phab:T228453|T228453]]
* 20:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/568600
* 00:05 James_F: layout: [mediawiki/extension/NSFileRepo] Restore tests now they're fixed [[phab:T196480|T196480]]
== 2020-01-26 ==
* 23:23 James_F: Zuul: Mark mediawiki/extensions/BlueSpiceMenues as archived [[phab:T240788|T240788]]
* 22:20 James_F: Zuul: Mark mediawiki/extensions/SemanticTasks as archived [[phab:T221549|T221549]]
* 18:51 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php ep[campus,online,coordinator,instructor] {{!}} [[phab:T241504|T241504]]
* 11:02 hauskatze: github - deleted mirror for extension-BlueSpiceMenues; refs. [[phab:T240788|T240788]]
== 2020-01-24 ==
* 21:15 halfak: deploying ores {{Gerrit|283f627}}
* 01:52 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/566911
* 01:45 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/566910
* 00:08 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/566895
* 00:02 James_F: Taking deployment-deploy01 offline to fix beta config update deadlock.
== 2020-01-23 ==
* 23:43 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/562365 and ttps://gerrit.wikimedia.org/r/562364
* 21:59 James_F: Drop REL1_32 testing, jobs, and pipelines, EOLed  [[phab:T242981|T242981]]
* 21:50 halfak: deploying ores {{Gerrit|039251f}} (reverting to last good state)
* 17:17 James_F: layout: [mediawiki/extensions/TagContent] Archive repo [[phab:T243093|T243093]]
* 15:31 halfak: deploying ores {{Gerrit|283f627a}}
== 2020-01-22 ==
* 20:19 James_F: Zuul: Adding TextExtracts as a dependency for TwitterCards [[phab:T242171|T242171]]
* 19:37 Krinkle: Restore php-apcu version on deployment-mediawiki-07 to …wmf1, ref [[phab:T236800|T236800]]
* 19:09 Krinkle: Installing newer version of php-apcu on deployment-mediawiki-07, ref [[phab:T236800|T236800]]
== 2020-01-21 ==
* 22:10 Krenair: deployment-prep added jbond as admin, [[phab:T243226|T243226]]
* 22:09 mutante: gerrit - rsyncing 'git' and 'plugin' data dirs and /var/lib/gerrit2/review_site/ from gerrit1001 to gerrit1002 WITH --delete [[phab:T239151|T239151]]
* 21:48 mutante: gerrit - rsyncing git data from gerrit1001 to gerrit1002 ([[phab:T239151|T239151]])
* 19:16 marxarelli: updating 105 jobs affected by https://gerrit.wikimedia.org/r/c/integration/config/+/566332
== 2020-01-20 ==
* 21:24 Krinkle: fix Jenkins Console Section patterns to account for change in Quibble output (no more colors and now "cmd: << Start/Finish x" instead of "cmd:x finished"
* 10:48 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/566001/ (node10 wikidata query gui)
* 10:10 addshore: the last zuul reload also pulled in "jjb: [java8-sonar-scanner] version bump to 0.5.4" .. [[phab:T238004|T238004]]
* 10:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/565985/
== 2020-01-17 ==
* 19:52 James_F: Docker: Publishing java8-sonar-scanner:0.5.4 on contint1001 [[phab:T238004|T238004]]
* 19:25 James_F: Zuul: [operations/debs/hhvm] Archive [[phab:T237038|T237038]]
* 19:01 James_F: Zuul: [AbuseFilterBypass] Install quibble for initial CI
* 18:53 James_F: Zuul: [skin-quibble] Disable PHP74 in gate for now [[phab:T242980|T242980]]
* 18:30 James_F: Zuul: [WebAuthn] Add phan dependency on OATHAuth
* 18:27 James_F: Zuul: [WikiSEO] Add phan and seccheck jobs
* 18:24 James_F: Zuul: [ShortDescription] Add phan and seccheck jobs
* 18:21 James_F: Zuul: [NativeSvgHandler] Add phan and seccheck jobs
* 18:19 James_F: Zuul: [Sofa] Add phan and seccheck jobs
* 01:31 Reedy: apt-get clean and purged some old kernels from deployment-logstash2 to free up ~1.7G space
== 2020-01-16 ==
* 20:09 James_F: Zuul: Add Aron Manning to CI whitelist
* 16:37 longma: bringing integration-slave-docker-1003 back online
* 16:30 longma: restarting integration-agent-docker-1003
* 16:24 longma: s/slave/agent
* 16:22 longma: bringing integration-slave-docker-1006 back online
* 16:15 longma: integration-slave-docker-1006:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '<nowiki>{</nowiki>print $3<nowiki>}</nowiki>')
* 14:28 Reedy: killed beta job to unblock beta-scap et al
* 09:31 Amir1: "delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" ([[phab:T219301|T219301]] [[phab:T225054|T225054]])
== 2020-01-15 ==
* 23:00 Reedy: deleted `contint2001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like [[phab:T134062|T134062]]
* 22:43 Reedy: deleted `contint1001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like [[phab:T134062|T134062]]
* 15:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/565042
* 01:15 marxarelli: deploy zuul layout change  https://gerrit.wikimedia.org/r/c/integration/config/+/564813
== 2020-01-14 ==
* 21:44 James_F: Zuul: Stop running JS tests for Parsoid and Parsoid-deploy [[phab:T242782|T242782]]
* 19:38 Krinkle: No syslog entries from php-fpm or puppet-agent on deployment-mediawiki-07, [[phab:T242659|T242659]], rebooting server
* 18:02 James_F: Zuul: Add JeremyNguyen to Jenkins whitelist [[phab:T235286|T235286]]
* 17:54 James_F: Zuul: [mediawiki/core] Stop publishing docker images for now [[phab:T242775|T242775]]
* 15:33 James_F: Depooling integration-agent-docker-1003 due to issues.
== 2020-01-13 ==
* 19:34 James_F: Zuul: [mediawiki/extensions/NativeSvgHandler] Add basic CI via quibble
* 16:50 James_F: Docker: Publishing java7-sonar-scanner:0.5.3 on contint1001
== 2020-01-10 ==
* 23:22 James_F: Zuul: Restore extension-coverage to projects with …-codehealth [[phab:T242444|T242444]]
* 18:35 bd808: Restarted zuul on contint1001 at 18:29Z; no logs since 2020-01-10 17:55:28,452
== 2020-01-09 ==
* 17:26 James_F: Zuul: Run sonar scanner analysis in codehealth pipeline for select Java projects [[phab:T238004|T238004]]
* 17:04 James_F: Docker: Publishing java7-sonar-scanner:0.5.2
* 16:29 James_F: Docker: Publish PHP 7.2.26-based images for [[phab:T241223|T241223]]
== 2020-01-08 ==
* 20:56 James_F: Docker: Publishing Quibble 0.0.40 images on contint1001 [[phab:T192167|T192167]] [[phab:T220586|T220586]] [[phab:T236222|T236222]] [[phab:T236680|T236680]]
* 20:42 James_F: Terminating some jobs Zuul doesn't know about.
* 20:41 mutante: contint1001 - restartted zuul service
* 20:39 James_F: Safe-restarting Jenkins to see if it fixes the lack of progress
* 19:57 thcipriani: root@integration-cumin:~# cumin -b1 'name:integration-agent-docker' 'sleep 1; git -C /srv/git/mediawiki/core.git fetch'
* 19:25 James_F: Tagged Quibble 0.0.40 @ {{Gerrit|ba365ab36fd84a87fcfb10e0910255e8b44432f1}} [[phab:T192167|T192167]] [[phab:T220586|T220586]] [[phab:T236222|T236222]] [[phab:T236680|T236680]]
* 16:26 halfak: deploiying ores {{Gerrit|039251f}}
== 2020-01-07 ==
* 18:18 James_F: Whitelist: Add RetroCraft and Ostrzyciel
* 18:16 James_F: Zuul: [mediawiki/extensions/SparkPost] Run phan
* 09:02 twentyafterfour: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/562417
== 2020-01-06 ==
* 20:43 James_F: Deleting https://doc.wikimedia.org/Wikibase/wmf/1.34.0-wmf.4 for [[phab:T240206|T240206]]
* 20:43 James_F: Deleting https://doc.wikimedia.org/Wikibase/wmf/1.33.0-wmf.23 for [[phab:T240206|T240206]]
* 20:11 James_F: Zuul: [SoftRedirector] Add phan and seccheck
* 19:45 James_F: Zuul: Add CI for new event schema repos [[phab:T240985|T240985]]
* 17:24 James_F: Zuul: Added ptrcnull to the CI whitelist
* 17:05 James_F: Zuul: [labs/tools/indic-wscontest] Provide CI with tox-docker [[phab:T241537|T241537]]
== 2020-01-02 ==
* 18:19 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php translate-proofr {{!}} [[phab:T241504|T241504]]
* 18:16 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php afttest-hide {{!}} [[phab:T241504|T241504]]
* 18:15 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php afttest {{!}} [[phab:T241504|T241504]]
* Away: Archive RecentActivityFeed extension [[phab:T241032|T241032]]
* Away: Update Revi's e-mail on the whitelist
* Away: Add Safan41 to the whitelist [[phab:T235286|T235286]]
* Away: Add BrandonXLF to the whitelist [[phab:T235286|T235286]]
* 10:40 dcausse: created missing elastic indices: [[phab:T241487|T241487]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 21:19, 22 October 2021

2021-10-22

  • 21:19 James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change T292910
  • 18:06 dancy: Updating deb packages on deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud
  • 00:31 James_F: Zuul: [DiscussionTools] Add EventLogging as dependency for Phan jobs

2021-10-21

  • 18:07 dduvall: replacing runner-1001 with new instance (T293835)
  • 09:49 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test T292728

2021-10-19

  • 22:24 brennen: gitlab-test: deleted unused gitlab-runner-test
  • 22:20 brennen: adding Dzahn and Aokoth to gitlab-test and gitlab-runners projects, cleaning up contractor access
  • 21:20 brennen: Updating dev-images docker-pkg files on primary contint for php 8 images (T283426)
  • 18:00 James_F: Zuul: [mediawiki/services/function-schemata] Switch from pipeline to direct CI
  • 11:32 hashar: Archiving integration/gear.git , no more used after Python gear 0.16.0 has been released # T289512

2021-10-18

  • 16:26 Lucas_WMDE: killed one beta-scap-sync-world job that seemed stuck for a long time
  • 14:43 James_F: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer T290274
  • 09:41 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # T248676

2021-10-16

  • 16:47 Lucas_WMDE: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T293070, based on T271808#6739578
  • 01:17 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/730949

2021-10-15

  • 16:14 brennen: gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for T292372

2021-10-14

2021-10-13

  • 23:12 bd808: Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors.
  • 16:48 brennen: Updating dev-images docker-pkg files on primary contint for T293277
  • 11:56 hashar: integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes'

2021-10-12

2021-10-11

2021-10-08

  • 21:17 hashar: Purging Docker images on all CI agents
  • 20:00 hashar: Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620
  • 19:23 hashar: Building CI Docker images for Quibble 1.1.1
  • 19:08 hashar: Tag quibble 1.1.1 @ b54af2aa60
  • 19:07 brennen: gitlab1001: restarting to check correct session_duration value of 604800
  • 11:01 hashar: Rebuilding CI Docker images based on Bullseye to include apt security updates

2021-10-07

  • 22:12 James_F: Zuul: [design/codex] Provide initial CI, node14-browser suite
  • 17:07 James_F: Zuul: [mediawiki/extensions/ParsoidBatchAPI] Mark as archived T252744
  • 13:29 hashar: restarting CI Jenkins for git plugin update
  • 13:14 hashar: Upgraded CI Jenkins on contint2001
  • 13:10 hashar: Upgraded releases Jenkins from 2.289.2 to 2.303.2
  • 13:04 hashar: Updating Jenkins jobs for Quibble 1.1.0 # T291549
  • 04:46 Krinkle: Temporarily change `mediawiki-fresnel-patch-docker` job change back to quibble-fresnel:1.0.1 ref T292684
  • 04:41 Krinkle: Reconfigure `mediawiki-fresnel-patch-docker` job to revert quibble change back to quibble-fresnel:1.0.1
  • 01:06 Reedy: unstick beta ci jobs

2021-10-06

  • 11:19 hashar: Tag Quibble 1.1.0 @ 3dabcab

2021-10-05

2021-10-04

  • 21:09 brennen: finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 (T292256)
  • 21:02 brennen: gitlab-runners: upgrading each shared runner to 14.3.2
  • 18:12 James_F: Docker: Publish composer 2.x-based images for T279857
  • 16:46 James_F: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
  • 15:54 James_F: Zuul: Add Lindsay Wardell personal e-mail to CI allow list
  • 14:31 brennen: gitlab1001: upgrading to 14.3.2 (T292256)

2021-09-30

2021-09-29

  • 15:52 hashar: Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # T277500
  • 15:03 brennen: gitlab: pausing shared runners while working through access-level implications

2021-09-27

2021-09-24

2021-09-23

2021-09-22

  • 15:47 Amir1: set row format for text table in simplewiki to compressed
  • 15:22 Amir1: restart mysql on db07 to add performance schema

2021-09-21

  • 16:26 dduvall: deleted wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to a branch cut job gone awry. re-running job to cut wmf/1.38.0-wmf.1 (T281165)
  • 16:04 dduvall: deleting wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to the branch cut job gone awry
  • 15:49 James_F: Re-running the branch cut job for T281165
  • 15:48 James_F: Updated the releases-jenkins automatic branch cut job to make 1.38.0 branches (missed ahead of T281165)

2021-09-17

  • 19:05 hashar: Building Docker images for [tox-buster] Install shellcheck and cascade [integration/config] - https://gerrit.wikimedia.org/r/721881
  • 18:08 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref T284774
  • 18:07 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref T28477
  • 16:19 dpifke: Enabled TLS on Jumbo Kafka instances in deployment-prep.

2021-09-16

2021-09-15

2021-09-14

  • 17:17 Amir1: delete from wb_changes_dispatch where chd_db = 'enwiki'; (T290985)

2021-09-13

  • 09:33 hashar: Castor cache: nuked files that were last changed more than six months ago to free up disk space

2021-09-10

  • 21:52 James_F: Created experimental integration-agent-docker-1021 for T252071
  • 21:48 James_F: Deleting CI agent integration-agent-docker-1001 for T252071
  • 21:44 James_F: Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for T252071
  • 21:23 James_F: Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job
  • 17:41 James_F: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI
  • 02:18 James_F: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12
  • 02:13 James_F: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job
  • 02:10 James_F: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12
  • 01:58 James_F: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental
  • 01:48 James_F: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices

2021-09-09

  • 22:20 brennen: gitlab-ansible-test: resetting instance data
  • 19:42 James_F: Docker: Building node{10,12}-test-browser-php80-composer for T290651
  • 10:56 hashar: Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17
  • 10:37 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496

2021-09-08

2021-09-07

2021-09-03

2021-09-02

  • 15:17 brennen: gitlab-test: testing upgrade path to 14.x

2021-09-01

  • 21:22 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/716041 in beta.
  • 16:55 urbanecm: deployment-prep: Unlock scap
  • 15:50 urbanecm: deployment-prep: Lock scap again
  • 15:40 urbanecm: deployment-prep: Lock scap to be able to test something
  • 14:08 urbanecm: deployment-prep: Create foundationwiki (T290164)
  • 14:07 urbanecm: urbanecm@deployment-mediawiki11:~$ sudo run-puppet-agent # T290164
  • 13:58 urbanecm: urbanecm@deployment-cache-text06:~$ sudo run-puppet-agent # T290164

2021-08-31

  • 21:38 dduvall: deploying new blubberoid to eqiad/codfw following successful testing in staging
  • 21:35 dduvall: staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276
  • 14:29 hashar: Restarting CI Jenkins for plugins upgrade

2021-08-30

  • 19:53 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta
  • 19:27 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)
  • 19:16 brennen: gitlab-test: powering off gitlab (former main test instance)
  • 19:11 James_F: Zuul: [mediawiki/services/apple-search] Add postmerge publish
  • 19:10 brennen: gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test
  • 18:23 brennen: gitlab-test: associating floating IP to primary test box
  • 16:36 James_F: Zuul: [mediawiki/services/apple-search] Remove composer-package
  • 15:16 James_F: Zuul: [mediawiki/services/apple-search] Add pipeline CI for T289224
  • 12:20 Amir1: foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5

2021-08-27

  • 19:24 James_F: Docker: Publish initial node14 CI images for T267888

2021-08-25

2021-08-24

  • 23:15 James_F: Zuul: Configure the REL1_37 test and gate pipelines T289587
  • 22:18 thcipriani: phab1001:sudo /srv/phab/phabricator/bin/bulk make-silent --id 2822 (releng-logspam -> unstewarded production error)
  • 20:02 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/712990

2021-08-23

  • 20:37 James_F: Docker: Publish php-ast with 1.0.14 (T289429) and no longer support PHP 7.0 or 7.1 (last trace!)

2021-08-21

  • 15:03 majavah: fixing deployment-prep puppet merge conflicts re: swift

2021-08-20

  • 21:11 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging/private$ rm mwblocker.log # remove weird blank log file
  • 18:56 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/713439
  • 18:55 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
  • 18:54 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in {1..20}; do echo "test $i" | mwscript edit.php --wiki={cswiki,enwiki} --user="Martin Urbanec (test $i)" --summary="test" Sandbox; done
  • 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=enwiki
  • 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
  • 18:46 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ for i in {1..20}; do mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki "Martin Urbanec (test $i)"; done
  • 18:40 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in {1..20}; do mwscript createAndPromote.php --wiki=cswiki "Martin Urbanec (test $i)" "$password"; done # to test a feature that needs a lot of different accounts
  • 16:30 majavah: restart sssd on deployment-cache-text06, T286502?
  • 16:24 majavah: deployment-prep: configure wikifunctions.beta.wmflabs.org dns zones and add to acme-chief T284162

2021-08-19

2021-08-17

2021-08-16

2021-08-15

  • 17:44 James_F: Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI

2021-08-13

  • 20:09 urbanecm: Manually start `beta-update-databases-eqiad` CI job
  • 20:06 urbanecm: deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py
  • 20:03 urbanecm: Kill beta-scap-sync-world job for the usual reason
  • 13:13 majavah: `mwscript extensions/CentralAuth/maintenance/importMissingLocalNames.php --wiki metawiki` on the beta cluster

2021-08-11

  • 00:52 James_F: Zuul: Add Aca to the CI allow list
  • 00:52 James_F: Zuul: [mediawiki/extensions/SimpleCalendar] Add basic quibble CI

2021-08-10

  • 16:57 James_F: Zuul: Update e-mail address for Zabe in the allow list
  • 16:06 James_F: Ran `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Wikibase` on doc1001 for T288396

2021-08-09

  • 21:02 urbanecm: Remove hanging beta-scap-sync-world job in CI to unblock beta auto-updates

2021-08-07

  • 00:48 James_F: Docker: Publish quibble-buster-php73-coverage 1.1.1 for T287918.
  • 00:29 James_F: Zuul: Add skin-coverage jobs to all Wikimedia production skins T287918
  • 00:27 James_F: Zuul: Provide a skin-coverage template T287918

2021-08-06

  • 23:53 James_F: Docker: Publishing quibble-buster-php73-coverage 1.1.0 for T287918
  • 23:47 James_F: Zuul: [mediawiki/skins/Mirage] Not a production skin; move to right section

2021-08-05

2021-08-04

2021-08-03

2021-08-02

2021-07-30

  • 21:27 dduvall: "Total reclaimed space: 141.4GB" on releases1002 following docker prune
  • 21:24 dduvall: running `docker system prune -af` on releases1002

2021-07-29

2021-07-28

2021-07-27

  • 16:55 dduvall: creating new gitlab runner instance runner-1002 for testing
  • 16:32 hashar: cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches

2021-07-26

2021-07-23

  • 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner (T287279)
  • 14:08 hashar: Building Docker images for quibble 1.0.1
  • 13:45 hashar: Tag Quibble 1.0.1 @ 5a2548699a # T287001

2021-07-21

  • 21:06 brennen: gitlab1001: running ansible for logging typo fix (T274462)
  • 20:36 dancy: Newest scap deployed to beta cluster
  • 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes (T274462, T275170)
  • 16:46 dancy: restarting Gerrit to fix plugins
  • 16:07 dancy: Updating plugins on releases-jenkins
  • 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki

2021-07-20

  • 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar (T281156)
  • 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47 # T287001

2021-07-19

  • 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values

2021-07-18

  • 08:48 majavah: set shared_acme_certificates: {} on deployment-prep shared hiera, T276653

2021-07-16

2021-07-14

2021-07-13

  • 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep. Should only affect deployment-webperf12.
  • 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for T281430
  • 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived T281430
  • 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet. Should only affect deployment-webperf12.
  • 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles # T278990
  • 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
  • 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # T278990
  • 07:46 hashar: Wiping all Docker images from contint2001l

2021-07-12

  • 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, T286506
  • 18:43 majavah: deployment-cache-text06 varnish not starting, T286506, causing an outage on text traffic on deployment-prep
  • 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
  • 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
  • 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' (T286479)
  • 14:44 majavah: fix merge conflict on deployment-puppetmaster04
  • 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
  • 13:19 James_F: Zuul: Add R4356th to the CI allow list
  • 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline T286207

2021-07-09

  • 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days (T286375)
  • 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days (T286358)

2021-07-08

  • 08:21 majavah: kick stuck puppet agent on deployment-logstash04

2021-07-07

  • 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance T286298

2021-07-06

2021-07-05

  • 15:02 Amir1: deployed 703212 (T286058)

2021-07-02

2021-06-30

  • 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
  • 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
  • 17:57 thcipriani: restart ci jenkins following upgrade
  • 17:54 thcipriani: restart releases-jenkins following upgrade
  • 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31

2021-06-29

  • 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # T285811

2021-06-28

  • 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
  • 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for T285125

2021-06-24

  • 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)

2021-06-23

  • 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick

2021-06-22

2021-06-21

  • 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.

2021-06-19

  • 13:44 majavah: remove deployment-deploy02 T278689
  • 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host T283013

2021-06-16

  • 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException T284732
  • 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert

2021-06-14

  • 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
  • 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
  • 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge T236915
  • 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing T236915
  • 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
  • 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
  • 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli

2021-06-13

2021-06-11

  • 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
  • 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test T284345
  • 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test T284345
  • 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
  • 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497 # T73480
  • 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
  • 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # T100837
  • 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for T271011. Comment left on gerrit patch and task, not reverting patch in master at this stage

2021-06-10

  • 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too T281195
  • 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency T281195

2021-06-08

  • 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
  • 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
  • 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
  • 17:40 brennen: gitlab1001: running k6 data generator
  • 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 T280491

2021-06-07

  • 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # T263617
  • 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #T263617
  • 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for T284492
  • 11:51 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1 # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470

2021-06-05

  • 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium

2021-06-04

  • 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
  • 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
  • 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 T284345
  • 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 T284345
  • 19:20 James_F: Docker: Publishing node12 CI images T284343

2021-06-03

  • 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/* # T284125
  • 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
  • 00:24 James_F: Zuul: Add Anysite to CI allowlist
  • 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline T281132

2021-06-02

2021-05-28

2021-05-27

  • 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
  • 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
  • 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
  • 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
  • 16:05 brennen: gitlab1001: re-running ansible and puppet per T279545
  • 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
  • 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs

2021-05-26

  • 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` (T279545)
  • 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org

2021-05-24

2021-05-21

2021-05-20

  • 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
  • 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
  • 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
  • 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
  • 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs T280669

2021-05-19

  • 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs T280669
  • 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps T279275
  • 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users T279552

2021-05-17

  • 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive T279221

2021-05-16

  • 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
  • 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04

2021-05-15

2021-05-14

  • 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php), T198673
  • 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, T198673, https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
  • 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, T198673, https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
  • 20:51 Krinkle: I broke beta `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
  • 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
  • 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
  • 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job T174018 T177260 T280901
  • 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI T281538

2021-05-13

2021-05-12

2021-05-11

  • 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set T237538 T249674
  • 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI

2021-05-10

  • 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
  • 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default (T271206)
  • 14:04 CFisch_WMDE: Forward renamed config name for improved template search features (T277028)

2021-05-07

  • 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue T282018

2021-05-06

  • 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds T264877
  • 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
  • 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job T271562

2021-05-05

2021-05-04

  • 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
  • 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02

2021-05-03

  • 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge T281256
  • 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
  • 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
  • 13:55 CFisch_WMDE: enable new search features for the template dialog (T271802)

2021-05-02

  • 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
  • 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
  • 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref T198673
  • 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
  • 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117

2021-05-01

  • 19:19 James_F: Zuul: Add atagar to the CI allow list
  • 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - T278641
  • 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004

2021-04-30

  • 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
  • 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for T257301
  • 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
  • 14:15 Majavah: revert above as it's not working, T206158
  • 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 T206158
  • 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
  • 02:37 James_F: Docker: Publishing node10 images based on buster T278203 T240955

2021-04-29

  • 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist T281418

2021-04-28

  • 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs T279180
  • 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # T249268
  • 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
  • 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip {} \+ # T249268
  • 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.

2021-04-27

  • 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for T277742.
  • 17:52 Majavah: delete deployment-sessionstore03 T263617 T278641
  • 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for T281283
  • 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster | https://gerrit.wikimedia.org/r/c/integration/config/+/680476
  • 12:44 hashar: Restarted CI Jenkins for plugins upgrade
  • 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2 (with ldap plugin 1.26)
  • 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
  • 06:40 Majavah: installing deployment-sessionstore04 T263617
  • 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs T281198

2021-04-26

  • 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list

2021-04-24

  • 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job

2021-04-23

  • 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
  • 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
  • 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts

2021-04-22

  • 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
  • 02:53 Reedy: killed a few stuck beta ci jobs
  • 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
  • 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 T280874
  • 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental T280874
  • 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for T280874
  • 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos T280874

2021-04-21

  • 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
  • 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs T280874
  • 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
  • 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for T280874
  • 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies

2021-04-20

  • 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog (T273425)
  • 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor (T271825)

2021-04-19

  • 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental T280491
  • 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
  • 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for T280491

2021-04-17

  • 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for T280420

2021-04-16

  • 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks T234020 T280167
  • 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
  • 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch T278203
  • 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs T252432
  • 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref T235142
  • 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref T198673
  • 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref T198673
  • 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
  • 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
  • 17:02 dancy: Updating dev-images docker-pkg files on primary contint
  • 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template T252434

2021-04-15

  • 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
  • 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts

2021-04-14

  • 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov T234020

2021-04-13

2021-04-12

  • 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; T279853)
  • 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # T279853
  • 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per T106915#6279270 - T135427
  • 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
  • 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)

2021-04-11

  • 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
  • 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov T234020
  • 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3{5,6} T274971
  • 00:02 James_F: Docker: Publishing quibble-buster images T252432

2021-04-10

  • 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list

2021-04-09

2021-04-08

2021-04-07

  • 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
  • 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
  • 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
  • 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 T278664
  • 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
  • 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382

2021-04-06

  • 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived T277626
  • 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs T279260
  • 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines T279459
  • 17:58 James_F: Zuul: [mediawiki/services/function-{orchestr,evalu}ator] Publish images

2021-04-05

  • 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
  • 17:45 brennen: halting gitlab-test for resize

2021-04-02

  • 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
  • 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12

2021-04-01

  • 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
  • 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
  • 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
  • 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298

2021-03-30

  • 18:05 Majavah: remove {trysty,precise}-deployment-prep repos from deployment-deploy01 aptly
  • 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
  • 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those T278689
  • 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
  • 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
  • 13:35 Majavah: create and install deployment-deploy03 T278689
  • 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
  • 07:26 Majavah: shutoff deployment-mediawiki-09 T278664
  • 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
  • 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11

2021-03-29

  • 15:37 Majavah: hard reboot deployment-sessionstore03 T263617
  • 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) T263617
  • 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 (T278664), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
  • 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
  • 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
  • 09:29 Urbanecm: Manually run puppet on mediawiki10
  • 09:28 Urbanecm: Re-enable puppet on mediawiki10
  • 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
  • 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - T278664
  • 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, T278664

2021-03-26

2021-03-25

2021-03-24

  • 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm T238707

2021-03-23

  • 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per T274015
  • 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job T207226
  • 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
  • 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade T252434
  • 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template T252434
  • 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch T252434
  • 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch T252434
  • 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use

2021-03-22

  • 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
  • 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
  • 12:07 Majavah: delete deployment-restbase[01-02], T250574
  • 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. (T276624)
  • 11:33 dcaro: Created subzone beta.wmcloud.org (T276624)

2021-03-19

2021-03-18

2021-03-17

  • 20:30 hashar: Reloaded Zuul for I236847
  • 15:37 Majavah: shutdown deployment-restbase01 for T250574
  • 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # T250574
  • 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
  • 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI

2021-03-16

  • 21:37 longma: Updating dev-images docker-pkg files on primary contint
  • 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
  • 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
  • 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
  • 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
  • 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1 # T277527

2021-03-15

2021-03-13

  • 17:10 twentyafterfour: restart apache on gerrit1001

2021-03-12

2021-03-11

2021-03-10

  • 19:36 Majavah: shutdown deployment-ircd T277081
  • 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 T277081
  • 18:05 Majavah: create deployment-ircd02 for T277081
  • 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
  • 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
  • 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
  • 17:09 Majavah: set beta cluster mediawiki as read write on mw config (T276968)
  • 17:03 Majavah: make deployment-db06 read-write T276968
  • 16:50 Majavah: `reset slave;` on new master deployment-db06 T276968
  • 16:49 Majavah: add deployment-db07 as a replica of db06 for T276968
  • 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # T276968
  • 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; (T276968)
  • 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . (T276968)
  • 15:57 Majavah: set deployment-db06 as readonly from mysql side T276968
  • 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` (T276968)
  • 15:54 Urbanecm: Start mariadb on db08 (T276968)
  • 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 (T276968)
  • 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
  • 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
  • 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade (T276968)
  • 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) (T276968)

2021-03-09

  • 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround (T276968)
  • 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 (T276968)
  • 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
  • 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user (T276968)
  • 20:33 Majavah: install mariadb on deployment-db08 T276968
  • 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db (T276968)
  • 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica (T276968)
  • 19:50 marxarelli: restoring database dump on deployment-db07 (T276968)
  • 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968)
  • 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 T276968
  • 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968)
  • 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968
  • 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump (T276968)
  • 18:09 Majavah: set deployment-db05 to read-only to avoid issues with T276968
  • 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968)
  • 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
  • 17:22 marxarelli: restarting deployment-db05 via horizon
  • 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
  • 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
  • 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
  • 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
  • 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
  • 13:32 arturo: hard-reboot deployment-db05 because issues related to T276922
  • 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other

2021-03-08

2021-03-07

  • 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
  • 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
  • 17:18 Majavah: shutdown deployment-memc[04-05] T276707
  • 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707
  • 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707
  • 15:44 Majavah: create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
  • 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585
  • 13:14 Majavah: create deployment-memc09 on Buster T276707

2021-03-06

  • 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
  • 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 T276654
  • 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix T276652

2021-03-05

  • 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
  • 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
  • 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
  • 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462
  • 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462
  • 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462
  • 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate T276462
  • 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
  • 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others
  • 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2

2021-03-04

  • 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384
  • 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref T276102 and it being decomissioned in T242855
  • 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for T276419, I believe everything has been moved to deployment-mwlog01
  • 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for T276419
  • 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
  • 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
  • 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref T276419
  • 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
  • 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # T276422 T180301
  • 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
  • 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch

2021-03-03

2021-03-02

  • 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
  • 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).

2021-03-01

  • 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
  • 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.

2021-02-27

  • 22:03 Reedy: re-armed beta keyholder... I think...

2021-02-26

2021-02-24

  • 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
  • 22:16 legoktm: rebuilding Rust docker images

2021-02-23

  • 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
  • 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)

2021-02-20

  • 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
  • 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI

2021-02-19

2021-02-18

  • 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
  • 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata

2021-02-17

2021-02-16

2021-02-15

  • 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # T209953

2021-02-14

  • 21:31 James_F: Zuul: Add 'check php' support for library repos
  • 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive T274738

2021-02-13

  • 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library

2021-02-12

  • 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
  • 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$

2021-02-11

  • 21:44 Krinkle: Logstash in beta is not receiving any events T274593
  • 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
  • 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension T274069
  • 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
  • 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
  • 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339

2021-02-10

  • 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
  • 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for T274306
  • 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
  • 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension

2021-02-09

2021-02-08

2021-02-06

  • 21:13 Reedy: unstuck beta jobs

2021-02-04

  • 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
  • 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI T273247

2021-02-03

  • 21:38 James_F: Zuul: Archive VirtualKeyboard extension T273801
  • 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
  • 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
  • 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL T273295

2021-02-02

  • 19:43 hashar: Pruning dangling Docker images on contint2001
  • 19:39 hasharDinner: Pruning dangling Docker images on contint1001
  • 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move T61245
  • 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
  • 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # T273599

2021-02-01

2021-01-29

  • 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes

2021-01-28

2021-01-27

  • 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02

2021-01-26

  • 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
  • 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
  • 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
  • 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
  • 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355

2021-01-25

2021-01-22

  • 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout

2021-01-19

  • 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI T271761

2021-01-18

  • 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker T272320

2021-01-17

  • 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job T248925

2021-01-16

  • 23:24 James_F: Docker: Building cascade of new php-ast image T271428

2021-01-14

  • 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job

2021-01-13

  • 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents

2021-01-12

  • 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
  • 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
  • 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
  • 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
  • 19:33 James_F: Zuul: Revert Parsoid integration job injection.
  • 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
  • 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension T237538
  • 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template

2021-01-11

  • 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # T271683

2021-01-09

  • 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing T271575
  • 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing T271575

2021-01-08

2021-01-07

2021-01-06

2021-01-05

2021-01-04

  • 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job T269719
  • 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled for now due to T271122
  • 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot # T271122
  • 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts

2021-01-02

  • 19:13 James_F: Zuul: Add CI for the Mirage skin T270979

2021-01-01

  • 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master T270976
  • 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting T251967

Archives