Release Engineering/SAL

2021-10-25

• 23:27 dduvall: fully provisioned runner-{1008,1011,1012,1013,1014,1015,1016,1017,1018,1019} instances for use as new gitlab runners and removed old instances (T293835)
• 20:50 James_F: Docker: Publishing php*-comile images without the PECL test so they work again.
• 20:32 James_F: Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72
• 19:21 James_F: Docker: Publishing new php74 and cascaded images with PHP 7.4 from Wikimedia package T293851
• 18:45 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734377 T294238
• 18:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734370 https://phabricator.wikimedia.org/T294238
• 17:01 dduvall: deleting bullseye runner-1006 and reverting to using buster due to some puppet provisioning issues
• 16:17 dduvall: launching new runner-1006 instance under gitlab-runners

2021-10-24

• 08:52 hashar: CI Jenkins: update Clover PHP plugin from 0.5 to 0.6 to solve `java.lang.NoClassDefFoundError: org/apache/commons/digester/Digester` # T292910
• 06:58 majavah: hard reboot crashed deployment-db08

2021-10-22

• 21:19 James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change T292910
• 18:06 dancy: Updating deb packages on deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud
• 00:31 James_F: Zuul: [DiscussionTools] Add EventLogging as dependency for Phan jobs

2021-10-21

• 18:07 dduvall: replacing runner-1001 with new instance (T293835)
• 09:49 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test T292728

2021-10-19

• 22:24 brennen: gitlab-test: deleted unused gitlab-runner-test
• 22:20 brennen: adding Dzahn and Aokoth to gitlab-test and gitlab-runners projects, cleaning up contractor access
• 21:20 brennen: Updating dev-images docker-pkg files on primary contint for php 8 images (T283426)
• 18:00 James_F: Zuul: [mediawiki/services/function-schemata] Switch from pipeline to direct CI
• 11:32 hashar: Archiving integration/gear.git , no more used after Python gear 0.16.0 has been released # T289512

2021-10-18

• 16:26 Lucas_WMDE: killed one beta-scap-sync-world job that seemed stuck for a long time
• 14:43 James_F: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer T290274
• 09:41 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # T248676

2021-10-16

• 16:47 Lucas_WMDE: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T293070, based on T271808#6739578
• 01:17 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/730949

2021-10-15

• 16:14 brennen: gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for T292372

2021-10-14

• 16:13 addshore: reloaded zuul for zuul: mw-ext-MailGun will need PHP 7.3+ T236396
• 14:13 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/730207

2021-10-13

• 23:12 bd808: Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors.
• 16:48 brennen: Updating dev-images docker-pkg files on primary contint for T293277
• 11:56 hashar: integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes'

2021-10-12

• 19:11 hashar: Building Docker images for https://gerrit.wikimedia.org/r/730279
• 17:28 James_F: Zuul: [mediawiki/extensions/THICC] Mark as archived for T292948

2021-10-11

• 15:35 hashar: gerrit: marked integration/composer read-only. We are no more using it and the few use cases left are tracked at https://phabricator.wikimedia.org/T249949
• 02:08 Krinkle: Browsers fail to connect with https://upload.wikimedia.beta.wmflabs.org/ (Certificate expired two days ago)

2021-10-08

• 21:17 hashar: Purging Docker images on all CI agents
• 20:00 hashar: Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620
• 19:23 hashar: Building CI Docker images for Quibble 1.1.1
• 19:08 hashar: Tag quibble 1.1.1 @ b54af2aa60
• 19:07 brennen: gitlab1001: restarting to check correct session_duration value of 604800
• 11:01 hashar: Rebuilding CI Docker images based on Bullseye to include apt security updates

2021-10-07

• 22:12 James_F: Zuul: [design/codex] Provide initial CI, node14-browser suite
• 17:07 James_F: Zuul: [mediawiki/extensions/ParsoidBatchAPI] Mark as archived T252744
• 13:29 hashar: restarting CI Jenkins for git plugin update
• 13:14 hashar: Upgraded CI Jenkins on contint2001
• 13:10 hashar: Upgraded releases Jenkins from 2.289.2 to 2.303.2
• 13:04 hashar: Updating Jenkins jobs for Quibble 1.1.0 # T291549
• 04:46 Krinkle: Temporarily change `mediawiki-fresnel-patch-docker` job change back to quibble-fresnel:1.0.1 ref T292684
• 04:41 Krinkle: Reconfigure `mediawiki-fresnel-patch-docker` job to revert quibble change back to quibble-fresnel:1.0.1
• 01:06 Reedy: unstick beta ci jobs

2021-10-06

• 11:19 hashar: Tag Quibble 1.1.0 @ 3dabcab

2021-10-05

• 21:47 brennen: updating dev-images docker-pkg files on primary contint for https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/7
• 12:03 majavah: root@deployment-cache-text06:~# systemctl restart sssd # T286502
• 11:11 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/726576

2021-10-04

• 21:09 brennen: finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 (T292256)
• 21:02 brennen: gitlab-runners: upgrading each shared runner to 14.3.2
• 18:12 James_F: Docker: Publish composer 2.x-based images for T279857
• 16:46 James_F: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
• 15:54 James_F: Zuul: Add Lindsay Wardell personal e-mail to CI allow list
• 14:31 brennen: gitlab1001: upgrading to 14.3.2 (T292256)

2021-09-30

• 21:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/683031

2021-09-29

• 15:52 hashar: Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # T277500
• 15:03 brennen: gitlab: pausing shared runners while working through access-level implications

2021-09-27

• 20:52 James_F: Zuul: [integration/visualdiff] Add basic node12 CI job
• 20:15 brennen: gitlab1001: login and user rename as "brennen" checks out; applying user mappings for remaining ~80 users
• 19:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/724103/3
• 19:24 hashar: Updating Jenkins jobs using a Stretch related image to latest version of the image. https://gerrit.wikimedia.org/r/c/integration/config/+/724156/ # T291425
• 17:23 hashar: Building java8-scala-spark image
• 07:11 hashar: Switching python based jobs to docker-registry.wikimedia.org/releng/tox-buster:0.5.0 # T291292

2021-09-24

• 15:41 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/coal/+/722948 and latest https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep. Should only affect deployment-webperf11.

2021-09-23

• 19:55 brennen: gitlab-ansible-test: sudo gitlab-ctl cleanse to drop test data and reset
• 18:41 James_F: Zuul: Create 'extension-apitests' template for simplicity
• 15:54 brennen: gitlab1001: brief downtime to apply gitlab cas: uid instead of CN; add nickname_key for T288392
• 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/723217
• 14:46 hashar: Building releng/tox-buster:0.5.0 image
• 11:44 hashar: Reloaded zuul for "zuul: promote shellcheck job for integration/config" https://gerrit.wikimedia.org/r/c/integration/config/+/723160/1
• 10:57 hashar: Building tox images for improved shellcheck package https://gerrit.wikimedia.org/r/721895

2021-09-22

• 15:47 Amir1: set row format for text table in simplewiki to compressed
• 15:22 Amir1: restart mysql on db07 to add performance schema

2021-09-21

• 16:26 dduvall: deleted wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to a branch cut job gone awry. re-running job to cut wmf/1.38.0-wmf.1 (T281165)
• 16:04 dduvall: deleting wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to the branch cut job gone awry
• 15:49 James_F: Re-running the branch cut job for T281165
• 15:48 James_F: Updated the releases-jenkins automatic branch cut job to make 1.38.0 branches (missed ahead of T281165)

2021-09-17

• 19:05 hashar: Building Docker images for [tox-buster] Install shellcheck and cascade [integration/config] - https://gerrit.wikimedia.org/r/721881
• 18:08 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref T284774
• 18:07 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref T28477
• 16:19 dpifke: Enabled TLS on Jumbo Kafka instances in deployment-prep.

2021-09-16

• 22:17 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/721567 in deployment-prep, should only affect deployment-webperf11.
• 16:51 addshore: reload zuul for Promote branchdeploy for query-builder to test https://gerrit.wikimedia.org/r/721581 (https://phabricator.wikimedia.org/T278706)
• 13:27 Reedy: Actually reloading Zuul to deploy https://gerrit.wikimedia.org/r/683589
• 13:01 addshore: reload zuul for https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) 1 experimental branchdeploy job

2021-09-15

• 22:59 dpifke: Un-cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep for now; more work needed.
• 22:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/721044 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep; should only affect deployment-webperf11.
• 08:58 Amir1: update wb_changes_dispatch set chd_disabled = 1 where chd_site = 'enwiki'; (T290985)

2021-09-14

• 17:17 Amir1: delete from wb_changes_dispatch where chd_db = 'enwiki'; (T290985)

2021-09-13

• 09:33 hashar: Castor cache: nuked files that were last changed more than six months ago to free up disk space

2021-09-10

• 21:52 James_F: Created experimental integration-agent-docker-1021 for T252071
• 21:48 James_F: Deleting CI agent integration-agent-docker-1001 for T252071
• 21:44 James_F: Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for T252071
• 21:23 James_F: Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job
• 17:41 James_F: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI
• 02:18 James_F: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12
• 02:13 James_F: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job
• 02:10 James_F: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12
• 01:58 James_F: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental
• 01:48 James_F: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices

2021-09-09

• 22:20 brennen: gitlab-ansible-test: resetting instance data
• 19:42 James_F: Docker: Building node{10,12}-test-browser-php80-composer for T290651
• 10:56 hashar: Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17
• 10:37 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496

2021-09-08

• 20:16 thcipriani: self +2 on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719500 to unbreak beta
• 18:47 brennen: contint1001 / contint2001: /srv/dev-images: git remote set-url origin 'https://gitlab.wikimedia.org/releng/dev-images.git'
• 18:28 dduvall: Running ./fab deploy_docker to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719350
• 17:05 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355
• 16:37 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719493
• 14:55 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/714328/
• 14:05 brennen: runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners (T289802)

2021-09-07

• 20:49 James_F: Marked https://gerrit.wikimedia.org/g/mediawiki/tools/cli as read-only and pointed users to GitLab.
• 20:46 brennen: migrating dev-images to https://gitlab.wikimedia.org/releng/dev-images and deactivating on gerrit
• 18:16 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719311
• 18:11 dduvall: creating 2 new jenkins jobs for deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/719311
• 16:25 James_F: Docker: Publishing node12-test-browser-php{72,80}-composer images
• 16:19 James_F: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add 4 dependencies

2021-09-03

• 23:02 Krinkle: Creating integration-agent-qemu-1002 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref T284774
• 17:42 dduvall: deploying blubberoid:2021-09-03-160524-production to eqiad/codfw (https://gerrit.wikimedia.org/r/c/blubber/+/716519) (T289367)
• 17:36 dduvall: staging blubberoid to deploy https://gerrit.wikimedia.org/r/c/blubber/+/716519

2021-09-02

• 15:17 brennen: gitlab-test: testing upgrade path to 14.x

2021-09-01

• 21:22 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/716041 in beta.
• 16:55 urbanecm: deployment-prep: Unlock scap
• 15:50 urbanecm: deployment-prep: Lock scap again
• 15:40 urbanecm: deployment-prep: Lock scap to be able to test something
• 14:08 urbanecm: deployment-prep: Create foundationwiki (T290164)
• 14:07 urbanecm: urbanecm@deployment-mediawiki11:~$ sudo run-puppet-agent # T290164
• 13:58 urbanecm: urbanecm@deployment-cache-text06:~$ sudo run-puppet-agent # T290164

2021-08-31

• 21:38 dduvall: deploying new blubberoid to eqiad/codfw following successful testing in staging
• 21:35 dduvall: staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276
• 14:29 hashar: Restarting CI Jenkins for plugins upgrade

2021-08-30

• 19:53 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta
• 19:27 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)
• 19:16 brennen: gitlab-test: powering off gitlab (former main test instance)
• 19:11 James_F: Zuul: [mediawiki/services/apple-search] Add postmerge publish
• 19:10 brennen: gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test
• 18:23 brennen: gitlab-test: associating floating IP to primary test box
• 16:36 James_F: Zuul: [mediawiki/services/apple-search] Remove composer-package
• 15:16 James_F: Zuul: [mediawiki/services/apple-search] Add pipeline CI for T289224
• 12:20 Amir1: foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5

2021-08-27

• 19:24 James_F: Docker: Publish initial node14 CI images for T267888

2021-08-25

• 17:48 twentyafterfour: updating zuul config after deploying https://gerrit.wikimedia.org/r/714538

2021-08-24

• 23:15 James_F: Zuul: Configure the REL1_37 test and gate pipelines T289587
• 22:18 thcipriani: phab1001:sudo /srv/phab/phabricator/bin/bulk make-silent --id 2822 (releng-logspam -> unstewarded production error)
• 20:02 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/712990

2021-08-23

• 20:37 James_F: Docker: Publish php-ast with 1.0.14 (T289429) and no longer support PHP 7.0 or 7.1 (last trace!)

2021-08-21

• 15:03 majavah: fixing deployment-prep puppet merge conflicts re: swift

2021-08-20

• 21:11 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging/private$ rm mwblocker.log # remove weird blank log file
• 18:56 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/713439
• 18:55 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
• 18:54 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in {1..20}; do echo "test $i" | mwscript edit.php --wiki={cswiki,enwiki} --user="Martin Urbanec (test $i)" --summary="test" Sandbox; done
• 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=enwiki
• 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
• 18:46 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ for i in {1..20}; do mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki "Martin Urbanec (test $i)"; done
• 18:40 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in {1..20}; do mwscript createAndPromote.php --wiki=cswiki "Martin Urbanec (test $i)" "$password"; done # to test a feature that needs a lot of different accounts
• 16:30 majavah: restart sssd on deployment-cache-text06, T286502?
• 16:24 majavah: deployment-prep: configure wikifunctions.beta.wmflabs.org dns zones and add to acme-chief T284162

2021-08-19

• 22:58 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/713612/

2021-08-17

• 21:23 brennen: Updating dev-images docker-pkg files on primary contint for add buster-apache2, log errors to stdio (T283416)

2021-08-16

• 23:08 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/711731

2021-08-15

• 17:44 James_F: Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI

2021-08-13

• 20:09 urbanecm: Manually start `beta-update-databases-eqiad` CI job
• 20:06 urbanecm: deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py
• 20:03 urbanecm: Kill beta-scap-sync-world job for the usual reason
• 13:13 majavah: `mwscript extensions/CentralAuth/maintenance/importMissingLocalNames.php --wiki metawiki` on the beta cluster

2021-08-11

• 00:52 James_F: Zuul: Add Aca to the CI allow list
• 00:52 James_F: Zuul: [mediawiki/extensions/SimpleCalendar] Add basic quibble CI

2021-08-10

• 16:57 James_F: Zuul: Update e-mail address for Zabe in the allow list
• 16:06 James_F: Ran `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Wikibase` on doc1001 for T288396

2021-08-09

• 21:02 urbanecm: Remove hanging beta-scap-sync-world job in CI to unblock beta auto-updates

2021-08-07

• 00:48 James_F: Docker: Publish quibble-buster-php73-coverage 1.1.1 for T287918.
• 00:29 James_F: Zuul: Add skin-coverage jobs to all Wikimedia production skins T287918
• 00:27 James_F: Zuul: Provide a skin-coverage template T287918

2021-08-06

• 23:53 James_F: Docker: Publishing quibble-buster-php73-coverage 1.1.0 for T287918
• 23:47 James_F: Zuul: [mediawiki/skins/Mirage] Not a production skin; move to right section

2021-08-05

• 22:26 brennen: gitlab: setting CI access on all repos to "Project members only", per https://www.mediawiki.org/wiki/GitLab/Policy#Permissions - may need revisited depending on effects
• 22:04 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/710108
• 15:10 hashar: integration: sudo cumin --force 'name:docker' 'docker container prune -f && docker image prune -f'
• 15:09 hashar: integration-agent-docker-1020: docker container prune && docker image prune
• 15:02 hashar: Building Docker image helm-linter for https://gerrit.wikimedia.org/r/c/integration/config/+/710276

2021-08-04

• 22:53 bd808: Updated composer-github-oauthtoken in Jenkins config to us a newer personal access token from GitHub per notices about https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
• 17:49 brennen: gitlab-test: testing upgrade to 13.12.9
• 16:41 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/710066 # T288111
• 15:13 thcipriani: puppet fixed on deployment-deploy{01,03}
• 15:08 thcipriani: rebase deployment-puppetmaster04:labs/private causing deployment-deploy{01,03} failure for...¯\_(ツ)_/¯

2021-08-03

• 21:12 brennen: Updating dev-images docker-pkg files on primary contint for add buster php images

2021-08-02

• 23:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709563
• 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709562
• 23:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709558
• 22:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709555 https://gerrit.wikimedia.org/r/709556
• 21:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709543

2021-07-30

• 21:27 dduvall: "Total reclaimed space: 141.4GB" on releases1002 following docker prune
• 21:24 dduvall: running `docker system prune -af` on releases1002

2021-07-29

• 22:12 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/708596
• 14:06 brennen: gerrit: added ldap/ops to ownership of operations/gitlab-ansible
• 13:21 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/708764 "Add michaelcochez to whitelist users"

2021-07-28

• 17:53 andrewbogott: rebooting deployment-logstash03 as it's in an inconsistent config state
• 14:44 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/708536

2021-07-27

• 16:55 dduvall: creating new gitlab runner instance runner-1002 for testing
• 16:32 hashar: cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches

2021-07-26

• 20:45 brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation (T287279)
• 18:10 dduvall: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
• 18:00 dduvall: creating 2 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
• 08:24 hashar: jjb: update jobs to Quibble 1.0.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/708038

2021-07-23

• 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner (T287279)
• 14:08 hashar: Building Docker images for quibble 1.0.1
• 13:45 hashar: Tag Quibble 1.0.1 @ 5a2548699a # T287001

2021-07-21

• 21:06 brennen: gitlab1001: running ansible for logging typo fix (T274462)
• 20:36 dancy: Newest scap deployed to beta cluster
• 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes (T274462, T275170)
• 16:46 dancy: restarting Gerrit to fix plugins
• 16:07 dancy: Updating plugins on releases-jenkins
• 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki

2021-07-20

• 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar (T281156)
• 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47 # T287001

2021-07-19

• 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values

2021-07-18

• 08:48 majavah: set shared_acme_certificates: {} on deployment-prep shared hiera, T276653

2021-07-16

• 21:01 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/698774
• 17:44 brennen: Try #2: Updating dev-images docker-pkg files on primary contint for Add a Swift language dev image for CI testing (T284195)
• 17:12 hashar: Tag Quibble 1.0.0 @ a13d133f7d1 # T286187 T280506 T90875 T218534 T227352

2021-07-14

• 22:47 brennen: Updating dev-images docker-pkg files on primary contint for Add a Swift language dev image for CI testing (T284195)
• 21:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in beta puppet again, should only affect deployment-webperf12.

2021-07-13

• 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep. Should only affect deployment-webperf12.
• 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for T281430
• 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived T281430
• 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet. Should only affect deployment-webperf12.
• 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles # T278990
• 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
• 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # T278990
• 07:46 hashar: Wiping all Docker images from contint2001l

2021-07-12

• 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, T286506
• 18:43 majavah: deployment-cache-text06 varnish not starting, T286506, causing an outage on text traffic on deployment-prep
• 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
• 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
• 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' (T286479)
• 14:44 majavah: fix merge conflict on deployment-puppetmaster04
• 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
• 13:19 James_F: Zuul: Add R4356th to the CI allow list
• 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline T286207

2021-07-09

• 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days (T286375)
• 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days (T286358)

2021-07-08

• 08:21 majavah: kick stuck puppet agent on deployment-logstash04

2021-07-07

• 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance T286298

2021-07-06

• 01:16 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/703219

2021-07-05

• 15:02 Amir1: deployed 703212 (T286058)

2021-07-02

• 17:32 brennen: gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/701068 (T274463)
• 09:49 James_F: Zuul: [pywikibot/core] Add deeptest to gate-and-submit section

2021-06-30

• 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
• 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
• 17:57 thcipriani: restart ci jenkins following upgrade
• 17:54 thcipriani: restart releases-jenkins following upgrade
• 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31

2021-06-29

• 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # T285811

2021-06-28

• 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
• 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for T285125

2021-06-24

• 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)

2021-06-23

• 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick

2021-06-22

• 23:03 jeena: increasing releases-jenkins executors from 2 to 4
• 21:25 jeena: Updating dev-images docker-pkg files on primary contint T273682
• 09:31 hashar: Removed DISPLAY=:94 being set from Zuul for most jobs https://gerrit.wikimedia.org/r/c/integration/config/+/700857/

2021-06-21

• 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.

2021-06-19

• 13:44 majavah: remove deployment-deploy02 T278689
• 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host T283013

2021-06-16

• 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException T284732
• 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert

2021-06-14

• 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
• 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
• 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge T236915
• 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing T236915
• 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
• 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
• 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli

2021-06-13

• 18:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699567
• 18:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699541

2021-06-11

• 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
• 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test T284345
• 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test T284345
• 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
• 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497 # T73480
• 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
• 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # T100837
• 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for T271011. Comment left on gerrit patch and task, not reverting patch in master at this stage

2021-06-10

• 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too T281195
• 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency T281195

2021-06-08

• 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
• 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
• 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
• 17:40 brennen: gitlab1001: running k6 data generator
• 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 T280491

2021-06-07

• 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # T263617
• 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #T263617
• 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for T284492
• 11:51 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1 # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470

2021-06-05

• 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium

2021-06-04

• 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
• 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
• 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 T284345
• 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 T284345
• 19:20 James_F: Docker: Publishing node12 CI images T284343

2021-06-03

• 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/* # T284125
• 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
• 00:24 James_F: Zuul: Add Anysite to CI allowlist
• 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline T281132

2021-06-02

• 19:21 urbanecm: deployment-prep: Enlarge mwlog's /srv partition to 5 GB (was 2 GB)
• 13:49 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project search/MjoLniR --change 685569,4 # Rerun sonar analysis against https://gerrit.wikimedia.org/r/c/search/MjoLniR/+/685569

2021-05-28

• 09:41 addshore: reload zuul for https://gerrit.wikimedia.org/r/697033
• 08:02 hashar: Successfully published image docker-registry.discovery.wmnet/releng/ci-bullseye:0.1.0 # T283777

2021-05-27

• 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
• 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
• 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
• 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
• 16:05 brennen: gitlab1001: re-running ansible and puppet per T279545
• 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
• 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs

2021-05-26

• 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` (T279545)
• 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org

2021-05-24

• 06:19 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693639

2021-05-21

• 18:30 James_F: Zuul: Add phan to all extensions and skins T283097
• 17:59 James_F: Docker: Publishing mediawiki-phan* images where the job passes if .phan/config.php is absent T283097
• 08:30 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693361 https://gerrit.wikimedia.org/r/691323

2021-05-20

• 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
• 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
• 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
• 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
• 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs T280669

2021-05-19

• 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs T280669
• 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps T279275
• 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users T279552

2021-05-17

• 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive T279221

2021-05-16

• 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
• 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04

2021-05-15

• 21:51 James_F: Zuul: [mediawiki/tools/cli] Make mw-cli-test experimental for now T248779
• 21:40 James_F: Zuul: [mediawiki/tools/cli] Add new bespoke job T248779
• 21:09 James_F: Zuul: [mediawiki/extensions/MediaWikiAuth] Mark repo as archived T282955
• 18:38 James_F: Zuul: Temporarily remove TwoColConflict from gated extensions T234002 T282935.
• 09:30 Majavah: create deployment-logstash04 to install elk7
• 09:22 Majavah: beta: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315, remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/683837 T277990
• 06:23 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/691494/ T281986

2021-05-14

• 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php), T198673
• 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, T198673, https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
• 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, T198673, https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
• 20:51 Krinkle: I broke beta `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
• 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
• 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
• 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job T174018 T177260 T280901
• 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI T281538

2021-05-13

• 23:14 Krinkle: root@integration-agent-qemu-1001:/home/addshore# rm qmeutest/out.img (Reclaiming space to be able to make other changes)
• 22:48 longma: restarting zuul to try getting tests to run for https://gerrit.wikimedia.org/r/c/mediawiki/skins/Modern/+/684509

2021-05-12

• 19:40 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/439774 in deployment-prep puppet. Hasn't been touched in over a year and is preventing merges.
• 19:32 James_F: Zuul: Add PageImages, PageViewInfo, and Graph to MediaWiki gated extension set T249674
• 19:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/683695 in deployment-prep puppet. Should only affect beta logstash.
• 16:15 James_F: Docker: Publishing civicrm 0.2.1 for T277500
• 16:00 James_F: Zuul: Add GrowthExperiments to the MediaWiki gated extension set T247507 T249674
• 15:36 _joe_: updating helm-linter docker image

2021-05-11

• 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set T237538 T249674
• 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI

2021-05-10

• 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
• 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default (T271206)
• 14:04 CFisch_WMDE: Forward renamed config name for improved template search features (T277028)

2021-05-07

• 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue T282018

2021-05-06

• 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds T264877
• 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
• 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job T271562

2021-05-05

• 23:52 James_F: Zuul: Ensure Parsoid's tests include the Disambiguator extension (take 2) T271863 T237538
• 20:38 James_F: Docker: Publishing quibble-buster* 0.0.47 images
• 19:58 James_F: Docker: Publishing quibble-stretch* 0.0.47-s1 images, now with node
• 18:23 James_F: Docker: Publishing quibble-stretch* 0.0.47 images
• 16:54 hashar: Tag Quibble 0.0.47 @ 8b200cfb0 # T271863 T199403 T281607
• 14:17 CFisch_WMDE: Disable ReferencePreviews beta mode on beta labs (T271206)
• 13:22 Majavah: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/684034 https://gerrit.wikimedia.org/r/c/operations/puppet/+/684088 - might cause beta to go read only for a bit - T110115

2021-05-04

• 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
• 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02

2021-05-03

• 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge T281256
• 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
• 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
• 13:55 CFisch_WMDE: enable new search features for the template dialog (T271802)

2021-05-02

• 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
• 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
• 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref T198673
• 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
• 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117

2021-05-01

• 19:19 James_F: Zuul: Add atagar to the CI allow list
• 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - T278641
• 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004

2021-04-30

• 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
• 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for T257301
• 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
• 14:15 Majavah: revert above as it's not working, T206158
• 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 T206158
• 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
• 02:37 James_F: Docker: Publishing node10 images based on buster T278203 T240955

2021-04-29

• 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist T281418

2021-04-28

• 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs T279180
• 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # T249268
• 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
• 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip {} \+ # T249268
• 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.

2021-04-27

• 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for T277742.
• 17:52 Majavah: delete deployment-sessionstore03 T263617 T278641
• 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for T281283
• 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster | https://gerrit.wikimedia.org/r/c/integration/config/+/680476
• 12:44 hashar: Restarted CI Jenkins for plugins upgrade
• 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2 (with ldap plugin 1.26)
• 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
• 06:40 Majavah: installing deployment-sessionstore04 T263617
• 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs T281198

2021-04-26

• 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list

2021-04-24

• 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job

2021-04-23

• 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
• 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
• 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts

2021-04-22

• 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
• 02:53 Reedy: killed a few stuck beta ci jobs
• 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
• 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 T280874
• 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental T280874
• 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for T280874
• 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos T280874

2021-04-21

• 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
• 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs T280874
• 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
• 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for T280874
• 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies

2021-04-20

• 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog (T273425)
• 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor (T271825)

2021-04-19

• 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental T280491
• 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
• 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for T280491

2021-04-17

• 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for T280420

2021-04-16

• 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks T234020 T280167
• 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
• 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch T278203
• 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs T252432
• 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref T235142
• 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref T198673
• 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref T198673
• 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
• 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
• 17:02 dancy: Updating dev-images docker-pkg files on primary contint
• 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template T252434

2021-04-15

• 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
• 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts

2021-04-14

• 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov T234020

2021-04-13

• 17:00 halfak: failed deploy to ORES (connection to host failed)
• 16:57 halfak: deploying ores f08a3cb
• 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
• 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 T280004
• 07:46 awight: enable syntax highlighting line numbering on all namespaces (T267911)

2021-04-12

• 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; T279853)
• 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # T279853
• 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per T106915#6279270 - T135427
• 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
• 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)

2021-04-11

• 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
• 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov T234020
• 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3{5,6} T274971
• 00:02 James_F: Docker: Publishing quibble-buster images T252432

2021-04-10

• 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list

2021-04-09

• 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, T233134.
• 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
• 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 T277750 phan for wikibase release branches
• 09:51 Majavah: deleting deployment-jobrunner03 T278664
• 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage T234080

2021-04-08

• 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
• 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563

2021-04-07

• 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
• 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
• 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
• 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 T278664
• 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
• 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382

2021-04-06

• 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived T277626
• 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs T279260
• 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines T279459
• 17:58 James_F: Zuul: [mediawiki/services/function-{orchestr,evalu}ator] Publish images

2021-04-05

• 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
• 17:45 brennen: halting gitlab-test for resize

2021-04-02

• 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
• 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12

2021-04-01

• 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
• 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
• 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
• 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298

2021-03-30

• 18:05 Majavah: remove {trysty,precise}-deployment-prep repos from deployment-deploy01 aptly
• 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
• 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those T278689
• 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
• 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
• 13:35 Majavah: create and install deployment-deploy03 T278689
• 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
• 07:26 Majavah: shutoff deployment-mediawiki-09 T278664
• 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
• 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11

2021-03-29

• 15:37 Majavah: hard reboot deployment-sessionstore03 T263617
• 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) T263617
• 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 (T278664), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
• 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
• 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
• 09:29 Urbanecm: Manually run puppet on mediawiki10
• 09:28 Urbanecm: Re-enable puppet on mediawiki10
• 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
• 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - T278664
• 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, T278664

2021-03-26

• 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
• 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 (T218729)

2021-03-25

• 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
• 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
• 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749

2021-03-24

• 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm T238707

2021-03-23

• 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per T274015
• 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job T207226
• 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
• 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade T252434
• 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template T252434
• 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch T252434
• 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch T252434
• 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use

2021-03-22

• 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
• 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
• 12:07 Majavah: delete deployment-restbase[01-02], T250574
• 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. (T276624)
• 11:33 dcaro: Created subzone beta.wmcloud.org (T276624)

2021-03-19

• 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
• 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
• 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, T218729
• 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
• 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down T250574
• 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for T250574

2021-03-18

• 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw (T277109)
• 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
• 18:27 brennen: updating gitlab-test to 13.9.4-ce
• 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 T277750
• 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
• 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # T269354
• 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 T277750 (apitests php versions)
• 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 T277060
• 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11 # T269354
• 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for T276179
• 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
• 08:44 Majavah: delete now unused deployment-ircd T277081
• 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now

2021-03-17

• 20:30 hashar: Reloaded Zuul for I236847
• 15:37 Majavah: shutdown deployment-restbase01 for T250574
• 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # T250574
• 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
• 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI

2021-03-16

• 21:37 longma: Updating dev-images docker-pkg files on primary contint
• 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
• 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
• 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
• 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
• 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1 # T277527

2021-03-15

• 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
• 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118

2021-03-13

• 17:10 twentyafterfour: restart apache on gerrit1001

2021-03-12

• 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
• 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
• 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
• 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
• 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ T276428

2021-03-11

• 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job(T277236)
• 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236)
• 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236)
• 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 T276419
• 16:49 Majavah: delete deployment-etcd-01 T276462
• 13:51 Majavah: shut down deployment-db06, now unused T277070
• 13:48 Majavah: set deployment-db07 as r/w T277070
• 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 T277070
• 13:41 Majavah: stop slave on deployment-db06 T276968
• 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 T277070
• 13:34 Majavah: stop and reset slave on deployment-db07 T277070
• 13:32 Majavah: set deployment-db06 as read only T277070
• 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/

2021-03-10

• 19:36 Majavah: shutdown deployment-ircd T277081
• 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 T277081
• 18:05 Majavah: create deployment-ircd02 for T277081
• 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
• 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
• 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
• 17:09 Majavah: set beta cluster mediawiki as read write on mw config (T276968)
• 17:03 Majavah: make deployment-db06 read-write T276968
• 16:50 Majavah: `reset slave;` on new master deployment-db06 T276968
• 16:49 Majavah: add deployment-db07 as a replica of db06 for T276968
• 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # T276968
• 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; (T276968)
• 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . (T276968)
• 15:57 Majavah: set deployment-db06 as readonly from mysql side T276968
• 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` (T276968)
• 15:54 Urbanecm: Start mariadb on db08 (T276968)
• 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 (T276968)
• 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
• 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
• 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade (T276968)
• 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) (T276968)

2021-03-09

• 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround (T276968)
• 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 (T276968)
• 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
• 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user (T276968)
• 20:33 Majavah: install mariadb on deployment-db08 T276968
• 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db (T276968)
• 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica (T276968)
• 19:50 marxarelli: restoring database dump on deployment-db07 (T276968)
• 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968)
• 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 T276968
• 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968)
• 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968
• 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump (T276968)
• 18:09 Majavah: set deployment-db05 to read-only to avoid issues with T276968
• 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968)
• 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
• 17:22 marxarelli: restarting deployment-db05 via horizon
• 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
• 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
• 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
• 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
• 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
• 13:32 arturo: hard-reboot deployment-db05 because issues related to T276922
• 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other

2021-03-08

• 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
• 08:57 hashar: Nuked castor cache for labs/striker # T276605

2021-03-07

• 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
• 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
• 17:18 Majavah: shutdown deployment-memc[04-05] T276707
• 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707
• 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707
• 15:44 Majavah: create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
• 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585
• 13:14 Majavah: create deployment-memc09 on Buster T276707

2021-03-06

• 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
• 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 T276654
• 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix T276652

2021-03-05

• 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
• 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
• 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
• 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462
• 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462
• 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462
• 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate T276462
• 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
• 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others
• 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2

2021-03-04

• 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384
• 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref T276102 and it being decomissioned in T242855
• 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for T276419, I believe everything has been moved to deployment-mwlog01
• 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for T276419
• 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
• 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
• 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref T276419
• 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
• 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # T276422 T180301
• 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
• 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch

2021-03-03

• 20:30 legoktm: added Majavah as projectadmin in deployment-prep (Beta Cluster)
• 19:59 James_F: Zuul: [mediawiki/services/function-schemata] Revert "Use bespoke pipeline jobs"
• 16:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Run phan with Echo
• 11:40 addshore: reload zuul for https://gerrit.wikimedia.org/r/667623
• 10:33 hashar: REplaced java-codehealth-patch job in favor of running sonar:sonar inline in all the java jobs. Thanks gehel! # T264873 | https://gerrit.wikimedia.org/r/c/integration/config/+/666869/

2021-03-02

• 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
• 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).

2021-03-01

• 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
• 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.

2021-02-27

• 22:03 Reedy: re-armed beta keyholder... I think...

2021-02-26

• 19:47 James_F: Zuul: [mediawiki/services/geoshapes] Add typescript service CI T274380
• 01:11 legoktm: update credentials in https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ for new ci-build user (T275559)

2021-02-24

• 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
• 22:16 legoktm: rebuilding Rust docker images

2021-02-23

• 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
• 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)

2021-02-20

• 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
• 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI

2021-02-19

• 13:51 hashar: Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897 did not get merged
• 13:49 hashar: Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." | https://gerrit.wikimedia.org/r/c/integration/config/+/651791/

2021-02-18

• 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
• 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata

2021-02-17

• 20:05 hashar: Updating all Jenkins jobs for https://gerrit.wikimedia.org/r/664897 # T275049
• 17:59 hashar: Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/664680 # T275049
• 03:26 James_F: Zuul: [mediawiki/core] PHP 8.0 version of composertest job to experimental

2021-02-16

• 20:33 brennen: updating gitlab-test.wmcloud.org to 13.8.4-ce.0
• 19:58 hashar: Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # T273822
• 17:02 greg-g: doing the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update dance
• 04:18 James_F: Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site
• 04:00 James_F: Zuul: Add Tim Abdullin from S&F to CI allow list

2021-02-15

• 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # T209953

2021-02-14

• 21:31 James_F: Zuul: Add 'check php' support for library repos
• 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive T274738

2021-02-13

• 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library

2021-02-12

• 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
• 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$

2021-02-11

• 21:44 Krinkle: Logstash in beta is not receiving any events T274593
• 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
• 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension T274069
• 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
• 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
• 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339

2021-02-10

• 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
• 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for T274306
• 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
• 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension

2021-02-09

• 22:27 brennen: once more, with feeling: attempting docker-pkg run for dev-images again
• 22:09 brennen: attempting to run docker-pkg manually for dev-images on contint2001
• 21:08 brennen: Updating dev-images docker-pkg files on primary contint for gerrit:635361 and gerrit:632173
• 20:10 James_F: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld
• 19:31 James_F: Docker: Building and publishing tox-buster &c. with tox 3.21.4 T274232
• 04:43 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref T273247
• 04:42 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify

2021-02-08

• 21:40 brennen: enabled administrative approval for new account signups on gitlab-test
• 21:40 brennen: upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0
• 17:22 hashar: Built image docker-registry.discovery.wmnet/releng/quibble-buster-php72:0.0.45-s4
• 17:17 hashar: Building some docker images on contint.wikimedia.org
• 16:51 hashar: Now really reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
• 16:43 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
• 14:55 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658946 # T272863
• 09:06 _joe_: cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 in deployment-prep

2021-02-06

• 21:13 Reedy: unstuck beta jobs

2021-02-04

• 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
• 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI T273247

2021-02-03

• 21:38 James_F: Zuul: Archive VirtualKeyboard extension T273801
• 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
• 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
• 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL T273295

2021-02-02

• 19:43 hashar: Pruning dangling Docker images on contint2001
• 19:39 hasharDinner: Pruning dangling Docker images on contint1001
• 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move T61245
• 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
• 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # T273599

2021-02-01

• 16:04 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta.
• 10:55 Urbanecm: deployment-prep: Create beta votewiki (T272608)
• 09:15 hashar: devtools: switched gerrit-prod-1001 to local puppetmaster

2021-01-29

• 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes

2021-01-28

• 22:15 marxarelli: deleting unused integration-registry-1003 instance
• 17:46 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
• 17:42 marxarelli: updating jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339

2021-01-27

• 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02

2021-01-26

• 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
• 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
• 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
• 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
• 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355

2021-01-25

• 15:57 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$

2021-01-22

• 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout

2021-01-19

• 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI T271761

2021-01-18

• 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker T272320

2021-01-17

• 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job T248925

2021-01-16

• 23:24 James_F: Docker: Building cascade of new php-ast image T271428

2021-01-14

• 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job

2021-01-13

• 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents

2021-01-12

• 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
• 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
• 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
• 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
• 19:33 James_F: Zuul: Revert Parsoid integration job injection.
• 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
• 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension T237538
• 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template

2021-01-11

• 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # T271683

2021-01-09

• 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing T271575
• 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing T271575

2021-01-08

• 02:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/654964

2021-01-07

• 18:34 James_F: Zuul: Provide experimental PHP 7.2-on-buster jobs T252434
• 18:32 James_F: Zuul: [labs/tools/massmailer] Add gate-and-submit-l10n jobs T271426
• 16:01 hashar: Tag Quibble 0.0.46 @ df9e75329ab # T225218 T266441 T263500

2021-01-06

• 19:36 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/620729 in beta puppet.
• 18:33 dpifke: Doing a test run in beta of the scap commands I'm going to run in today's backport window to roll out profiler changes.

2021-01-05

• 23:55 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/651267 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/621095/3 in beta. Should be a no-op (removes hopefully unused code).

2021-01-04

• 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job T269719
• 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled for now due to T271122
• 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot # T271122
• 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts

2021-01-02

• 19:13 James_F: Zuul: Add CI for the Mirage skin T270979

2021-01-01

• 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master T270976
• 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting T251967

Archives

• Archive 1 (September 2014 - December 2015)
• Archive 2 (2016)
• Archive 3 (2017)
• Archive 4 (2018)
• Archive 5 (2019)
• Archive 6 (2020)