Release Engineering/SAL

2022-02-22

• 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
• 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

• 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
• 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
• 07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

• 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

• 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
• 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
• 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

• 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for T293705
• 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for T293705
• 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 T301453
• 09:21 hashar: Reloading Zuul for I1494ab

2022-02-17

• 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
• 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
• 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
• 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
• 11:12 hashar: Bringing deployment-deploy03 back
• 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

• 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
• 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
• 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
• 10:09 hashar: Reloading Zuul for I997fee
• 09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

• 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
• 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
• 16:24 hashar: Restarting CI Jenkins for plugins updates
• 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
• 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
• 15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

• 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
• 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
• 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
• 14:26 jnuche: Jenkins upgrade complete T301361
• 13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

• 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

• 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602

2022-02-09

• 15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

• 17:34 taavi: remove scap from deployment-kafka-main/jumbo
• 16:23 taavi: hard reboot misbehaving deployment-echostore01
• 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

• 20:55 taavi: added Zabe as member of the deployment-prep project T301179
• 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550

2022-02-04

• 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622

2022-02-03

• 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
• 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
• 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

• 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
• 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

• 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
• 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged T300591
• 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster T300591

2022-01-31

• 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
• 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

• 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
• 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

• 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
• 19:34 Amir1: Reloading Zuul to deploy 757464
• 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
• 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
• 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
• 00:24 thcipriani: restarting jenkins

2022-01-26

• 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
• 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
• 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
• 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
• 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

• 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
• 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
• 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
• 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
• 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
• 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
• 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
• 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
• 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
• 09:22 Reedy: unblocked beta again
• 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

• 21:44 Reedy: unstick beta ci jobs
• 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
• 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
• 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
• 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

• 13:40 taavi: apply T299827 on deployment-prep centralauth database
• 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

• 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
• 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

• 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
• 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
• 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
• 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
• 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
• 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
• 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
• 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
• 12:10 hashar: contint2001 : docker container prune && docker image prune
• 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
• 12:04 hashar: contint1001 `docker image prune`
• 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

• 18:20 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
• 17:31 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # T299542
• 10:38 Reedy: kill some stuck jobs T299485

2022-01-18

• 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
• 18:01 taavi: added ryankemper as a member of the deployment-prep project
• 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
• 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
• 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
• 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

• 16:28 hashar: Building Quibble 1.3.0 Docker images
• 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
• 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

• 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
• 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
• 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
• 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

• 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
• 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
• 16:27 dancy: testing scap prep auto on deployment-deploy03
• 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
• 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
• 10:52 hashar: Restarting Jenkins CI for plugins update
• 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
• 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

• 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/

2022-01-11

• 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

• 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
• 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

• 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
• 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697 to get rid of the wmf-quibble-apache jobs # T285649
• 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
• 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/

2022-01-04

• 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms" https://gerrit.wikimedia.org/r/751422
• 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
• 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383

2022-01-03

• 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
• 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002

Archives

• Archive 1 (September 2014 - December 2015)
• Archive 2 (2016)
• Archive 3 (2017)
• Archive 4 (2018)
• Archive 5 (2019)
• Archive 6 (2020)
• Archive 7 (2021)
• Archive 8 (2022)