Release Engineering/SAL: Difference between revisions

Revision as of 18:39, 22 May 2022

2022-05-22

18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/

2022-05-21

23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as T308908

2022-05-20

16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809

2022-05-19

19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 | Don't re-trigger the test pipeline on patches with C+2 already

2022-05-18

19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071

2022-05-17

23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558

2022-05-16

19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday T299797
19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday T299797
08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809

2022-05-14

23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
20:25 urbanecm: add TheresNoTime (samtar) as a project member per request

2022-05-13

22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
22:52 James_F: Zuul: Add Tranve to CI allow list
22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk T299797
17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group T305317

2022-05-12

22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
15:05 brennen: gitlab-prod-1001.devtools: soft reboot
00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)

2022-05-11

23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster

2022-05-10

12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled | https://gerrit.wikimedia.org/r/c/integration/config/+/790641 | T300340
10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377

2022-05-09

21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for T295375
20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # T307963

2022-05-08

12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057

2022-05-06

12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071

2022-05-05

22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
14:25 hashar: Created integration-castor05
12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
07:45 hashar: deployment-prep: removed a few queued Jenkins builds from https://integration.wikimedia.org/ci/view/Beta/

2022-05-04

21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # T307590
20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only T307507

2022-05-03

23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682

2022-05-02

15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # T301766

2022-04-29

21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants (T305301)
18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for T297687

2022-04-28

20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for T293509
18:57 brennen: finished editing mediawiki-new-errors
18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
09:03 hashar: Gerrit upgraded to 3.4.4 at roughly 8:00 UTC

2022-04-27

19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # T292759
15:20 hashar: Updating non-quibble jobs to composer 2.3.3 | T303867 | https://gerrit.wikimedia.org/r/c/integration/config/+/777029

2022-04-26

15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204

2022-04-25

17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI

2022-04-20

16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

2022-04-18

19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
16:48 brennen: phabricator: adding self to acl*blog-admins
15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.

2022-04-14

22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh

2022-04-12

21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes (T304290, T305143)
16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141

2022-04-08

11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287

2022-04-07

06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461

2022-04-06

20:03 thcipriani: rebooting phabricator
11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596

2022-04-04

22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for T294260
18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
08:54 jnuche: redeploying Zuul

2022-04-02

12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases

2022-03-31

20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957

2022-03-29

14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771

2022-03-28

16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # T252071
16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 T252071
10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` T220606
10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`. They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
09:53 hashar: Tag Quibble 1.4.5 @ abe16d574 | T291549

2022-03-27

13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736

2022-03-26

02:37 Reedy: beta-update-databases-eqiad is back to @hourly

2022-03-25

23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord

2022-03-24

20:00 hashar: reloading Zuul for Id844e1 # T299320
20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck

2022-03-23

17:35 brennen: restarting phabricator for T304540, brief downtime expected
14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster

2022-03-22

14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` T304226
13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945

2022-03-21

08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203

2022-03-18

14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)

2022-03-17

19:11 hashar: Building Docker images for Quibble 1.4.4
19:06 hashar: Tag Quibble 1.4.4 @ 56b2c9ba52c # T300340
16:25 hashar: Switching Quibble jobs to use memcached rather than APCu | https://gerrit.wikimedia.org/r/c/integration/config/+/770468 | T300340
14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable | https://gerrit.wikimedia.org/r/770921 | T216244 | T252071
14:07 hashar: Building Docker image to support `CASTOR_HOST` | https://gerrit.wikimedia.org/r/770921 | T216244

2022-03-16

22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for T303958
20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for T302607 and T302608
19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep

2022-03-15

18:26 brennen: gitlab: removed most existing /people groups
18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership (T274461, T300935)
16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # T216244
14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # T216244
01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job

2022-03-14

23:57 James_F: Zuul: [ooui] Switch from node12 to node14
23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079

2022-03-11

04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki

2022-03-10

20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
20:03 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/768843
15:12 hashar: updating Quibble jenkins jobs
14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for T250496 / T303417.
11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
09:59 dwalden: restarted apache on deployment-mediawiki11 # T302699

2022-03-09

17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails | https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 | https://phabricator.wikimedia.org/T288312

2022-03-08

20:31 brennen: requiring 2fa for all users under /repos

2022-03-07

10:53 zabe: restarted apache on deployment-mediawiki11 # T302699

2022-03-04

20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
19:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768068

2022-03-03

19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for T278203
14:43 hashar: Reloading Zuul for Iae45ca
12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
10:30 hashar: Building Docker images for Quibble 1.4.3
10:22 hashar: Tagged Quibble 1.4.3 @ cf5cd1a0a07
09:24 hashar: Building Docker images for Quibble 1.4.2
09:20 hashar: Tag Quibble 1.4.2 @ 63d2855a1e # T302226 T302707

2022-03-02

19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
15:04 taavi: resolve merge conflicts on deployment-puppetmaster04

2022-02-28

19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
19:17 brennen: adding mutante (dzahn) as application-level gitlab admin

2022-02-26

20:05 zabe: apply T302658 on deployment-prep centralauth databases
13:24 zabe: apply T302660 on deployment-prep centralauth databases
13:19 zabe: apply T302659 on deployment-prep centralauth databases

2022-02-24

16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1

2022-02-23

23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
20:10 dancy: Updating scap in beta
19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774

2022-02-22

16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

12:19 taavi: restart trafficserver-tls on deployment-cache-text06
02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for T293705
22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for T293705
13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 T301453
09:21 hashar: Reloading Zuul for I1494ab

2022-02-17

21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
11:12 hashar: Bringing deployment-deploy03 back
11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
10:09 hashar: Reloading Zuul for I997fee
09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
16:24 hashar: Restarting CI Jenkins for plugins updates
16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
14:26 jnuche: Jenkins upgrade complete T301361
13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602

2022-02-09

15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

17:34 taavi: remove scap from deployment-kafka-main/jumbo
16:23 taavi: hard reboot misbehaving deployment-echostore01
13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

20:55 taavi: added Zabe as member of the deployment-prep project T301179
18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550

2022-02-04

00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622

2022-02-03

18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged T300591
00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster T300591

2022-01-31

19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
19:34 Amir1: Reloading Zuul to deploy 757464
16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
00:24 thcipriani: restarting jenkins

2022-01-26

20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
09:22 Reedy: unblocked beta again
07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

21:44 Reedy: unstick beta ci jobs
21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

13:40 taavi: apply T299827 on deployment-prep centralauth database
11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
12:10 hashar: contint2001 : docker container prune && docker image prune
12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
12:04 hashar: contint1001 `docker image prune`
11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

18:20 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
17:31 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # T299542
10:38 Reedy: kill some stuck jobs T299485

2022-01-18

19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
18:01 taavi: added ryankemper as a member of the deployment-prep project
15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

16:28 hashar: Building Quibble 1.3.0 Docker images
16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
16:27 dancy: testing scap prep auto on deployment-deploy03
15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
10:52 hashar: Restarting Jenkins CI for plugins update
10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/

2022-01-11

09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697 to get rid of the wmf-quibble-apache jobs # T285649
10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/

2022-01-04

12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms" https://gerrit.wikimedia.org/r/751422
09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383

2022-01-03

14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
14:35 hashar: Upgraded Java 11 on releases1002 && releases2002