You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(mutante: releases1002 - releases2002, something broke puppet here about 27.5 hours ago. lookup() did not find a value for the name 'profile::docker::storage::physical_volumes')
imported>Stashbot
(James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage)
(186 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== 2021-10-28 ==
== 2022-08-09 ==
* 22:06 mutante: releases1002 - releases2002, something broke puppet here about 27.5 hours ago. lookup() did not find a value for the name 'profile::docker::storage::physical_volumes'
* 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
* 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
* 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
* 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for [[phab:T293924|T293924]]
* 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for [[phab:T293851|T293851]]


== 2021-10-27 ==
== 2022-08-08 ==
* 16:51 hashar: Tag Quibble 1.2.0 @ {{Gerrit|bdabd84}} # [[phab:T259456|T259456]] [[phab:T292772|T292772]] [[phab:T256402|T256402]]
* 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
* 13:33 majavah: removed role::beta::puppetmaster in https://gerrit.wikimedia.org/r/c/operations/puppet/+/734962/
* 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for [[phab:T309872|T309872]]
* 12:39 hashar: reloaded Zuul for [GrowthExperiments] Add CirrusSearch to phan dependencies - https://gerrit.wikimedia.org/r/734954
* 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for [[phab:T314630|T314630]]
* 12:22 hashar: integration-castor03: sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node12-docker # [[phab:T294426|T294426]] [[phab:T293937|T293937]]
* 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166


== 2021-10-26 ==
== 2022-08-05 ==
* 20:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734728
* 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for [[phab:T296967|T296967]]
* 10:17 hashar: Updating all Jenkins job to remove `--compress` option from rsync when saving cache {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/734412 {{!}} [[phab:T188375|T188375]]
* 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
* 08:29 hashar: zuul: reverted Enable PHP74 jobs on gate-and-submit-wmf pipeline"  https://gerrit.wikimedia.org/r/734325
* 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for [[phab:T314630|T314630]]
* 07:55 hashar: Created #wikimedia-quibble IRC channel for https://doc.wikimedia.org/quibble/
* 14:48 James_F: Zuul: Add WelpThatWorked to allow list
* 04:59 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [[phab:T293924|T293924]]
* 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin


== 2021-10-25 ==
== 2022-08-04 ==
* 23:27 dduvall: fully provisioned runner-<nowiki>{</nowiki>1008,1011,1012,1013,1014,1015,1016,1017,1018,1019<nowiki>}</nowiki> instances for use as new gitlab runners and removed old instances ([[phab:T293835|T293835]])
* 15:21 dancy: Deleting beta-mediawiki-config-update-eqiad job
* 20:50 James_F: Docker: Publishing php*-comile images without the PECL test so they work again.
* 15:16 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405
* 20:32 James_F: Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72
* 10:01 TheresNoTime: clearing out stuck beta deployment jobs [[phab:T314378|T314378]] [[phab:T72597|T72597]]
* 19:21 James_F: Docker: Publishing new php74 and cascaded images with PHP 7.4 from Wikimedia package [[phab:T293851|T293851]]
* 18:45 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734377 [[phab:T294238|T294238]]
* 18:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734370 https://phabricator.wikimedia.org/T294238
* 17:01 dduvall: deleting bullseye runner-1006 and reverting to using buster due to some puppet provisioning issues
* 16:17 dduvall: launching new runner-1006 instance under gitlab-runners


== 2021-10-24 ==
== 2022-08-03 ==
* 08:52 hashar: CI Jenkins: update Clover PHP plugin from 0.5 to 0.6 to solve `java.lang.NoClassDefFoundError: org/apache/commons/digester/Digester`  # [[phab:T292910|T292910]]
* 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
* 06:58 majavah: hard reboot crashed deployment-db08
* 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
* 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
* 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
* 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for [[phab:T309872|T309872]]
* 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
* 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too


== 2021-10-22 ==
== 2022-08-02 ==
* 21:19 James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change [[phab:T292910|T292910]]
* 19:26 James_F: Zuul: [design/codex] Switch coverage job back to -direct
* 18:06 dancy: Updating deb packages on deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud
* 15:23 dancy: Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028)
* 00:31 James_F: Zuul: [DiscussionTools] Add EventLogging as dependency for Phan jobs
* 15:22 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028
* 07:55 TheresNoTime: cleared stuck beta deployment jobs [[phab:T72597|T72597]]


== 2021-10-21 ==
== 2022-08-01 ==
* 18:07 dduvall: replacing runner-1001 with new instance ([[phab:T293835|T293835]])
* 23:16 James_F: Zuul: [design/codex] Switch to node16
* 09:49 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test [[phab:T292728|T292728]]
* 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
* 22:53 TheresNoTime: remove stuck beta deployment jobs
* 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
* 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
* 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
* 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
* 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
* 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for [[phab:T314306|T314306]]
* 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
* 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
* 09:14 TheresNoTime: clearing stuck beta CI jobs


== 2021-10-19 ==
== 2022-07-29 ==
* 22:24 brennen: gitlab-test: deleted unused gitlab-runner-test
* 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for [[phab:T313919|T313919]]
* 22:20 brennen: adding Dzahn and Aokoth to gitlab-test and gitlab-runners projects, cleaning up contractor access
* 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes
* 21:20 brennen: Updating dev-images docker-pkg files on primary contint for php 8 images ([[phab:T283426|T283426]])
* 18:00 James_F: Zuul: [mediawiki/services/function-schemata] Switch from pipeline to direct CI
* 11:32 hashar: Archiving integration/gear.git , no more used after Python gear 0.16.0 has been released # [[phab:T289512|T289512]]


== 2021-10-18 ==
== 2022-07-28 ==
* 16:26 Lucas_WMDE: killed one beta-scap-sync-world job that seemed stuck for a long time
* 17:54 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/818189/
* 14:43 James_F: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [[phab:T290274|T290274]]
* 09:41 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # [[phab:T248676|T248676]]


== 2021-10-16 ==
== 2022-07-27 ==
* 16:47 Lucas_WMDE: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T293070|T293070]], based on [[phab:T271808|T271808]]#6739578
* 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch [[phab:T300463|T300463]]
* 01:17 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/730949
* 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for [[phab:T274965|T274965]]
* 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
* 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
* 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref [[phab:T143162|T143162]], [[phab:T193565|T193565]]


== 2021-10-15 ==
== 2022-07-25 ==
* 16:14 brennen: gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for [[phab:T292372|T292372]]
* 22:16 dduvall: re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 21:25 dduvall: disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769


== 2021-10-14 ==
== 2022-07-23 ==
* 16:13 addshore: reloaded zuul for zuul: mw-ext-MailGun will need PHP 7.3+ [[phab:T236396|T236396]]
* 17:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/816251
* 14:13 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/730207


== 2021-10-13 ==
== 2022-07-21 ==
* 23:12 bd808: Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors.
* 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster
* 16:48 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T293277|T293277]]
* 11:56 hashar: integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes'


== 2021-10-12 ==
== 2022-07-20 ==
* 19:11 hashar: Building Docker images for https://gerrit.wikimedia.org/r/730279
* 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
* 17:28 James_F: Zuul: [mediawiki/extensions/THICC] Mark as archived for [[phab:T292948|T292948]]
* 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline


== 2021-10-11 ==
== 2022-07-19 ==
* 15:35 hashar: gerrit: marked integration/composer read-only. We are no more using it and the few use cases left are tracked at https://phabricator.wikimedia.org/T249949
* 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
* 02:08 Krinkle: Browsers fail to connect with https://upload.wikimedia.beta.wmflabs.org/ (Certificate expired two days ago)
* 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # [[phab:T310905|T310905]]


== 2021-10-08 ==
== 2022-07-18 ==
* 21:17 hashar: Purging Docker images on all CI agents
* 19:43 dancy: Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster
* 20:00 hashar: Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620
* 13:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # [[phab:T311752|T311752]]
* 19:23 hashar: Building CI Docker images for Quibble 1.1.1
* 10:40 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745
* 19:08 hashar: Tag quibble 1.1.1 @ {{Gerrit|b54af2aa60}}
* 09:58 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included
* 19:07 brennen: gitlab1001: restarting to check correct session_duration value of 604800
* 09:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/814728
* 11:01 hashar: Rebuilding CI Docker images based on Bullseye to include apt security updates


== 2021-10-07 ==
== 2022-07-17 ==
* 22:12 James_F: Zuul: [design/codex] Provide initial CI, node14-browser suite
* 13:00 taavi: reloading zuul for https://gerrit.wikimedia.org/r/814356
* 17:07 James_F: Zuul: [mediawiki/extensions/ParsoidBatchAPI] Mark as archived [[phab:T252744|T252744]]
* 13:29 hashar: restarting CI Jenkins for git plugin update
* 13:14 hashar: Upgraded CI Jenkins on contint2001
* 13:10 hashar: Upgraded releases Jenkins from 2.289.2 to  2.303.2
* 13:04 hashar: Updating Jenkins jobs for Quibble 1.1.0 # [[phab:T291549|T291549]]
* 04:46 Krinkle: Temporarily change `mediawiki-fresnel-patch-docker` job change back to quibble-fresnel:1.0.1 ref [[phab:T292684|T292684]]
* 04:41 Krinkle: Reconfigure `mediawiki-fresnel-patch-docker` job to revert quibble change back to quibble-fresnel:1.0.1
* 01:06 Reedy: unstick beta ci jobs


== 2021-10-06 ==
== 2022-07-16 ==
* 11:19 hashar: Tag Quibble 1.1.0 @ {{Gerrit|3dabcab}}
* 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt


== 2021-10-05 ==
== 2022-07-15 ==
* 21:47 brennen: updating dev-images docker-pkg files on primary contint for https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/7
* 15:59 hashar: Built pcov php docker images [[phab:T280170|T280170]]
* 12:03 majavah: root@deployment-cache-text06:~# systemctl restart sssd # [[phab:T286502|T286502]]
* 15:46 hashar: contint2001: `docker-system-prune-dangling.service`  it failed overnight cause Docker was not running. That should clear Icinga state # [[phab:T313119|T313119]]
* 11:11 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/726576
* 14:05 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Switch to node16 jobs
* 13:05 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]], this time actually.
* 12:30 hashar: Starting docker on contint2001.wikimedia.org # [[phab:T313119|T313119]]
* 12:20 hashar: rebuilding `php??` images for pcov https://gerrit.wikimedia.org/r/c/integration/config/+/694621 # [[phab:T280170|T280170]]
* 10:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813967
* 10:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813932


== 2021-10-04 ==
== 2022-07-14 ==
* 21:09 brennen: finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 ([[phab:T292256|T292256]])
* 18:50 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]]
* 21:02 brennen: gitlab-runners: upgrading each shared runner to 14.3.2
* 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for [[phab:T203215|T203215]]
* 18:12 James_F: Docker: Publish composer 2.x-based images for [[phab:T279857|T279857]]
* 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
* 16:46 James_F: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
* 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for [[phab:T268085|T268085]]
* 15:54 James_F: Zuul: Add Lindsay Wardell personal e-mail to CI allow list
* 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI
* 14:31 brennen: gitlab1001: upgrading to 14.3.2 ([[phab:T292256|T292256]])


== 2021-09-30 ==
== 2022-07-13 ==
* 21:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/683031
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813720
* 20:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813707


== 2021-09-29 ==
== 2022-07-12 ==
* 15:52 hashar: Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # [[phab:T277500|T277500]]
* 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki ([[phab:T312865|T312865]])
* 15:03 brennen: gitlab: pausing shared runners while working through access-level implications


== 2021-09-27 ==
== 2022-07-11 ==
* 20:52 James_F: Zuul: [integration/visualdiff] Add basic node12 CI job
* 22:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/812934
* 20:15 brennen: gitlab1001: login and user rename as "brennen" checks out; applying user mappings for remaining ~80 users
* 19:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/812467
* 19:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/724103/3
* 19:24 hashar: Updating Jenkins jobs using a Stretch related image to latest version of the image. https://gerrit.wikimedia.org/r/c/integration/config/+/724156/ # [[phab:T291425|T291425]]
* 17:23 hashar: Building java8-scala-spark image
* 07:11 hashar: Switching python based jobs to docker-registry.wikimedia.org/releng/tox-buster:0.5.0 # [[phab:T291292|T291292]]


== 2021-09-24 ==
== 2022-07-10 ==
* 15:41 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/coal/+/722948 and latest https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep.  Should only affect deployment-webperf11.
* 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet


== 2021-09-23 ==
== 2022-07-09 ==
* 19:55 brennen: gitlab-ansible-test: sudo gitlab-ctl cleanse to drop test data and reset
* 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
* 18:41 James_F: Zuul: Create 'extension-apitests' template for simplicity
* 17:25 ori: Cherry-picked {{Gerrit|Ief73cc553}} (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
* 15:54 brennen: gitlab1001: brief downtime to apply [[gerrit:714382{{!}}gitlab cas: uid instead of CN; add nickname_key]] for [[phab:T288392|T288392]]
* 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
* 15:13 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/723217
* 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
* 14:46 hashar: Building releng/tox-buster:0.5.0 image
* 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump  # ref [[phab:T312689|T312689]]
* 11:44 hashar: Reloaded zuul for "zuul: promote shellcheck job for integration/config" https://gerrit.wikimedia.org/r/c/integration/config/+/723160/1
* 10:57 hashar: Building tox images for improved shellcheck package https://gerrit.wikimedia.org/r/721895


== 2021-09-22 ==
== 2022-07-07 ==
* 15:47 Amir1: set row format for text table in simplewiki to compressed
* 22:42 TheresNoTime: clear stuck beta deployment jobs (again), [[phab:T72597|T72597]]
* 15:22 Amir1: restart mysql on db07 to add performance schema
* 21:10 TheresNoTime: clear stuck beta deployment jobs, [[phab:T72597|T72597]]
* 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
* 12:22 hashar: integration: rebooting `integration-agent-docker-1039` [[phab:T312534|T312534]]


== 2021-09-21 ==
== 2022-07-05 ==
* 16:26 dduvall: deleted wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to a branch cut job gone awry. re-running job to cut wmf/1.38.0-wmf.1 ([[phab:T281165|T281165]])
* 14:17 dwalden: restarted mathoid service on deployment-docker-mathoid01
* 16:04 dduvall: deleting wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to the branch cut job gone awry
* 11:39 hashar: Reloaded Zuul for `skip selenium for Wikibase repo/rest-api` https://gerrit.wikimedia.org/r/c/integration/config/+/811258
* 15:49 James_F: Re-running the branch cut job for [[phab:T281165|T281165]]
* 08:49 hauskatze: Diffusion rORES repository. Changed URI settings: enabled SSH push for mirroring; disabled HTTP {{!}} [[phab:T311390|T311390]]
* 15:48 James_F: Updated the releases-jenkins automatic branch cut job to make 1.38.0 branches (missed ahead of [[phab:T281165|T281165]])


== 2021-09-17 ==
== 2022-06-30 ==
* 19:05 hashar: Building Docker images for [tox-buster] Install shellcheck and cascade [integration/config] - https://gerrit.wikimedia.org/r/721881
* 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at [[phab:T72597|T72597]]
* 18:08 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
* 18:07 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T28477|T28477]]
* 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053
* 16:19 dpifke: Enabled TLS on Jumbo Kafka instances in deployment-prep.


== 2021-09-16 ==
== 2022-06-29 ==
* 22:17 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/721567 in deployment-prep, should only affect deployment-webperf11.
* 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
* 16:51 addshore: reload zuul for Promote branchdeploy for query-builder to test https://gerrit.wikimedia.org/r/721581 (https://phabricator.wikimedia.org/T278706)
* 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')
* 13:27 Reedy: Actually reloading Zuul to deploy  https://gerrit.wikimedia.org/r/683589
* 13:01 addshore: reload zuul for https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) 1 experimental branchdeploy job


== 2021-09-15 ==
== 2022-06-28 ==
* 22:59 dpifke: Un-cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep for now; more work needed.
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 22:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/721044 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep; should only affect deployment-webperf11.
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 08:58 Amir1: update wb_changes_dispatch set chd_disabled = 1 where chd_site = 'enwiki'; ([[phab:T290985|T290985]])
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]


== 2021-09-14 ==
== 2022-06-27 ==
* 17:17 Amir1: delete from wb_changes_dispatch where chd_db = 'enwiki'; ([[phab:T290985|T290985]])
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406


== 2021-09-13 ==
== 2022-06-24 ==
* 09:33 hashar: Castor cache: nuked files that were last changed more than six months ago to free up disk space
* 20:52 taavi: added `denisse` as a member


== 2021-09-10 ==
== 2022-06-23 ==
* 21:52 James_F: Created experimental integration-agent-docker-1021 for [[phab:T252071|T252071]]
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
* 21:48 James_F: Deleting CI agent integration-agent-docker-1001 for [[phab:T252071|T252071]]
* 21:44 James_F: Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for [[phab:T252071|T252071]]
* 21:23 James_F: Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job
* 17:41 James_F: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI
* 02:18 James_F: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12
* 02:13 James_F: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job
* 02:10 James_F: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12
* 01:58 James_F: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental
* 01:48 James_F: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices


== 2021-09-09 ==
== 2022-06-22 ==
* 22:20 brennen: gitlab-ansible-test: resetting instance data
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 19:42 James_F: Docker: Building node<nowiki>{</nowiki>10,12<nowiki>}</nowiki>-test-browser-php80-composer for [[phab:T290651|T290651]]
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 10:56 hashar: Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 10:37 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2021-09-08 ==
== 2022-06-21 ==
* 20:16 thcipriani: self +2 on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719500 to unbreak beta
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 18:47 brennen: contint1001 / contint2001: /srv/dev-images: git remote set-url origin 'https://gitlab.wikimedia.org/releng/dev-images.git'
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 18:28 dduvall: Running ./fab deploy_docker to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719350
* 17:05 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355
* 16:37 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719493
* 14:55 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/714328/
* 14:05 brennen: runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners ([[phab:T289802|T289802]])


== 2021-09-07 ==
== 2022-06-20 ==
* 20:49 James_F: Marked https://gerrit.wikimedia.org/g/mediawiki/tools/cli as read-only and pointed users to GitLab.
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 20:46 brennen: migrating dev-images to https://gitlab.wikimedia.org/releng/dev-images and deactivating on gerrit
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 18:16 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 18:11 dduvall: creating 2 new jenkins jobs for deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 16:25 James_F: Docker: Publishing node12-test-browser-php<nowiki>{</nowiki>72,80<nowiki>}</nowiki>-composer images
* 16:19 James_F: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add 4 dependencies


== 2021-09-03 ==
== 2022-06-17 ==
* 23:02 Krinkle: Creating integration-agent-qemu-1002 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 17:42 dduvall: deploying blubberoid:2021-09-03-160524-production to eqiad/codfw (https://gerrit.wikimedia.org/r/c/blubber/+/716519) ([[phab:T289367|T289367]])
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 17:36 dduvall: staging blubberoid to deploy https://gerrit.wikimedia.org/r/c/blubber/+/716519


== 2021-09-02 ==
== 2022-06-16 ==
* 15:17 brennen: gitlab-test: testing upgrade path to 14.x
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]


== 2021-09-01 ==
== 2022-06-15 ==
* 21:22 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/716041 in beta.
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 16:55 urbanecm: deployment-prep: Unlock scap
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 15:50 urbanecm: deployment-prep: Lock scap again
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
* 15:40 urbanecm: deployment-prep: Lock scap to be able to test something
* 14:08 urbanecm: deployment-prep: Create foundationwiki ([[phab:T290164|T290164]])
* 14:07 urbanecm: urbanecm@deployment-mediawiki11:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]
* 13:58 urbanecm: urbanecm@deployment-cache-text06:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]


== 2021-08-31 ==
== 2022-06-14 ==
* 21:38 dduvall: deploying new blubberoid to eqiad/codfw following successful testing in staging
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 21:35 dduvall: staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 14:29 hashar: Restarting CI Jenkins for plugins upgrade
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production


== 2021-08-30 ==
== 2022-06-13 ==
* 19:53 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 19:27 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
* 19:16 brennen: gitlab-test: powering off gitlab (former main test instance)
* 19:11 James_F: Zuul: [mediawiki/services/apple-search] Add postmerge publish
* 19:10 brennen: gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test
* 18:23 brennen: gitlab-test: associating floating IP to primary test box
* 16:36 James_F: Zuul: [mediawiki/services/apple-search] Remove composer-package
* 15:16 James_F: Zuul: [mediawiki/services/apple-search] Add pipeline CI for [[phab:T289224|T289224]]
* 12:20 Amir1: foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5


== 2021-08-27 ==
== 2022-06-12 ==
* 19:24 James_F: Docker: Publish initial node14 CI images for [[phab:T267888|T267888]]
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777


== 2021-08-25 ==
== 2022-06-10 ==
* 17:48 twentyafterfour: updating zuul config after deploying https://gerrit.wikimedia.org/r/714538
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check


== 2021-08-24 ==
== 2022-06-09 ==
* 23:15 James_F: Zuul: Configure the REL1_37 test and gate pipelines [[phab:T289587|T289587]]
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 22:18 thcipriani: phab1001:sudo /srv/phab/phabricator/bin/bulk make-silent --id 2822 (releng-logspam -> unstewarded production error)
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration
* 20:02 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/712990


== 2021-08-23 ==
== 2022-06-08 ==
* 20:37 James_F: Docker: Publish php-ast with 1.0.14 ([[phab:T289429|T289429]]) and no longer support PHP 7.0 or 7.1 (last trace!)
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134


== 2021-08-21 ==
== 2022-06-06 ==
* 15:03 majavah: fixing deployment-prep puppet merge conflicts re: swift
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]


== 2021-08-20 ==
== 2022-06-02 ==
* 21:11 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging/private$ rm mwblocker.log # remove weird blank log file
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 18:56 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/713439
* 11:26 hashar: Restarting Jenkins on contint2001
* 18:55 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 11:19 hashar: Restarting Jenkins on releases1002
* 18:54 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do echo "test $i" {{!}} mwscript edit.php --wiki=<nowiki>{</nowiki>cswiki,enwiki<nowiki>}</nowiki> --user="Martin Urbanec (test $i)" --summary="test" Sandbox; done
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=enwiki
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 18:46 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki "Martin Urbanec (test $i)"; done
* 18:40 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript createAndPromote.php --wiki=cswiki "Martin Urbanec (test $i)" "$password"; done # to test a feature that needs a lot of different accounts
* 16:30 majavah: restart sssd on deployment-cache-text06, [[phab:T286502|T286502]]?
* 16:24 majavah: deployment-prep: configure wikifunctions.beta.wmflabs.org dns zones and add to acme-chief [[phab:T284162|T284162]]


== 2021-08-19 ==
== 2022-05-31 ==
* 22:58 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/713612/
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2021-08-17 ==
== 2022-05-30 ==
* 21:23 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:709831{{!}}add buster-apache2, log errors to stdio]] ([[phab:T283416|T283416]])
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]


== 2021-08-16 ==
== 2022-05-28 ==
* 23:08 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/711731
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]


== 2021-08-15 ==
== 2022-05-27 ==
* 17:44 James_F: Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2021-08-13 ==
== 2022-05-26 ==
* 20:09 urbanecm: Manually start `beta-update-databases-eqiad` CI job
* 18:33 dancy: Updated Jenkins beta-* job configs
* 20:06 urbanecm: deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 20:03 urbanecm: Kill beta-scap-sync-world job for the usual reason
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch
* 13:13 majavah: `mwscript extensions/CentralAuth/maintenance/importMissingLocalNames.php --wiki metawiki` on the beta cluster


== 2021-08-11 ==
== 2022-05-25 ==
* 00:52 James_F: Zuul: Add Aca to the CI allow list
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 00:52 James_F: Zuul: [mediawiki/extensions/SimpleCalendar] Add basic quibble CI
* 15:46 dancy: Restarted apache2 on gerrit1001


== 2021-08-10 ==
== 2022-05-24 ==
* 16:57 James_F: Zuul: Update e-mail address for Zabe in the allow list
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 16:06 James_F: Ran `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Wikibase` on doc1001 for [[phab:T288396|T288396]]
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update


== 2021-08-09 ==
== 2022-05-23 ==
* 21:02 urbanecm: Remove hanging beta-scap-sync-world job in CI to unblock beta auto-updates
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2021-08-07 ==
== 2022-05-22 ==
* 00:48 James_F: Docker: Publish quibble-buster-php73-coverage 1.1.1 for [[phab:T287918|T287918]].
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 00:29 James_F: Zuul: Add skin-coverage jobs to all Wikimedia production skins [[phab:T287918|T287918]]
* 00:27 James_F: Zuul: Provide a skin-coverage template [[phab:T287918|T287918]]


== 2021-08-06 ==
== 2022-05-21 ==
* 23:53 James_F: Docker: Publishing quibble-buster-php73-coverage 1.1.0 for [[phab:T287918|T287918]]
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 23:47 James_F: Zuul: [mediawiki/skins/Mirage] Not a production skin; move to right section
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]


== 2021-08-05 ==
== 2022-05-20 ==
* 22:26 brennen: gitlab: setting CI access on all repos to "Project members only", per https://www.mediawiki.org/wiki/GitLab/Policy#Permissions - may need revisited depending on effects
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
* 22:04 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/710108
* 15:10 hashar: integration: sudo cumin --force 'name:docker' 'docker container prune -f && docker image prune -f'
* 15:09 hashar: integration-agent-docker-1020: docker container prune && docker image prune
* 15:02 hashar: Building Docker image helm-linter for https://gerrit.wikimedia.org/r/c/integration/config/+/710276


== 2021-08-04 ==
== 2022-05-19 ==
* 22:53 bd808: Updated composer-github-oauthtoken in Jenkins config to us a newer personal access token from GitHub per notices about https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 17:49 brennen: gitlab-test: testing upgrade to 13.12.9
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already
* 16:41 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/710066 # [[phab:T288111|T288111]]
* 15:13 thcipriani: puppet fixed on deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki>
* 15:08 thcipriani: rebase deployment-puppetmaster04:labs/private causing deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki> failure for...¯\_(ツ)_/¯


== 2021-08-03 ==
== 2022-05-18 ==
* 21:12 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:698880{{!}}add buster php images]]
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]


== 2021-08-02 ==
== 2022-05-17 ==
* 23:27 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/709563
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709562
* 23:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709558
* 22:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709555 https://gerrit.wikimedia.org/r/709556
* 21:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709543


== 2021-07-30 ==
== 2022-05-16 ==
* 21:27 dduvall: "Total reclaimed space: 141.4GB" on releases1002 following docker prune
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 21:24 dduvall: running `docker system prune -af` on releases1002
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809


== 2021-07-29 ==
== 2022-05-14 ==
* 22:12 jeena: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/708596
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 14:06 brennen: gerrit: added ldap/ops to ownership of operations/gitlab-ansible
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 13:21 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/708764 "Add michaelcochez to whitelist users"
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request


== 2021-07-28 ==
== 2022-05-13 ==
* 17:53 andrewbogott: rebooting deployment-logstash03 as it's in an inconsistent config state
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 14:44 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/708536
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2021-07-27 ==
== 2022-05-12 ==
* 16:55 dduvall: creating new gitlab runner instance runner-1002 for testing
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 16:32 hashar: cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2021-07-26 ==
== 2022-05-11 ==
* 20:45 brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation ([[phab:T287279|T287279]])
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 18:10 dduvall: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
* 18:00 dduvall: creating 2 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 08:24 hashar: jjb: update jobs to Quibble 1.0.1  # https://gerrit.wikimedia.org/r/c/integration/config/+/708038


== 2021-07-23 ==
== 2022-05-10 ==
* 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner ([[phab:T287279|T287279]])
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 14:08 hashar: Building Docker images for quibble 1.0.1
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 13:45 hashar: Tag Quibble 1.0.1 @ {{Gerrit|5a2548699a}} # [[phab:T287001|T287001]]
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377


== 2021-07-21 ==
== 2022-05-09 ==
* 21:06 brennen: gitlab1001: running ansible for logging typo fix ([[phab:T274462|T274462]])
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 20:36 dancy: Newest scap deployed to beta cluster
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes ([[phab:T274462|T274462]], [[phab:T275170|T275170]])
* 16:46 dancy: restarting Gerrit to fix plugins
* 16:07 dancy: Updating plugins on releases-jenkins
* 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki


== 2021-07-20 ==
== 2022-05-08 ==
* 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar ([[phab:T281156|T281156]])
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47  # [[phab:T287001|T287001]]


== 2021-07-19 ==
== 2022-05-06 ==
* 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]


== 2021-07-18 ==
== 2022-05-05 ==
* 08:48 majavah: set shared_acme_certificates: <nowiki>{</nowiki><nowiki>}</nowiki> on deployment-prep shared hiera, [[phab:T276653|T276653]]
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins  builds from https://integration.wikimedia.org/ci/view/Beta/


== 2021-07-16 ==
== 2022-05-04 ==
* 21:01 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/698774
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 17:44 brennen: Try #2: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 17:12 hashar: Tag Quibble 1.0.0 @ {{Gerrit|a13d133f7d1}} # [[phab:T286187|T286187]] [[phab:T280506|T280506]] [[phab:T90875|T90875]] [[phab:T218534|T218534]] [[phab:T227352|T227352]]
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2021-07-14 ==
== 2022-05-03 ==
* 22:47 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 21:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in beta puppet again, should only affect deployment-webperf12.
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2021-07-13 ==
== 2022-05-02 ==
* 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep. Should only affect deployment-webperf12.
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for [[phab:T281430|T281430]]
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]
* 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived [[phab:T281430|T281430]]
* 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet.  Should only affect deployment-webperf12.
* 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles  # [[phab:T278990|T278990]]
* 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
* 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # [[phab:T278990|T278990]]
* 07:46 hashar: Wiping all Docker images from contint2001l


== 2021-07-12 ==
== 2022-04-29 ==
* 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, [[phab:T286506|T286506]]
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 18:43 majavah: deployment-cache-text06 varnish not starting, [[phab:T286506|T286506]], causing an outage on text traffic on deployment-prep
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]
* 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
* 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
* 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' ([[phab:T286479|T286479]])
* 14:44 majavah: fix merge conflict on deployment-puppetmaster04
* 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
* 13:19 James_F: Zuul: Add R4356th to the CI allow list
* 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [[phab:T286207|T286207]]


== 2021-07-09 ==
== 2022-04-28 ==
* 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days ([[phab:T286375|T286375]])
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days ([[phab:T286358|T286358]])
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC


== 2021-07-08 ==
== 2022-04-27 ==
* 08:21 majavah: kick stuck puppet agent on deployment-logstash04
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029


== 2021-07-07 ==
== 2022-04-26 ==
* 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance [[phab:T286298|T286298]]
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2021-07-06 ==
== 2022-04-25 ==
* 01:16 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/703219
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI


== 2021-07-05 ==
== 2022-04-20 ==
* 15:02 Amir1: deployed 703212 ([[phab:T286058|T286058]])
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service


== 2021-07-02 ==
== 2022-04-18 ==
* 17:32 brennen: gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/701068 ([[phab:T274463|T274463]])
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 09:49 James_F: Zuul: [pywikibot/core] Add deeptest to gate-and-submit section
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.


== 2021-06-30 ==
== 2022-04-14 ==
* 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 17:57 thcipriani: restart ci jenkins following upgrade
* 17:54 thcipriani: restart releases-jenkins following upgrade
* 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31


== 2021-06-29 ==
== 2022-04-12 ==
* 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # [[phab:T285811|T285811]]
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141


== 2021-06-28 ==
== 2022-04-08 ==
* 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for [[phab:T285125|T285125]]


== 2021-06-24 ==
== 2022-04-07 ==
* 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]


== 2021-06-23 ==
== 2022-04-06 ==
* 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick
* 20:03 thcipriani: rebooting phabricator
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]


== 2021-06-22 ==
== 2022-04-04 ==
* 23:03 jeena: increasing releases-jenkins executors from 2 to 4
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 21:25 jeena: Updating dev-images docker-pkg files on primary contint [[phab:T273682|T273682]]
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 09:31 hashar: Removed DISPLAY=:94 being set from Zuul for most jobs https://gerrit.wikimedia.org/r/c/integration/config/+/700857/
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul


== 2021-06-21 ==
== 2022-04-02 ==
* 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases


== 2021-06-19 ==
== 2022-03-31 ==
* 13:44 majavah: remove deployment-deploy02 [[phab:T278689|T278689]]
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
* 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host [[phab:T283013|T283013]]


== 2021-06-16 ==
== 2022-03-29 ==
* 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException [[phab:T284732|T284732]]
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771


== 2021-06-14 ==
== 2022-03-28 ==
* 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge [[phab:T236915|T236915]]
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing [[phab:T236915|T236915]]
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]
* 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli


== 2021-06-13 ==
== 2022-03-27 ==
* 18:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699567
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]
* 18:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699541


== 2021-06-11 ==
== 2022-03-26 ==
* 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
* 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497  # [[phab:T73480|T73480]]
* 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
* 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # [[phab:T100837|T100837]]
* 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for [[phab:T271011|T271011]].  Comment left on gerrit patch and task,  not reverting patch in master at this stage


== 2021-06-10 ==
== 2022-03-25 ==
* 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too [[phab:T281195|T281195]]
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency [[phab:T281195|T281195]]
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord


== 2021-06-08 ==
== 2022-03-24 ==
* 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 17:40 brennen: gitlab1001: running k6 data generator
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
* 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 [[phab:T280491|T280491]]


== 2021-06-07 ==
== 2022-03-23 ==
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # [[phab:T263617|T263617]]
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #[[phab:T263617|T263617]]
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for [[phab:T284492|T284492]]
* 11:51 hashar: zuul enqueue  --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1 # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470


== 2021-06-05 ==
== 2022-03-22 ==
* 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945


== 2021-06-04 ==
== 2022-03-21 ==
* 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]
* 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 [[phab:T284345|T284345]]
* 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [[phab:T284345|T284345]]
* 19:20 James_F: Docker: Publishing node12 CI images [[phab:T284343|T284343]]


== 2021-06-03 ==
== 2022-03-18 ==
* 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/*  # [[phab:T284125|T284125]]
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 00:24 James_F: Zuul: Add Anysite to CI allowlist
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
* 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline [[phab:T281132|T281132]]


== 2021-06-02 ==
== 2022-03-17 ==
* 19:21 urbanecm: deployment-prep: Enlarge mwlog's /srv partition to 5 GB (was 2 GB)
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 13:49 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project search/MjoLniR --change 685569,4 # Rerun sonar analysis against https://gerrit.wikimedia.org/r/c/search/MjoLniR/+/685569
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]


== 2021-05-28 ==
== 2022-03-16 ==
* 09:41 addshore: reload zuul for https://gerrit.wikimedia.org/r/697033
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 08:02 hashar: Successfully published image docker-registry.discovery.wmnet/releng/ci-bullseye:0.1.0 # [[phab:T283777|T283777]]
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep


== 2021-05-27 ==
== 2022-03-15 ==
* 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
* 18:26 brennen: gitlab: removed most existing /people groups
* 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 16:05 brennen: gitlab1001: re-running ansible and puppet per [[phab:T279545|T279545]]
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2021-05-26 ==
== 2022-03-14 ==
* 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` ([[phab:T279545|T279545]])
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2021-05-24 ==
== 2022-03-11 ==
* 06:19 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693639
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki


== 2021-05-21 ==
== 2022-03-10 ==
* 18:30 James_F: Zuul: Add phan to all extensions and skins [[phab:T283097|T283097]]
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 17:59 James_F: Docker: Publishing mediawiki-phan* images where the job passes if .phan/config.php is absent [[phab:T283097|T283097]]
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 08:30 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693361 https://gerrit.wikimedia.org/r/691323
* 20:03 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2021-05-20 ==
== 2022-03-09 ==
* 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
* 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
* 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
* 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs [[phab:T280669|T280669]]


== 2021-05-19 ==
== 2022-03-08 ==
* 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs [[phab:T280669|T280669]]
* 20:31 brennen: requiring 2fa for all users under /repos
* 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps [[phab:T279275|T279275]]
* 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users [[phab:T279552|T279552]]


== 2021-05-17 ==
== 2022-03-07 ==
* 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive [[phab:T279221|T279221]]
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2021-05-16 ==
== 2022-03-04 ==
* 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04
* 19:13 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/768068


== 2021-05-15 ==
== 2022-03-03 ==
* 21:51 James_F: Zuul: [mediawiki/tools/cli] Make mw-cli-test experimental for now [[phab:T248779|T248779]]
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 21:40 James_F: Zuul: [mediawiki/tools/cli] Add new bespoke job [[phab:T248779|T248779]]
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 21:09 James_F: Zuul: [mediawiki/extensions/MediaWikiAuth] Mark repo as archived [[phab:T282955|T282955]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 18:38 James_F: Zuul: Temporarily remove TwoColConflict from gated extensions [[phab:T234002|T234002]] [[phab:T282935|T282935]].
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 09:30 Majavah: create deployment-logstash04 to install elk7
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 09:22 Majavah: beta: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315, remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/683837 [[phab:T277990|T277990]]
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 06:23 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/691494/ [[phab:T281986|T281986]]
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2021-05-14 ==
== 2022-03-02 ==
* 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php),  [[phab:T198673|T198673]]
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, [[phab:T198673|T198673]],  https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, [[phab:T198673|T198673]], https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
* 15:27 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/767493
* 20:51 Krinkle: I broke beta  `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04
* 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
* 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
* 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job [[phab:T174018|T174018]] [[phab:T177260|T177260]] [[phab:T280901|T280901]]
* 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI [[phab:T281538|T281538]]


== 2021-05-13 ==
== 2022-02-28 ==
* 23:14 Krinkle: root@integration-agent-qemu-1001:/home/addshore# rm qmeutest/out.img (Reclaiming space to be able to make other changes)
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 22:48 longma: restarting zuul to try getting tests to run for https://gerrit.wikimedia.org/r/c/mediawiki/skins/Modern/+/684509
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin


== 2021-05-12 ==
== 2022-02-26 ==
* 19:40 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/439774 in deployment-prep puppet.  Hasn't been touched in over a year and is preventing merges.
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 19:32 James_F: Zuul: Add PageImages, PageViewInfo, and Graph to MediaWiki gated extension set [[phab:T249674|T249674]]
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 19:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/683695 in deployment-prep puppet.  Should only affect beta logstash.
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases
* 16:15 James_F: Docker: Publishing civicrm 0.2.1 for [[phab:T277500|T277500]]
* 16:00 James_F: Zuul: Add GrowthExperiments to the MediaWiki gated extension set [[phab:T247507|T247507]] [[phab:T249674|T249674]]
* 15:36 _joe_: updating helm-linter docker image


== 2021-05-11 ==
== 2022-02-24 ==
* 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set [[phab:T237538|T237538]] [[phab:T249674|T249674]]
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2021-05-10 ==
== 2022-02-23 ==
* 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default ([[phab:T271206|T271206]])
* 20:10 dancy: Updating scap in beta
* 14:04 CFisch_WMDE: Forward renamed config name for improved template search features ([[phab:T277028|T277028]])
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/  # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2021-05-07 ==
== 2022-02-22 ==
* 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue [[phab:T282018|T282018]]
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])


== 2021-05-06 ==
== 2022-02-21 ==
* 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds [[phab:T264877|T264877]]
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job [[phab:T271562|T271562]]
* 07:27 hashar: Refreshing all Jenkins jobs


== 2021-05-05 ==
== 2022-02-20 ==
* 23:52 James_F: Zuul: Ensure Parsoid's tests include the Disambiguator extension (take 2) [[phab:T271863|T271863]] [[phab:T237538|T237538]]
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
* 20:38 James_F: Docker: Publishing quibble-buster* 0.0.47 images
* 19:58 James_F: Docker: Publishing quibble-stretch* 0.0.47-s1 images, now with node
* 18:23 James_F: Docker: Publishing quibble-stretch* 0.0.47 images
* 16:54 hashar: Tag Quibble 0.0.47 @ {{Gerrit|8b200cfb0}} # [[phab:T271863|T271863]] [[phab:T199403|T199403]] [[phab:T281607|T281607]]
* 14:17 CFisch_WMDE: Disable ReferencePreviews beta mode on beta labs ([[phab:T271206|T271206]])
* 13:22 Majavah: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/684034 https://gerrit.wikimedia.org/r/c/operations/puppet/+/684088 - might cause beta to go read only for a bit - [[phab:T110115|T110115]]


== 2021-05-04 ==
== 2022-02-19 ==
* 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]


== 2021-05-03 ==
== 2022-02-18 ==
* 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [[phab:T281256|T281256]]
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 13:55 CFisch_WMDE: enable new search features for the template dialog ([[phab:T271802|T271802]])
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}


== 2021-05-02 ==
== 2022-02-17 ==
* 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref [[phab:T198673|T198673]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117
* 11:12 hashar: Bringing deployment-deploy03 back
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome


== 2021-05-01 ==
== 2022-02-16 ==
* 19:19 James_F: Zuul: Add atagar to the CI allow list
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - [[phab:T278641|T278641]]
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}


== 2021-04-30 ==
== 2022-02-15 ==
* 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for [[phab:T257301|T257301]]
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 14:15 Majavah: revert above as it's not working, [[phab:T206158|T206158]]
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 [[phab:T206158|T206158]]
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}
* 02:37 James_F: Docker: Publishing node10 images based on buster [[phab:T278203|T278203]] [[phab:T240955|T240955]]


== 2021-04-29 ==
== 2022-02-14 ==
* 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist [[phab:T281418|T281418]]
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 16:16 Amir1: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon


== 2021-04-28 ==
== 2022-02-12 ==
* 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs [[phab:T279180|T279180]]
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # [[phab:T249268|T249268]]
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
* 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip <nowiki>{</nowiki><nowiki>}</nowiki> \+  # [[phab:T249268|T249268]]
* 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.


== 2021-04-27 ==
== 2022-02-10 ==
* 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for [[phab:T277742|T277742]].
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 17:52 Majavah: delete deployment-sessionstore03 [[phab:T263617|T263617]] [[phab:T278641|T278641]]
* 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for [[phab:T281283|T281283]]
* 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/680476
* 12:44 hashar: Restarted CI Jenkins for plugins upgrade
* 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2  (with ldap plugin 1.26)
* 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
* 06:40 Majavah: installing deployment-sessionstore04 [[phab:T263617|T263617]]
* 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs [[phab:T281198|T281198]]


== 2021-04-26 ==
== 2022-02-09 ==
* 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list
* 15:22 taavi: deleted shutoff deployment-mx02


== 2021-04-24 ==
== 2022-02-08 ==
* 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03


== 2021-04-23 ==
== 2022-02-07 ==
* 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
* 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts


== 2021-04-22 ==
== 2022-02-04 ==
* 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
* 02:53 Reedy: killed a few stuck beta ci jobs
* 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
* 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 [[phab:T280874|T280874]]
* 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental [[phab:T280874|T280874]]
* 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos [[phab:T280874|T280874]]


== 2021-04-21 ==
== 2022-02-03 ==
* 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs [[phab:T280874|T280874]]
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies


== 2021-04-20 ==
== 2022-02-02 ==
* 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog ([[phab:T273425|T273425]])
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor ([[phab:T271825|T271825]])
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta


== 2021-04-19 ==
== 2022-02-01 ==
* 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental [[phab:T280491|T280491]]
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for [[phab:T280491|T280491]]
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]


== 2021-04-17 ==
== 2022-01-31 ==
* 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for [[phab:T280420|T280420]]
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]


== 2021-04-16 ==
== 2022-01-28 ==
* 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks [[phab:T234020|T234020]] [[phab:T280167|T280167]]
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`
* 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch [[phab:T278203|T278203]]
* 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs [[phab:T252432|T252432]]
* 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref [[phab:T235142|T235142]]
* 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref [[phab:T198673|T198673]]
* 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref [[phab:T198673|T198673]]
* 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
* 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
* 17:02 dancy: Updating dev-images docker-pkg files on primary contint
* 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template [[phab:T252434|T252434]]


== 2021-04-15 ==
== 2022-01-27 ==
* 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2  # [[phab:T299431|T299431]]
* 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins


== 2021-04-14 ==
== 2022-01-26 ==
* 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov [[phab:T234020|T234020]]
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2021-04-13 ==
== 2022-01-25 ==
* 17:00 halfak: failed deploy to ORES (connection to host failed)
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 16:57 halfak: deploying ores {{Gerrit|f08a3cb}}
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 [[phab:T280004|T280004]]
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 07:46 awight: enable syntax highlighting line numbering on all namespaces ([[phab:T267911|T267911]])
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2021-04-12 ==
== 2022-01-24 ==
* 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; [[phab:T279853|T279853]])
* 21:44 Reedy: unstick beta ci jobs
* 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # [[phab:T279853|T279853]]
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per [[phab:T106915|T106915]]#6279270 - [[phab:T135427|T135427]]
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2021-04-11 ==
== 2022-01-22 ==
* 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov [[phab:T234020|T234020]]
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
* 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3<nowiki>{</nowiki>5,6<nowiki>}</nowiki> [[phab:T274971|T274971]]
* 00:02 James_F: Docker: Publishing quibble-buster images [[phab:T252432|T252432]]


== 2021-04-10 ==
== 2022-01-21 ==
* 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`


== 2021-04-09 ==
== 2022-01-20 ==
* 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, [[phab:T233134|T233134]].
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 [[phab:T277750|T277750]] phan for wikibase release branches
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 09:51 Majavah: deleting deployment-jobrunner03 [[phab:T278664|T278664]]
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage [[phab:T234080|T234080]]
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2021-04-08 ==
== 2022-01-19 ==
* 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
* 18:20 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]


== 2021-04-07 ==
== 2022-01-18 ==
* 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0  with proper PHP version in the images # [[phab:T299389|T299389]]
* 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 [[phab:T278664|T278664]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2021-04-06 ==
== 2022-01-17 ==
* 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived [[phab:T277626|T277626]]
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs [[phab:T279260|T279260]]
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines [[phab:T279459|T279459]]
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
* 17:58 James_F: Zuul: [mediawiki/services/function-<nowiki>{</nowiki>orchestr,evalu<nowiki>}</nowiki>ator] Publish images


== 2021-04-05 ==
== 2022-01-14 ==
* 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 17:45 brennen: halting gitlab-test for resize
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta


== 2021-04-02 ==
== 2022-01-13 ==
* 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2021-04-01 ==
== 2022-01-12 ==
* 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
* 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
* 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
* 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298


== 2021-03-30 ==
== 2022-01-11 ==
* 18:05 Majavah: remove <nowiki>{</nowiki>trysty,precise<nowiki>}</nowiki>-deployment-prep repos from deployment-deploy01 aptly
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
* 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those [[phab:T278689|T278689]]
* 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
* 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
* 13:35 Majavah: create and install deployment-deploy03 [[phab:T278689|T278689]]
* 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 07:26 Majavah: shutoff deployment-mediawiki-09 [[phab:T278664|T278664]]
* 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
* 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11


== 2021-03-29 ==
== 2022-01-10 ==
* 15:37 Majavah: hard reboot deployment-sessionstore03 [[phab:T263617|T263617]]
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) [[phab:T263617|T263617]]
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]
* 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 ([[phab:T278664|T278664]]), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
* 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
* 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
* 09:29 Urbanecm: Manually run puppet on mediawiki10
* 09:28 Urbanecm: Re-enable puppet on mediawiki10
* 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
* 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - [[phab:T278664|T278664]]
* 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, [[phab:T278664|T278664]]


== 2021-03-26 ==
== 2022-01-05 ==
* 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 ([[phab:T218729|T218729]])
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2021-03-25 ==
== 2022-01-04 ==
* 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383


== 2021-03-24 ==
== 2022-01-03 ==
* 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm [[phab:T238707|T238707]]
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001.  Restarted CI Jenkins.
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


== 2021-03-23 ==
* 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per [[phab:T274015|T274015]]
* 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job [[phab:T207226|T207226]]
* 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
* 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade [[phab:T252434|T252434]]
* 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template [[phab:T252434|T252434]]
* 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use
== 2021-03-22 ==
* 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
* 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
* 12:07 Majavah: delete deployment-restbase[01-02], [[phab:T250574|T250574]]
* 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T276624|T276624]])
* 11:33 dcaro: Created subzone beta.wmcloud.org ([[phab:T276624|T276624]])
== 2021-03-19 ==
* 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
* 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
* 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, [[phab:T218729|T218729]]
* 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
* 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down [[phab:T250574|T250574]]
* 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for [[phab:T250574|T250574]]
== 2021-03-18 ==
* 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw ([[phab:T277109|T277109]])
* 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
* 18:27 brennen: updating gitlab-test to 13.9.4-ce
* 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 [[phab:T277750|T277750]]
* 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
* 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # [[phab:T269354|T269354]]
* 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 [[phab:T277750|T277750]] (apitests php versions)
* 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 [[phab:T277060|T277060]]
* 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11  # [[phab:T269354|T269354]]
* 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for [[phab:T276179|T276179]]
* 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
* 08:44 Majavah: delete now unused deployment-ircd [[phab:T277081|T277081]]
* 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now
== 2021-03-17 ==
* 20:30 hashar: Reloaded Zuul for {{Gerrit|I2368478e4c4ab8752581f55a7c5ab493fafdeb41}}
* 15:37 Majavah: shutdown deployment-restbase01 for [[phab:T250574|T250574]]
* 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # [[phab:T250574|T250574]]
* 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
* 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI
== 2021-03-16 ==
* 21:37 longma: Updating dev-images docker-pkg files on primary contint
* 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
* 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
* 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
* 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
* 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1  # [[phab:T277527|T277527]]
== 2021-03-15 ==
* 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
* 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118
== 2021-03-13 ==
* 17:10 twentyafterfour: restart apache on gerrit1001
== 2021-03-12 ==
* 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
* 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
* 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
* 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
* 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ [[phab:T276428|T276428]]
== 2021-03-11 ==
* 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job([[phab:T277236|T277236]])
* 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 [[phab:T276419|T276419]]
* 16:49 Majavah: delete deployment-etcd-01 [[phab:T276462|T276462]]
* 13:51 Majavah: shut down deployment-db06, now unused [[phab:T277070|T277070]]
* 13:48 Majavah: set deployment-db07 as r/w [[phab:T277070|T277070]]
* 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 [[phab:T277070|T277070]]
* 13:41 Majavah: stop slave on deployment-db06 [[phab:T276968|T276968]]
* 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 [[phab:T277070|T277070]]
* 13:34 Majavah: stop and reset slave on deployment-db07 [[phab:T277070|T277070]]
* 13:32 Majavah: set deployment-db06 as read only [[phab:T277070|T277070]]
* 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/
== 2021-03-10 ==
* 19:36 Majavah: shutdown deployment-ircd [[phab:T277081|T277081]]
* 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 [[phab:T277081|T277081]]
* 18:05 Majavah: create deployment-ircd02 for [[phab:T277081|T277081]]
* 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
* 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
* 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
* 17:09 Majavah: set beta cluster mediawiki as read write on mw config ([[phab:T276968|T276968]])
* 17:03 Majavah: make deployment-db06 read-write [[phab:T276968|T276968]]
* 16:50 Majavah: `reset slave;` on new master deployment-db06 [[phab:T276968|T276968]]
* 16:49 Majavah: add deployment-db07 as a replica of db06 for [[phab:T276968|T276968]]
* 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # [[phab:T276968|T276968]]
* 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; ([[phab:T276968|T276968]])
* 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . ([[phab:T276968|T276968]])
* 15:57 Majavah: set deployment-db06 as readonly from mysql side [[phab:T276968|T276968]]
* 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` ([[phab:T276968|T276968]])
* 15:54 Urbanecm: Start mariadb on db08 ([[phab:T276968|T276968]])
* 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 ([[phab:T276968|T276968]])
* 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
* 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
* 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade ([[phab:T276968|T276968]])
* 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) ([[phab:T276968|T276968]])
== 2021-03-09 ==
* 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
* 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user ([[phab:T276968|T276968]])
* 20:33 Majavah: install mariadb on deployment-db08 [[phab:T276968|T276968]]
* 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db ([[phab:T276968|T276968]])
* 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica ([[phab:T276968|T276968]])
* 19:50 marxarelli: restoring database dump on deployment-db07 ([[phab:T276968|T276968]])
* 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` ([[phab:T276968|T276968]])
* 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 [[phab:T276968|T276968]]
* 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 ([[phab:T276968|T276968]])
* 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster [[phab:T276968|T276968]]
* 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump ([[phab:T276968|T276968]])
* 18:09 Majavah: set deployment-db05 to read-only to avoid issues with [[phab:T276968|T276968]]
* 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances ([[phab:T276968|T276968]])
* 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
* 17:22 marxarelli: restarting deployment-db05 via horizon
* 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
* 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
* 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
* 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
* 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
* 13:32 arturo: hard-reboot deployment-db05 because issues related to [[phab:T276922|T276922]]
* 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other
== 2021-03-08 ==
* 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
* 08:57 hashar: Nuked castor cache for labs/striker # [[phab:T276605|T276605]]
== 2021-03-07 ==
* 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
* 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
* 17:18 Majavah: shutdown deployment-memc[04-05] [[phab:T276707|T276707]]
* 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 [[phab:T276707|T276707]]
* 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 [[phab:T276707|T276707]]
* 15:44 Majavah: create deployment-memc10 on Buster [[phab:T276707|T276707]], beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
* 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, [[phab:T276707|T276707]] [[phab:T250585|T250585]]
* 13:14 Majavah: create deployment-memc09 on Buster [[phab:T276707|T276707]]
== 2021-03-06 ==
* 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
* 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 [[phab:T276654|T276654]]
* 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix [[phab:T276652|T276652]]
== 2021-03-05 ==
* 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
* 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week [[phab:T276462|T276462]]
* 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref [[phab:T276462|T276462]]
* 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref [[phab:T276462|T276462]]
* 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate [[phab:T276462|T276462]]
* 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
* 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; [[phab:T276521|T276521]] and possibly others
* 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair [[phab:T241481|T241481]], directory is different on deployment-logstash2
== 2021-03-04 ==
* 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image.  That brings in tox installed with python3 instead of python2 # [[phab:T276384|T276384]]
* 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref [[phab:T276102|T276102]] and it being decomissioned in [[phab:T242855|T242855]]
* 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for [[phab:T276419|T276419]], I believe everything has been moved to deployment-mwlog01
* 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for [[phab:T276419|T276419]]
* 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
* 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
* 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref [[phab:T276419|T276419]]
* 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
* 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # [[phab:T276422|T276422]] [[phab:T180301|T180301]]
* 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
* 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch
== 2021-03-03 ==
* 20:30 legoktm: added Majavah as projectadmin in deployment-prep (Beta Cluster)
* 19:59 James_F: Zuul: [mediawiki/services/function-schemata] Revert "Use bespoke pipeline jobs"
* 16:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Run phan with Echo
* 11:40 addshore: reload zuul for https://gerrit.wikimedia.org/r/667623
* 10:33 hashar: REplaced java-codehealth-patch job in favor of running sonar:sonar inline in all the java jobs.  Thanks gehel!  # [[phab:T264873|T264873]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/666869/
== 2021-03-02 ==
* 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
* 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).
== 2021-03-01 ==
* 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
* 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.
== 2021-02-27 ==
* 22:03 Reedy: re-armed beta keyholder... I think...
== 2021-02-26 ==
* 19:47 James_F: Zuul: [mediawiki/services/geoshapes] Add typescript service CI [[phab:T274380|T274380]]
* 01:11 legoktm: update credentials in https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ for new ci-build user ([[phab:T275559|T275559]])
== 2021-02-24 ==
* 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
* 22:16 legoktm: rebuilding Rust docker images
== 2021-02-23 ==
* 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
* 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)
== 2021-02-20 ==
* 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
* 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI
== 2021-02-19 ==
* 13:51 hashar: Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897  did not get merged
* 13:49 hashar: Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/651791/
== 2021-02-18 ==
* 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
* 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata
== 2021-02-17 ==
* 20:05 hashar: Updating all Jenkins jobs for https://gerrit.wikimedia.org/r/664897 # [[phab:T275049|T275049]]
* 17:59 hashar: Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/664680  # [[phab:T275049|T275049]]
* 03:26 James_F: Zuul: [mediawiki/core] PHP 8.0 version of composertest job to experimental
== 2021-02-16 ==
* 20:33 brennen: updating gitlab-test.wmcloud.org to 13.8.4-ce.0
* 19:58 hashar: Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # [[phab:T273822|T273822]]
* 17:02 greg-g: doing the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update dance
* 04:18 James_F: Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site
* 04:00 James_F: Zuul: Add Tim Abdullin from S&F to CI allow list
== 2021-02-15 ==
* 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # [[phab:T209953|T209953]]
== 2021-02-14 ==
* 21:31 James_F: Zuul: Add 'check php' support for library repos
* 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive [[phab:T274738|T274738]]
== 2021-02-13 ==
* 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library
== 2021-02-12 ==
* 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
* 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2021-02-11 ==
* 21:44 Krinkle: Logstash in beta is not receiving any events [[phab:T274593|T274593]]
* 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
* 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension [[phab:T274069|T274069]]
* 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
* 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
* 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339
== 2021-02-10 ==
* 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
* 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for [[phab:T274306|T274306]]
* 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
* 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension
== 2021-02-09 ==
* 22:27 brennen: once more, with feeling: attempting docker-pkg run for dev-images again
* 22:09 brennen: attempting to run docker-pkg manually for dev-images on contint2001
* 21:08 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:635361]] and [[gerrit:632173]]
* 20:10 James_F: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld
* 19:31 James_F: Docker: Building and publishing tox-buster &c. with tox 3.21.4 [[phab:T274232|T274232]]
* 04:43 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref [[phab:T273247|T273247]]
* 04:42 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify
== 2021-02-08 ==
* 21:40 brennen: enabled administrative approval for new account signups on gitlab-test
* 21:40 brennen: upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0
* 17:22 hashar: Built image docker-registry.discovery.wmnet/releng/quibble-buster-php72:0.0.45-s4
* 17:17 hashar: Building some docker images on contint.wikimedia.org
* 16:51 hashar: Now really reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 16:43 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 14:55 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658946  # [[phab:T272863|T272863]]
* 09:06 _joe_: cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 in deployment-prep
== 2021-02-06 ==
* 21:13 Reedy: unstuck beta jobs
== 2021-02-04 ==
* 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI [[phab:T273247|T273247]]
== 2021-02-03 ==
* 21:38 James_F: Zuul: Archive VirtualKeyboard extension [[phab:T273801|T273801]]
* 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
* 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
* 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL [[phab:T273295|T273295]]
== 2021-02-02 ==
* 19:43 hashar: Pruning dangling Docker images on contint2001
* 19:39 hasharDinner: Pruning dangling Docker images on contint1001
* 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [[phab:T61245|T61245]]
* 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
* 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # [[phab:T273599|T273599]]
== 2021-02-01 ==
* 16:04 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta.
* 10:55 Urbanecm: deployment-prep: Create beta votewiki ([[phab:T272608|T272608]])
* 09:15 hashar: devtools: switched gerrit-prod-1001 to local puppetmaster
== 2021-01-29 ==
* 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes
== 2021-01-28 ==
* 22:15 marxarelli: deleting unused integration-registry-1003 instance
* 17:46 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
* 17:42 marxarelli: updating jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
== 2021-01-27 ==
* 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02
== 2021-01-26 ==
* 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
* 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
* 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
* 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
* 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355
== 2021-01-25 ==
* 15:57 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2021-01-22 ==
* 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout
== 2021-01-19 ==
* 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI [[phab:T271761|T271761]]
== 2021-01-18 ==
* 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker [[phab:T272320|T272320]]
== 2021-01-17 ==
* 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job [[phab:T248925|T248925]]
== 2021-01-16 ==
* 23:24 James_F: Docker: Building cascade of new php-ast image [[phab:T271428|T271428]]
== 2021-01-14 ==
* 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job
== 2021-01-13 ==
* 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents
== 2021-01-12 ==
* 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
* 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
* 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
* 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
* 19:33 James_F: Zuul: Revert Parsoid integration job injection.
* 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
* 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension [[phab:T237538|T237538]]
* 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template
== 2021-01-11 ==
* 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # [[phab:T271683|T271683]]
== 2021-01-09 ==
* 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]
* 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]
== 2021-01-08 ==
* 02:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/654964
== 2021-01-07 ==
* 18:34 James_F: Zuul: Provide experimental PHP 7.2-on-buster jobs [[phab:T252434|T252434]]
* 18:32 James_F: Zuul: [labs/tools/massmailer] Add gate-and-submit-l10n jobs [[phab:T271426|T271426]]
* 16:01 hashar: Tag Quibble 0.0.46 @ {{Gerrit|df9e75329ab}} # [[phab:T225218|T225218]] [[phab:T266441|T266441]] [[phab:T263500|T263500]]
== 2021-01-06 ==
* 19:36 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/620729 in beta puppet.
* 18:33 dpifke: Doing a test run in beta of the scap commands I'm going to run in today's backport window to roll out profiler changes.
== 2021-01-05 ==
* 23:55 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/651267 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/621095/3 in beta.  Should be a no-op (removes hopefully unused code).
== 2021-01-04 ==
* 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job [[phab:T269719|T269719]]
* 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled  for now due to [[phab:T271122|T271122]]
* 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot  # [[phab:T271122|T271122]]
* 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts
== 2021-01-02 ==
* 19:13 James_F: Zuul: Add CI for the Mirage skin [[phab:T270979|T270979]]
== 2021-01-01 ==
* 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master [[phab:T270976|T270976]]
* 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting [[phab:T251967|T251967]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 22:11, 9 August 2022

2022-08-09

  • 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
  • 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
  • 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
  • 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for T293924
  • 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for T293851

2022-08-08

  • 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
  • 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for T309872
  • 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for T314630
  • 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166

2022-08-05

  • 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for T296967
  • 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
  • 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for T314630
  • 14:48 James_F: Zuul: Add WelpThatWorked to allow list
  • 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin

2022-08-04

2022-08-03

  • 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
  • 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
  • 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
  • 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
  • 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for T309872
  • 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
  • 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too

2022-08-02

2022-08-01

  • 23:16 James_F: Zuul: [design/codex] Switch to node16
  • 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
  • 22:53 TheresNoTime: remove stuck beta deployment jobs
  • 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
  • 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
  • 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
  • 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
  • 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
  • 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for T314306
  • 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
  • 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
  • 09:14 TheresNoTime: clearing stuck beta CI jobs

2022-07-29

  • 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for T313919
  • 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes

2022-07-28

2022-07-27

  • 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch T300463
  • 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for T274965
  • 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
  • 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
  • 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref T143162, T193565

2022-07-25

2022-07-23

2022-07-21

  • 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster

2022-07-20

  • 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
  • 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline

2022-07-19

  • 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
  • 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # T310905

2022-07-18

2022-07-17

2022-07-16

  • 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt

2022-07-15

2022-07-14

  • 18:50 James_F: Docker: Building node16 images for CI for T313075
  • 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for T203215
  • 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
  • 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for T268085
  • 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI

2022-07-13

2022-07-12

  • 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki (T312865)

2022-07-11

2022-07-10

  • 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet

2022-07-09

  • 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
  • 17:25 ori: Cherry-picked Ief73cc553 (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
  • 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
  • 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
  • 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump # ref T312689

2022-07-07

  • 22:42 TheresNoTime: clear stuck beta deployment jobs (again), T72597
  • 21:10 TheresNoTime: clear stuck beta deployment jobs, T72597
  • 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
  • 12:22 hashar: integration: rebooting `integration-agent-docker-1039` T312534

2022-07-05

2022-06-30

  • 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at T72597
  • 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
  • 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053

2022-06-29

  • 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
  • 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')

2022-06-28

2022-06-27

2022-06-24

  • 20:52 taavi: added `denisse` as a member

2022-06-23

2022-06-22

  • 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
  • 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
  • 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
  • 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
  • 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
  • 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
  • 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180

2022-06-21

  • 22:01 brennen: gitlab-runners: re-registering all shared runners
  • 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster

2022-06-20

  • 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
  • 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
  • 14:07 ori: restarted acme-chief on deployment-acme-chief03

2022-06-17

  • 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
  • 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack

2022-06-16

  • 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
  • 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
  • 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]

2022-06-15

  • 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
  • 16:31 hashar: integration-agent-docker-1035: docker image prune
  • 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster

2022-06-14

  • 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
  • 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
  • 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
  • 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production

2022-06-13

  • 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
  • 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore

2022-06-12

2022-06-10

  • 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
  • 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
  • 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
  • 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check

2022-06-09

  • 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
  • 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration

2022-06-08

  • 17:14 hashar: Reloaded Zuul for I393422
  • 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for T237033
  • 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134

2022-06-06

  • 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711

2022-06-02

  • 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
  • 11:26 hashar: Restarting Jenkins on contint2001
  • 11:19 hashar: Restarting Jenkins on releases1002

2022-05-31

  • 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
  • 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
  • 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
  • 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster

2022-05-30

  • 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
  • 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142

2022-05-28

  • 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437

2022-05-27

  • 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
  • 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
  • 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
  • 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
  • 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
  • 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
  • 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
  • 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep

2022-05-26

  • 18:33 dancy: Updated Jenkins beta-* job configs
  • 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 2c7b5825
  • 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch

2022-05-25

  • 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] T309230
  • 15:46 dancy: Restarted apache2 on gerrit1001

2022-05-24

2022-05-23

  • 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
  • 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
  • 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
  • 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
  • 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
  • 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
  • 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982

2022-05-22

2022-05-21

2022-05-20

2022-05-19

2022-05-18

  • 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
  • 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
  • 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
  • 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071

2022-05-17

  • 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558

2022-05-16

2022-05-14

  • 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
  • 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
  • 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request

2022-05-13

2022-05-12

  • 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
  • 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
  • 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
  • 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
  • 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)

2022-05-11

  • 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
  • 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster

2022-05-10

2022-05-09

2022-05-08

  • 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057

2022-05-06

  • 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071

2022-05-05

2022-05-04

2022-05-03

2022-05-02

2022-04-29

2022-04-28

2022-04-27

2022-04-26

  • 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
  • 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
  • 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
  • 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
  • 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
  • 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204

2022-04-25

2022-04-20

  • 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

2022-04-18

  • 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
  • 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
  • 16:48 brennen: phabricator: adding self to acl*blog-admins
  • 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.

2022-04-14

2022-04-12

2022-04-08

2022-04-07

  • 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
  • 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461

2022-04-06

  • 20:03 thcipriani: rebooting phabricator
  • 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596

2022-04-04

2022-04-02

2022-03-31

2022-03-29

  • 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
  • 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
  • 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771

2022-03-28

2022-03-27

  • 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736

2022-03-26

  • 02:37 Reedy: beta-update-databases-eqiad is back to @hourly

2022-03-25

  • 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
  • 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
  • 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
  • 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
  • 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord

2022-03-24

  • 20:00 hashar: reloading Zuul for Id844e1 # T299320
  • 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
  • 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
  • 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck

2022-03-23

  • 17:35 brennen: restarting phabricator for T304540, brief downtime expected
  • 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster

2022-03-22

2022-03-21

  • 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
  • 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203

2022-03-18

  • 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
  • 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
  • 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)

2022-03-17

2022-03-16

2022-03-15

2022-03-14

  • 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
  • 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
  • 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
  • 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
  • 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
  • 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
  • 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
  • 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
  • 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
  • 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
  • 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
  • 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079

2022-03-11

  • 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki

2022-03-10

2022-03-09

2022-03-08

  • 20:31 brennen: requiring 2fa for all users under /repos

2022-03-07

  • 10:53 zabe: restarted apache on deployment-mediawiki11 # T302699

2022-03-04

2022-03-03

2022-03-02

  • 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
  • 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
  • 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
  • 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04

2022-02-28

  • 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
  • 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
  • 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin

2022-02-26

  • 20:05 zabe: apply T302658 on deployment-prep centralauth databases
  • 13:24 zabe: apply T302660 on deployment-prep centralauth databases
  • 13:19 zabe: apply T302659 on deployment-prep centralauth databases

2022-02-24

  • 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
  • 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
  • 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
  • 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
  • 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
  • 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
  • 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
  • 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1

2022-02-23

  • 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
  • 20:10 dancy: Updating scap in beta
  • 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
  • 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
  • 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
  • 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774

2022-02-22

  • 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
  • 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

  • 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
  • 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
  • 07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

  • 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

  • 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
  • 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
  • 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

2022-02-17

  • 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
  • 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
  • 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
  • 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
  • 11:12 hashar: Bringing deployment-deploy03 back
  • 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

  • 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
  • 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
  • 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
  • 10:09 hashar: Reloading Zuul for I997fee
  • 09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

  • 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
  • 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
  • 16:24 hashar: Restarting CI Jenkins for plugins updates
  • 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
  • 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
  • 15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

  • 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
  • 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
  • 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
  • 14:26 jnuche: Jenkins upgrade complete T301361
  • 13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

  • 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

2022-02-09

  • 15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

  • 17:34 taavi: remove scap from deployment-kafka-main/jumbo
  • 16:23 taavi: hard reboot misbehaving deployment-echostore01
  • 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

2022-02-04

2022-02-03

  • 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
  • 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
  • 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

  • 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
  • 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

2022-01-31

  • 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
  • 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

  • 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
  • 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

  • 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
  • 19:34 Amir1: Reloading Zuul to deploy 757464
  • 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
  • 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
  • 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
  • 00:24 thcipriani: restarting jenkins

2022-01-26

  • 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
  • 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
  • 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
  • 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
  • 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

  • 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
  • 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
  • 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
  • 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
  • 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
  • 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
  • 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
  • 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
  • 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
  • 09:22 Reedy: unblocked beta again
  • 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

  • 21:44 Reedy: unstick beta ci jobs
  • 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
  • 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

  • 13:40 taavi: apply T299827 on deployment-prep centralauth database
  • 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

  • 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
  • 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

  • 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
  • 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
  • 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
  • 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
  • 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
  • 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
  • 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
  • 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
  • 12:10 hashar: contint2001 : docker container prune && docker image prune
  • 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
  • 12:04 hashar: contint1001 `docker image prune`
  • 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

2022-01-18

  • 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
  • 18:01 taavi: added ryankemper as a member of the deployment-prep project
  • 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
  • 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
  • 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
  • 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

  • 16:28 hashar: Building Quibble 1.3.0 Docker images
  • 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
  • 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

  • 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
  • 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
  • 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
  • 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

  • 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
  • 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
  • 16:27 dancy: testing scap prep auto on deployment-deploy03
  • 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
  • 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
  • 10:52 hashar: Restarting Jenkins CI for plugins update
  • 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
  • 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

2022-01-11

  • 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

  • 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
  • 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

2022-01-04

2022-01-03

  • 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
  • 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


Archives