You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(dpifke: Testing changes 607369 and 607370 in beta, should only affect deployment-webperf12.)
imported>Stashbot
(RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350)
(533 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== 2020-06-25 ==
== 2022-08-16 ==
* 22:02 dpifke: Testing changes 607369 and 607370 in beta, should only affect deployment-webperf12.
* 20:51 RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350
* 15:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/607818
* 20:30 hashar: Repooled integration-agent-docker-1028 , it was mysteriously unreachable [[phab:T315372|T315372]]
* 12:52 hashar: for volans: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 603434,15
* 19:18 Krinkle: mediawiki/extensions/EventLogging$ git remote-wildcard-br-d 'wmf/1.35*' 'wmf/1.36*'  'wmf/1.37*' 'wmf/1.38*'
* 19:17 Krinkle: mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref [[phab:T303828|T303828]]
* 19:16 TheresNoTime: manually running `/usr/local/bin/wmf-beta-update-databases.py` on `deployment-deploy03`
* 17:16 TheresNoTime: soft-rebooting deployment-mediawiki12


== 2020-06-24 ==
== 2022-08-12 ==
* 18:56 halfak: deploying ORES {{Gerrit|1b87365}}
* 17:47 dancy: Restarting zuul
* 16:32 dpifke: Cherry-picking new version of change 603550 into beta puppet for testing, and pointing beta XHGui at new instance.
* 17:42 dancy: Restarting Jenkins in an attempt to get CI jobs running again
* 08:40 James_F: Zuul: Add proc to CI allowlist
* 00:54 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service . Certificate was close to expiry


== 2020-06-23 ==
== 2022-08-11 ==
* 17:15 bstorm_: to fix puppet on several hosts, setting profile::java::hardened_tls: false in project puppet on horizon
* 21:11 mutante: restarted phd service on phab2001
* 17:09 bstorm_: restarted postgresql on deployment-puppetdb03
* 19:12 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16
* 17:05 bstorm_: restarted puppetdb.service on deployment-puppetdb03
* 12:26 jnuche: Reenabled CI beta sync jobs after cluster incident
* 11:48 jnuche: Temporarily disabled CI beta sync jobs until issue in cluster is resolved
* 10:25 zabe: take deployment-prep out of read-only mode


== 2020-06-22 ==
== 2022-08-10 ==
* 13:03 hashar: Deleted integration-cumin-01 , replaced by integration-cumin
* 11:36 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/822052
* 12:40 James_F: Docker: Publishing images using npm 6.14.5
* 12:38 hashar: integration: created and provisioned a new cumin master based on Buster:  integration-cumin.integration.eqiad.wmflabs
* 12:38 hashar: deployment-prep: created and provisioned a new cumin master based on Buster:  deployment-cumin.deployment-prep.eqiad.wmflabs


== 2020-06-19 ==
== 2022-08-09 ==
* 10:31 hashar: Triggering documentation generation for transferpy ([[phab:T253219|T253219]]): contint2001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/transferpy --change 602719,9
* 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
* 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
* 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
* 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for [[phab:T293924|T293924]]
* 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for [[phab:T293851|T293851]]


== 2020-06-18 ==
== 2022-08-08 ==
* 18:28 hashar: integration-castor03: remove labstore::traffic_shaping::egress: 100mbps  in horizon. It is now applied project wide via puppet.git # [[phab:T232644|T232644]] [[phab:T255371|T255371]]
* 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
* 17:39 James_F: Zuul: [mediawiki/services/push-notifications] Enable new node codehealth job [[phab:T240989|T240989]] [[phab:T251441|T251441]]
* 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for [[phab:T309872|T309872]]
* 16:56 hnowlan: disabled non-docker cpjobqueue instance in deployment-prep
* 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for [[phab:T314630|T314630]]
* 16:46 James_F: Docker: Publishing java8-sonar-scanner:0.6.1 for [[phab:T240989|T240989]]
* 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166


== 2020-06-17 ==
== 2022-08-05 ==
* 19:46 brennen: Updating dev-images docker-pkg files on primary contint for jobrunner changes ([[phab:T246935|T246935]])
* 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for [[phab:T296967|T296967]]
* 15:22 hashar: Reloading zuul for gerrit-test.wikimedia.org  {{Gerrit|Iaaa7e8797d779578a6a42f735bb7c809e29f415a}}
* 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
* 15:05 James_F: Private config for MediaModeration extension added on Beta Cluster for [[phab:T247943|T247943]]
* 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for [[phab:T314630|T314630]]
* 14:53 hashar: Reloading Zuul for {{Gerrit|I5f10c7438a1514d705d1b0b8569c325305982366}}
* 14:48 James_F: Zuul: Add WelpThatWorked to allow list
* 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin


== 2020-06-16 ==
== 2022-08-04 ==
* 21:03 Platonides: testing a [[phab:T230245|T230245]] fix
* 15:21 dancy: Deleting beta-mediawiki-config-update-eqiad job
* 20:29 Platonides: done!
* 15:16 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405
* 20:14 Platonides: livehacking MWExceptionHandler on deploy01
* 10:01 TheresNoTime: clearing out stuck beta deployment jobs [[phab:T314378|T314378]] [[phab:T72597|T72597]]
* 19:07 Krinkle: Done fiddling with deploy01
* 18:48 hashar: Reloading Zuul for {{Gerrit|Ibb8b8655f1837ced28783ef44f2b7b1e4a15e9ea}}
* 17:25 Krinkle: Fiddling on deploy01 with Task 230245
* 10:23 hashar__: Reloading Zuul to deploy {{Gerrit|I31d8c714e2b5c76d0cd823d22fac2f7bbbf583e5}} and {{Gerrit|I95ea11a5224fc52cde4b03e0adf4673d683332f8}}
* 02:55 Krinkle: It appears logstash-beta.wmflabs.org is down/unresponsive


== 2020-06-15 ==
== 2022-08-03 ==
* 10:32 James_F: Zuul: Rename CI allow list [[phab:T255378|T255378]]
* 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
* 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
* 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
* 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
* 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for [[phab:T309872|T309872]]
* 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
* 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too


== 2020-06-14 ==
== 2022-08-02 ==
* 19:22 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/605417
* 19:26 James_F: Zuul: [design/codex] Switch coverage job back to -direct
* 15:23 dancy: Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028)
* 15:22 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028
* 07:55 TheresNoTime: cleared stuck beta deployment jobs [[phab:T72597|T72597]]


== 2020-06-13 ==
== 2022-08-01 ==
* 21:03 Krinkle: Done fiddling
* 23:16 James_F: Zuul: [design/codex] Switch to node16
* 20:43 Krinkle: Fiddling on deployment-deploy01 to investigate Task 230245
* 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
* 22:53 TheresNoTime: remove stuck beta deployment jobs
* 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
* 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
* 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
* 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
* 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
* 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for [[phab:T314306|T314306]]
* 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
* 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
* 09:14 TheresNoTime: clearing stuck beta CI jobs


== 2020-06-12 ==
== 2022-07-29 ==
* 23:01 Krinkle: Done fiddling on deployment-deploy01
* 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for [[phab:T313919|T313919]]
* 19:04 Krinkle: Fiddling on deployment-deploy01 to investigate [[phab:T230245|T230245]]
* 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes
* 16:47 addshore: deployed a bunch of phpmetric jobs to jenkins
* 16:23 James_F: Zuul: Factor out generic-node10-rundoc-docker into node10-rundoc-docker template
* 15:57 James_F: Docker: Publishing new mediawiki-phan helper image version
* 15:13 James_F: Zuul: Update trusted user list per zeljkof
* 08:15 awight: Granted dewiki-beta sysop to Adamw via commandline
* 07:57 awight: Update QuickSurveys config


== 2020-06-11 ==
== 2022-07-28 ==
* 23:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604892
* 17:54 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/818189/
* 23:39 Reedy: deployed quibble-donationinterface-REL1_31-php70-docker with https://gerrit.wikimedia.org/r/604892
* 23:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604719 [[phab:T255096|T255096]]
* 14:44 Reedy: rm -rf doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-libs-PasswordBlacklist [[phab:T254799|T254799]]
* 14:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604707
* 13:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604706 [[phab:T254799|T254799]]
* 10:37 Urbanecm: Run `update page set page_content_model="json" where page_content_model = "CollaborationListContent" OR page_content_model = "CollaborationHubContent";` at beta enwiki ([[phab:T255107|T255107]])
* 02:36 dpifke: Cherry-picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/603550/ and https://gerrit.wikimedia.org/r/#/c/labs/private/+/604498/ into deployment-prep puppet to beta-test new XHGui ([[phab:T180761|T180761]]).


== 2020-06-10 ==
== 2022-07-27 ==
* 15:18 James_F: Zuul: [mediawiki/extensions/QuickSurveys] Stop running ruby jobs [[phab:T177261|T177261]]
* 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch [[phab:T300463|T300463]]
* 14:37 James_F: Docker: Publishing php-ast:1.0.6 and related images [[phab:T255004|T255004]]
* 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for [[phab:T274965|T274965]]
* 06:23 James_F: Docker: Publishing mediawiki-phan:0.6.0
* 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
* 06:08 James_F: Zuul: Drop extension-seccheck template, unused
* 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
* 06:06 James_F: Zuul: [mediawiki/extensions/Petition] Replace seccheck with phan
* 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref [[phab:T143162|T143162]], [[phab:T193565|T193565]]
* 06:05 James_F: Zuul: [mediawiki/extensions/TEI] Replace seccheck with phan
* 06:01 James_F: Zuul: [mediawiki/extensions/Sudo] Replace seccheck with phan
* 05:59 James_F: Zuul: [mediawiki/extensions/ChessBrowser] Replace seccheck with phan


== 2020-06-09 ==
== 2022-07-25 ==
* 17:16 Krinkle: Messing with deploy01 in Beta to debug [[phab:T230245|T230245]]
* 22:16 dduvall: re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 15:56 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604058
* 21:25 dduvall: disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 14:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/604018


== 2020-06-08 ==
== 2022-07-23 ==
* 20:47 James_F: Zuul: [mediawiki/extensions/MassEditRegex] Replace seccheck with phan
* 17:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/816251
* 20:46 James_F: Zuul: [mediawiki/extensions/DisableAccount] Replace seccheck with phan
* 20:42 James_F: Zuul: [mediawiki/extensions/RelatedSites] Replace seccheck with phan
* 20:33 James_F: Zuul: [mediawiki/extensions/PageNotice] Replace seccheck with phan
* 18:38 hauskatze: Rebooting deployment-logstash0[2,3]
* 18:29 James_F: Zuul: [mediawiki/extensions/AdminLinks] Add phan
* 18:27 hauskatze: Deployment-prep: Rebooting deployment-logstash2, looks hung and cannot be reached
* 17:55 James_F: Docker: Publishing operations-puppet image, now with upgraded shellcheck [[phab:T254480|T254480]]
* 15:26 James_F: Zuul: Add Adham Khatean to CI allow list


== 2020-06-06 ==
== 2022-07-21 ==
* 15:19 Amir1: created deployment-mailman01 in beta cluster for testing upgrade to mailmain 3.1 ([[phab:T52864|T52864]] and [[phab:T130554|T130554]])
* 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster


== 2020-06-05 ==
== 2022-07-20 ==
* 15:43 hashar: Building image docker-registry.discovery.wmnet/releng/operations-puppet:0.7.2 https://gerrit.wikimedia.org/r/#/c/integration/config/+/602710/
* 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
* 07:45 hashar: jenkins: upgrading ansicolor plugin 0.6.3 > 0.7.0  # [[phab:T254408|T254408]]
* 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline


== 2020-06-04 ==
== 2022-07-19 ==
* 22:53 James_F: Docker: Publishing Quibble 44 images
* 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
* 20:31 James_F: Zuul: [mediawiki] Drop seccheck experimental, done in phan proper
* 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # [[phab:T310905|T310905]]
* 17:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/602435
* 16:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/602424 https://gerrit.wikimedia.org/r/602425
* 11:21 hashar: Built docker-registry.discovery.wmnet/releng/doxygen:0.8.0 # [[phab:T253793|T253793]]


== 2020-06-03 ==
== 2022-07-18 ==
* 19:47 James_F: Zuul: [mediawiki/extensions/MachineVision] Add codehealth.
* 19:43 dancy: Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster
* 18:57 hauskatze: GitHub: Mirror deleted: mediawiki-extensions-AzharAuth {{!}} refs. [[phab:T198601|T198601]]
* 13:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # [[phab:T311752|T311752]]
* 16:20 James_F: Docker: Publishing docker-registry.discovery.wmnet/releng/quibble-buster-php72
* 10:40 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745
* 16:18 brennen: Updating dev-images docker-pkg files on primary contint
* 09:58 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included
* 09:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/814728


== 2020-06-02 ==
== 2022-07-17 ==
* 21:16 Reedy: ran `mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikimedia apiportalwiki api.wikimedia.beta.wmflabs.org` [[phab:T254185|T254185]]
* 13:00 taavi: reloading zuul for https://gerrit.wikimedia.org/r/814356
* 21:11 Reedy: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/601808/ onto deployment-puppetmaster04 [[phab:T254185|T254185]]
* 13:35 hashar: Built releng/helm-linter:0.2.3 https://gerrit.wikimedia.org/r/#/c/integration/config/+/601707/
* 10:31 Lucas_WMDE: wikidata/query/gui-deploy: recreated production branch (commit {{Gerrit|9cab76d1281a1198766b485c815bc753d4171e94}}), pointed HEAD to production, deleted master branch, cf. [[phab:T254027|T254027]]
* 10:26 Lucas_WMDE: but also, correctly I think, deleted deploy and production branches on wikidata/query/gui ([[phab:T254027|T254027]])
* 10:25 Lucas_WMDE: CORRECTION accidentally deleted deploy and production branches on wikidata/query/gui-deploy, production should not have been deleted there ([[phab:T254027|T254027]])
* 10:20 Lucas_WMDE: wikidata/query/gui: deleted deploy and production branches, cf. [[phab:T254027|T254027]]
* 08:27 legoktm: reload zuul config for https://gerrit.wikimedia.org/r/600403


== 2020-06-01 ==
== 2022-07-16 ==
* 20:49 James_F: Zuul: Disabled new mediawiki-quibble-apitests-vendor-docker for Math as it doesn't pass [[phab:T254031|T254031]]
* 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt
* 20:31 James_F: Zuul: EventBus now depends on EventStreamConfig for non-phan too [[phab:T229863|T229863]]
* 17:47 James_F: Zuul: EventBus now depends on EventStreamConfig [[phab:T229863|T229863]]
* 17:02 James_F: layout: Add CI for operations/debs/kubeyaml using debian-glue


== 2020-05-30 ==
== 2022-07-15 ==
* 16:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/599963
* 15:59 hashar: Built pcov php docker images [[phab:T280170|T280170]]
* 13:18 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki/php-master$ mwscript maintenance/createAndPromote.php --wiki=fawiki --bureaucrat --force --interface-admin --sysop Ladsgroup (Part of [[phab:T253291|T253291]])
* 15:46 hashar: contint2001: `docker-system-prune-dangling.service`  it failed overnight cause Docker was not running. That should clear Icinga state # [[phab:T313119|T313119]]
* 14:05 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Switch to node16 jobs
* 13:05 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]], this time actually.
* 12:30 hashar: Starting docker on contint2001.wikimedia.org # [[phab:T313119|T313119]]
* 12:20 hashar: rebuilding `php??` images for pcov https://gerrit.wikimedia.org/r/c/integration/config/+/694621 # [[phab:T280170|T280170]]
* 10:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813967
* 10:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813932


== 2020-05-29 ==
== 2022-07-14 ==
* 22:46 James_F: Zuul: Archive mediawiki/extensions/PopupPages [[phab:T251000|T251000]]
* 18:50 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]]
* 18:42 hauskatze: gerrit: replication start mediawiki/extensions/WikiShare --wait refs. [[phab:T250400|T250400]]
* 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for [[phab:T203215|T203215]]
* 18:17 hauskatze: GitHub: Deleted mirror wikimedia/mediawiki-extensions-PopupPages refs. [[phab:T251000|T251000]]
* 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
* 14:36 hashar: Built  Successfully tagged docker-registry.discovery.wmnet/releng/java8:0.6.4
* 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for [[phab:T268085|T268085]]
* 09:09 hashar: Updating all Jenkins jobs: tox -e jenkins-jobs -- --flush-cache --conf jenkins_jobs.ini update --workers 2 ./jjb
* 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI
* 07:18 hashar: Deleted last php5.6 job hurrah and thanks James ! https://integration.wikimedia.org/ci/job/composer-php56-docker/ # [[phab:T224906|T224906]]
* 07:18 hashar: Deleted last php5.6 job hurrah and thanks James ! https://integration.wikimedia.org/ci/job/composer-php56-docker/


== 2020-05-28 ==
== 2022-07-13 ==
* 17:52 James_F: Zuul: [integration/docroot] Switch from php56 to php70
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813720
* 16:10 James_F: Zuul: Move MediaModeration to production section and enforce [[phab:T247943|T247943]]
* 20:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813707
* 13:31 hashar: Rolling back Xunit Jenkins plugin  # [[phab:T194096|T194096]]


== 2020-05-27 ==
== 2022-07-12 ==
* 21:55 James_F: Nicely restarting Jenkins for xunit plugin upgrade.
* 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki ([[phab:T312865|T312865]])
* 15:33 James_F: Zuul: Re-pulling config forwards ~2 weeks on contint2001; forgotten in [[phab:T224591|T224591]]?
* 13:29 ottomata: start failed puppetdb on deployment-puppetdb03


== 2020-05-26 ==
== 2022-07-11 ==
* 12:26 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.1 # [[phab:T251869|T251869]]
* 22:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/812934
* 19:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/812467


== 2020-05-25 ==
== 2022-07-10 ==
* 23:08 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/598547
* 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet
* 14:21 hashar: Built image releng/doxygen:0.7.0 for Doxygen 1.8.17 # [[phab:T242155|T242155]]


== 2020-05-21 ==
== 2022-07-09 ==
* 16:42 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597825
* 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
* 16:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597820
* 17:25 ori: Cherry-picked {{Gerrit|Ief73cc553}} (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
* 16:19 James_F: Zuul: [mediawiki/extensions/Bootstrap] Switch down to quibble-composer for now.
* 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/597810
* 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
* 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump  # ref [[phab:T312689|T312689]]


== 2020-05-20 ==
== 2022-07-07 ==
* 22:13 James_F: Docker: Publishing new zuul-cloner image with zuul-clonemap.yaml in it [[phab:T252955|T252955]]
* 22:42 TheresNoTime: clear stuck beta deployment jobs (again), [[phab:T72597|T72597]]
* 21:18 James_F: Docker: Publishing new npm-test images with npm-install-dev.py in them [[phab:T252955|T252955]]
* 21:10 TheresNoTime: clear stuck beta deployment jobs, [[phab:T72597|T72597]]
* 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
* 12:22 hashar: integration: rebooting `integration-agent-docker-1039` [[phab:T312534|T312534]]


== 2020-05-19 ==
== 2022-07-05 ==
* 16:33 James_F: Docker: Publishing helm-linter:0.2.2 [[phab:T252428|T252428]]
* 14:17 dwalden: restarted mathoid service on deployment-docker-mathoid01
* 11:39 hashar: Reloaded Zuul for `skip selenium for Wikibase repo/rest-api` https://gerrit.wikimedia.org/r/c/integration/config/+/811258
* 08:49 hauskatze: Diffusion rORES repository. Changed URI settings: enabled SSH push for mirroring; disabled HTTP {{!}} [[phab:T311390|T311390]]


== 2020-05-18 ==
== 2022-06-30 ==
* 21:36 ottomata: started  puppetdb on deployment-puppetdb03. it was not running
* 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at [[phab:T72597|T72597]]
* 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
* 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053


== 2020-05-17 ==
== 2022-06-29 ==
* 01:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/596848
* 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
* 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')


== 2020-05-15 ==
== 2022-06-28 ==
* 19:30 James_F: Zuul: Collapse extension-quibble-only-selenium users back into extension-quibble now it's split
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 19:27 James_F: Zuul: Always split out the selenium run into their own jobs
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 19:23 James_F: Zuul: Add CI for mediawiki/extensions/FlexiSkin
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]
* 18:21 James_F: Zuul: Add TwoColConflict to the gate [[phab:T251276|T251276]]
* 17:48 James_F: Zuul: Add BetaFeatures to the gate [[phab:T249674|T249674]]
* 17:21 James_F: Zuul: Add CI for mediawiki/extensions/PageExchange [[phab:T252473|T252473]]


== 2020-05-13 ==
== 2022-06-27 ==
* 18:43 hashar: Building releng/helm-linter:0.2.1 for helm upgrade # [[phab:T252428|T252428]]
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 18:35 James_F: Zuul: [labs/toollabs] Stop running the glue job for Jessie [[phab:T247496|T247496]]
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
* 16:01 James_F: Zuul: Mark mediawiki/debian as archived in CI [[phab:T240521|T240521]]


== 2020-05-12 ==
== 2022-06-24 ==
* 22:23 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/594823
* 20:52 taavi: added `denisse` as a member
* 22:14 marxarelli: created 4 jenkins jobs with jjb for https://gerrit.wikimedia.org/r/c/integration/config/+/594823
* 20:10 cdanis: reverting to standard php7.2-fpm on deployment-mediawiki-07
* 19:34 cdanis: installing patched version of php7.2-fpm on deployment-mediawiki-07
* 18:44 brennen: cloning all (public) wmf repos from gerrit replica to a local system for number crunching
* 16:02 James_F: Zuul: Manually running fabric against contint1001 to add Privacybatm to the CI allow list [[phab:T248256|T248256]]
* 15:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/595968 [[phab:T248256|T248256]]


== 2020-05-11 ==
== 2022-06-23 ==
* 19:17 ottomata: set profile::trafficserver::tls::inbound_tls_settings.prioritize_chacha: 1, profile::cache::varnish::frontend::fe_vcl_config.large_objects_cutoff: 262144 in prefixpuppet deployment-cache-text horizon hiera to fix puppet on deployment-cache-text06
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
* 19:09 ottomata: started dead puppetdb service on deployment-puppetdb03


== 2020-05-10 ==
== 2022-06-22 ==
* 16:23 paladox: ssh -p 29418 paladox@gerrit.wikimedia.org gerrit create-project labs/tools/gdrive-to-commons.git --owner {{Gerrit|204871d01b8ce076f6559767264c7af2f8b737c7}} --empty-commit - per https://www.mediawiki.org/w/index.php?title=Gerrit%2FNew_repositories%2FRequests%2FEntries&type=revision&diff=3844303&oldid=3844168
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2020-05-07 ==
== 2022-06-21 ==
* 19:52 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/595026
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 16:20 James_F: Deleted integration-cumin.integration.eqiad.wmflabs for last part of [[phab:T236576|T236576]]
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 16:17 James_F: Shut down integration-cumin-02, non-functional buster cumin host.
* 16:15 James_F: Shutting off integration-cumin for [[phab:T236576|T236576]]
* 16:14 James_F: Dropped integration-cumin (172.16.1.103) from WMCS integration cumin master list
* 16:11 James_F: Shutting down integration-agent-stretch-1001, unused.
* 16:07 hashar: integration-agent-stretch-1001 : clearing out /var/cache/pbuilder/build
* 15:43 James_F: Added integration-cumin-01 (172.16.1.185) to WMCS integration cumin master list
* 15:38 Daimona: foreachwiki extensions/AbuseFilter/maintenance/updateVarDumps.php --print-orphaned-records {{!}} [[phab:T246539|T246539]]
* 15:12 hashar: Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/ # [[phab:T236576|T236576]]
* 15:12 hashar: Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/
* 15:11 Daimona: mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=enwiki --print-orphaned-records {{!}} [[phab:T246539|T246539]]
* 12:34 mutante: removing role::labs::lvm::srv from deployment servers since this is now included in role:deployment_server and should neve have been a role in the first place
* 12:07 mutante: - puppet still broken on deployment_servers due to unrelated pre-existing issues, also no alerts about it in shinken
* 12:04 mutante: - puppet broken on deployment_servers - fix deployed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/594932


== 2020-05-06 ==
== 2022-06-20 ==
* 20:54 James_F: Created integration-cumin-01 instance in WMCS based on stretch for final part of [[phab:T236576|T236576]]
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 20:20 hashar: Running jjb for all Jenkins jobs to drop ansicolor definition (now globally enabled) https://gerrit.wikimedia.org/r/#/c/integration/config/+/594716/ # [[phab:T233688|T233688]]
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 17:01 James_F: Deleting integration-agent-puppet-docker-1001 from WMC for final stage of [[phab:T250502|T250502]].
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 13:50 hashar: Jenkins: enable xterm ansi color map for all builds via https://integration.wikimedia.org/ci/configure # [[phab:T233688|T233688]]
* 13:29 hashar: Updating Jenkins ansicolor from 0.6.2 to 0.6.3 # [[phab:T233688|T233688]]
* 01:03 thcipriani: integration-cumin:sudo cumin -p 50 'name:docker' 'docker pull docker-registry.wikimedia.org/wikimedia-stretch:latest'


== 2020-05-05 ==
== 2022-06-17 ==
* 22:48 James_F: jjb: Updating all jobs to master.
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 20:33 James_F: jjb: Bumped wmf-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test.
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 20:23 James_F: Bumped mediawiki-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test.
* 19:05 James_F: Docker: Publishing quibble 0.0.43 images.
* 15:02 James_F: Zuul: Change pipeline trigger for train-deploy-notes job
* 14:36 James_F: Zuul: Install service pipeline for mediawiki/services/push-notifications [[phab:T251901|T251901]]


== 2020-05-04 ==
== 2022-06-16 ==
* 18:31 Krinkle: Un-stall beta cluster puppetmaster (100 patches behind due to merge conflict)
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 17:43 James_F: Shutting off integration-agent-puppet-docker-1001 in WMCS for [[phab:T250502|T250502]]
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 17:41 James_F: Dropping integration-agent-puppet-docker-1001 from jenkins for [[phab:T250502|T250502]]
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]
* 17:27 James_F: Depooling integration-agent-puppet-docker-1001 from running puppet CI [[phab:T250502|T250502]]
* 17:10 James_F: Pooling integration-agent-puppet-docker-1002 to run puppet CI [[phab:T250502|T250502]]
* 16:36 James_F: Docker: [composer-php70] Drop composer-install-dev-only, unused
* 16:13 hashar: Cleaning git repositories on deployment-deploy01


== 2020-05-01 ==
== 2022-06-15 ==
* 19:44 James_F: Zuul: [operations/puppet] Add operations-puppet-tests-buster-docker-test as experimental [[phab:T250502|T250502]]
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 19:40 James_F: jjb: Dropping operations-puppet-tests-stretch-docker, not triggered for over a month
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 19:20 James_F: Added self to integration sudoers policy
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
* 18:56 James_F: Created integration-agent-puppet-docker-1002 based on stretch for [[phab:T250502|T250502]]
* 18:43 James_F: Deleting integration-trigger-01 from WMCS for [[phab:T247109|T247109]]
* 18:41 James_F: Deleting integration-slave-jessie-1002 and integration-slave-jessie-1004 instances from WMCS for [[phab:T236576|T236576]]


== 2020-04-30 ==
== 2022-06-14 ==
* 19:49 hashar: deployment-logstash2  emptied /var/log/demon.log
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 18:32 James_F: Zuul: Install CI for mediawiki/services/texvcinfo [[phab:T249908|T249908]]
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production


== 2020-04-29 ==
== 2022-06-13 ==
* 23:59 James_F: Zuul: [mediawiki/extensions/Sentry] Move out of deployed section [[phab:T91649|T91649]]
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 16:14 James_F: Zuul: CI for wikimedia/meet-accountmanager [[phab:T251425|T251425]]
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
* 16:03 James_F: Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for [[phab:T236576|T236576]]
* 16:03 James_F: Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for [[phab:T236576|T236576]]


== 2020-04-28 ==
== 2022-06-12 ==
* 18:20 James_F: Zuul: [mediawiki/extensions/reCaptcha] Mark as archived [[phab:T250641|T250641]]
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 10:00 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$


== 2020-04-27 ==
== 2022-06-10 ==
* 23:33 James_F: Zuul: Migrate WDQS jobs to Docker [[phab:T247123|T247123]]
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 21:26 James_F: Pushed Jeena's new version of wikimedia-portals-build based on Docker for [[phab:T242909|T242909]]
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 19:45 James_F: Zuul: Run quibble jobs for mediawiki/extensions/GroupWhitelist
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 16:58 Krinkle: krinkle@contint1001$ Replay zuul event for luasandbox@3.0.3 tag to publish docs at https://doc.wikimedia.org/mediawiki-php-luasandbox/3.0.3/  - [[phab:T186302|T186302]]
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check
* 16:57 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/592706 (


== 2020-04-21 ==
== 2022-06-09 ==
* 15:05 Krinkle: install 'qemu-system-x86' package on integration-agent-qemu-1001
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 14:48 Krinkle: Creating integration-agent-qemu-1001 to experiment with VM-based CI jobs – [[phab:T250808|T250808]]
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration


== 2020-04-20 ==
== 2022-06-08 ==
* 19:47 halfak: Deploying ORES {{Gerrit|514f94a}}
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 19:16 halfak: Deploying ORES {{Gerrit|ac2eb2f}}
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 18:52 halfak: Deploying ORES {{Gerrit|a5a5bce}}
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134
* 17:07 James_F: Zuul: Unarchive the WikiShare extension [[phab:T250400|T250400]]
* 16:55 James_F: Zuul: Archive the TemplateInfo extension [[phab:T250591|T250591]]
* 15:41 halfak: Deploying ORES {{Gerrit|5d977f4}}


== 2020-04-19 ==
== 2022-06-06 ==
* 23:23 Krinkle: Change Jenkins Console Section pattern for quibble/Start to exclude fresnel user command
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 23:23 Krinkle: Fix Jenkins Console Section pattern for "setup quibble-fresnel"


== 2020-04-17 ==
== 2022-06-02 ==
* 21:21 hashar: Building Docker image  releng/node10-portals:0.1.2 for longma
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 21:11 James_F: Zuul: Add Scribunto to the gate after only five years of talking about it [[phab:T125050|T125050]]
* 11:26 hashar: Restarting Jenkins on contint2001
* 19:59 James_F: Docker: Rebuild the quibble world for 0.0.42
* 11:19 hashar: Restarting Jenkins on releases1002
* 19:01 longma: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/c/integration/config/+/589414
* 17:39 James_F: Docker: Rebuilding the PHP world to have native zip and unzip [[phab:T250496|T250496]]
* 15:49 brennen: Updating dev-images docker-pkg files on contint2001 for [[phab:T231864|T231864]]


== 2020-04-16 ==
== 2022-05-31 ==
* 20:01 hashar: gerrit: operations/debs/doxygen : renamed branch debian/buster-backports to debian/buster-wikimedia
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 17:43 James_F: Zuul: Add in-wikimedia-production to production items and assert shape
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 17:01 James_F: Zuul: Add Priyanshu varshney in CI whitelist
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2020-04-15 ==
== 2022-05-30 ==
* 23:52 thcipriani: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/589159
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 19:59 dpifke: navtiming on deployment-webperf11 reverted to master.
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]
* 19:46 dpifke: Testing patch 587775 to navtiming in beta cluster; will revert when done.
* 19:01 James_F: Zuul: Marking the ScanSet extension as archived for CI [[phab:T250240|T250240]]


== 2020-04-14 ==
== 2022-05-28 ==
* 17:00 James_F: Zuul: Archiving MagicNumberedHeadings [[phab:T248758|T248758]]; WikipediaExtracts [[phab:T248760|T248760]]; OracleTextSearch [[phab:T249656|T249656]].
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]
* 16:57 James_F: Zuul: Adding Lex Nasser to the CI whitelist


== 2020-04-13 ==
== 2022-05-27 ==
* 21:20 James_F: Zuul: Archive the MSSQLBackCompat extension [[phab:T248965|T248965]]
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 21:03 James_F: Zuul: [mediawiki/extensions/CirrusSearch] Support standalone
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 18:13 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/588444
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 18:04 James_F: Restarting deployment-mediawiki-09 too.
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 18:03 James_F: Re-enabling puppet agent on deployment-mediawiki-09; disabled for 7 weeks with no comment.
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 17:50 James_F: Zuul: Add CI for labs/tools/github-pr-closer
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 17:39 James_F: Restarting deployment-mediawiki-07 to see if it comes back.
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2020-04-11 ==
== 2022-05-26 ==
* 15:07 Krenair: Migrated from deployment-cache-text05 (stretch) to deployment-cache-text06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance - [[phab:T250006|T250006]]
* 18:33 dancy: Updated Jenkins beta-* job configs
* 14:52 Krenair: Migrated from deployment-cache-upload05 (stretch) to deployment-cache-upload06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance which coincidentally would turn one year old tomorrow
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch


== 2020-04-10 ==
== 2022-05-25 ==
* 23:01 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/588054/
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 19:28 longma: Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/c/integration/config/+/588029
* 15:46 dancy: Restarted apache2 on gerrit1001
* 18:40 James_F: Docker: Publishing composer-php70:0.2.0 and cascade.
* 17:12 twentyafterfour: deploying docker image on contint2001 with `tox -e fabric -- deploy_docker`
* 17:08 James_F: Docker: Publishing release-notes:0.0.5 on contint2001
* 16:46 hashar: contint1001: deleting docker-pkg maintained images. They are in the registry anyway.  # [[phab:T224591|T224591]]
* 07:53 Amir1: ladsgroup@deployment-deploy01:~$ mwscript createAndPromote.php --wiki=enwiki --custom-groups=checkuser --force Ladsgroup
* 00:03 Krinkle: Fix Jenkins Console Section for "setup quibble" to not consume output of 'mwext-php72-phan-seccheck-docker' jobs


== 2020-04-09 ==
== 2022-05-24 ==
* 19:50 James_F: Zuul: Adjust parsoid CI config and parsoid/deploy CI config
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 17:17 James_F: Zuul: Move last production extensions to use vendor
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 16:55 James_F: Zuul: Split Wikibase-dependent repos into non-selenium & selenium jobs
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 16:49 James_F: Zuul: parameter_functions: Drop dependencies for three archived extensions
* 16:32 James_F: Zuul: [mediawiki/extensions/TemplateStyles] Move to vendor
* 10:07 hashar: gerrit: created empty repository wikimedia/fundraising/crm/drush/vendor owned by fundraising. # [[phab:T249547|T249547]]


== 2020-04-08 ==
== 2022-05-23 ==
* 21:28 James_F: layout: [mediawiki/extensions/Scribunto] Standalone, always [[phab:T225068|T225068]]
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 21:18 James_F: layout: [mediawiki/extensions/Scribunto] Standalone, experimentally [[phab:T225068|T225068]]
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 20:25 James_F: Docker: Publishing quibble 0.0.41 images
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 19:44 hashar: Tag Quibble 0.0.41 {{Gerrit|cc2b408}} # [[phab:T234738|T234738]] [[phab:T225068|T225068]]
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 16:04 James_F: Updating all docker JJB jobs with seccomp over-ride for [[phab:T236675|T236675]]
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 16:01 James_F: Updating composer-package-php72-docker with seccomp over-ride for final [[phab:T236675|T236675]] testing
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 16:00 James_F: Updating wikipeg-npm-php-docker with seccomp over-ride for final [[phab:T236675|T236675]] testing
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 15:31 hashar: zuul-merger stopped on contint2001. It is being reimaged to Buster!  # [[phab:T224591|T224591]]
* 14:53 mutante: contint2001 down for reimaging


== 2020-04-07 ==
== 2022-05-22 ==
* 10:11 Lucas_WMDE: manually killed a Jenkins job for a gate-and-submit of a non-latest patch set
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 03:46 James_F: Zuul: [FundraisingEmailUnsubscribe] Disable MW-master testing
* 01:19 James_F: Zuul: FR voting for ContributionTracking & FundraisingEmailUnsubscribe
* 00:57 James_F: Zuul: Add experimental jobs for Fundraising extensions using PHP70/REL1_31


== 2020-04-06 ==
== 2022-05-21 ==
* 21:23 James_F: Zuul: Run CI for mediawiki/extensions/S5SlideShow
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 21:23 James_F: Zuul: Run CI for labs/tools/articles-needing-links
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]
* 21:23 James_F: Zuul: Run CI for MathSearch [[phab:T117659|T117659]]
* 16:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586394
* 16:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586388
* 16:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/586387


== 2020-04-04 ==
== 2022-05-20 ==
* 15:27 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/585884
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809


== 2020-04-03 ==
== 2022-05-19 ==
* 20:30 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246953|T246953]]
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 19:02 longma: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/584701
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already


== 2020-04-02 ==
== 2022-05-18 ==
* 22:53 James_F: Updated ooui-npm-special-node10-plus-composer-docker with COMPOSER_GITHUB_OAUTHTOKEN injected [[phab:T248387|T248387]]
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 22:00 James_F: Docker: [composer-php70] Inject GitHub OAuth token into composer runtime [[phab:T106452|T106452]] [[phab:T248387|T248387]]
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 21:46 hashar: deployment-deploy02: apt-get autoremove --purge
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 17:07 liw: testing upcoming scap on beta
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]
* 16:44 James_F: Deleted integration-agent-jessie-docker-1001 from WMCS [[phab:T249226|T249226]]
* 16:43 James_F: Deleted integration-agent-jessie-docker-1001 from jenkins [[phab:T249226|T249226]]
* 16:43 James_F: Shut off integration-agent-jessie-docker-1001 ahead of removal [[phab:T249226|T249226]]
* 16:15 James_F: Zuul: Stop running Math tests with every change in Wikibase [[phab:T249235|T249235]]
* 15:52 hashar: deployment-deploy01:  sudo chown jenkins-deploy:wikidev /srv/packages/db/*
* 15:20 hashar: Updated https://integration.wikimedia.org/ci/job/scap-beta-deb # [[phab:T249225|T249225]]
* 13:49 hashar: Add Jenkins bot wmf-insecte to #wikimedia-analytics # [[phab:T210271|T210271]]


== 2020-04-01 ==
== 2022-05-17 ==
* 21:40 hashar: gerrit: deleting old sandbox branches in mediawiki/core that were used for testing purpose # [[phab:T227159|T227159]]
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 17:07 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/585275 # [[phab:T249076|T249076]]


== 2020-03-31 ==
== 2022-05-16 ==
* 15:36 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T223643|T223643]]
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 14:45 James_F: Zuul: Add pipeline CI job for mediawiki/tools/cli [[phab:T248412|T248412]]
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809


== 2020-03-30 ==
== 2022-05-14 ==
* 20:36 James_F: Reverting [[phab:T246358|T246358]] work on JJB to fix [[phab:T248891|T248891]].
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 19:18 andrewbogott: restarting puppetdb on deployment-puppetdb03 and restarted apache2 on deployment-puppetmaster04 but puppet runs still fail everywhere
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 17:48 James_F: Zuul: [operations/software/censorship-monitoring] Switch to custom job [[phab:T248002|T248002]]
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
* 17:41 James_F: Docker: Publishing tox-censorshipmonitoring:0.1.0 [[phab:T248002|T248002]]
* 17:31 James_F: jjb: Deployed 'assert-no-mediawiki-errors' to all quibble jobs [[phab:T246358|T246358]]
* 17:12 James_F: jjb: Add 'assert-no-mediawiki-errors' to mediawiki-quibble-api-testing-vendor-*-php72-docker as a test [[phab:T246358|T246358]]
* 17:04 James_F: jjb: Add 'wikibase-client-docker' to wikibase-*-docker as a test [[phab:T246358|T246358]]
* 16:49 James_F: zuul: [operations/puppet] Use buster instead of stretch job
* 16:37 James_F: zuul: [operations/puppet] Add buster as experimental
* 16:37 James_F: https://gerrit.wikimedia.org/r/#/c/584628/
* 16:27 James_F: jjb: Created operations-puppet-tests-buster-docker
* 16:25 James_F: jjb: Sweitch operations-puppet-wmf-style-guide over to buster
* 16:17 James_F: Docker: Publish operations-puppet:0.7.0
* 16:16 James_F: [[phab:T248759|T248759]] layout: [mediawiki/extensions/ExtTab] Archive
* 16:07 James_F: [[phab:T242191|T242191]] layout: [labs/tools/qrcode-generator] Setup gate-and-submit-l10n


== 2020-03-29 ==
== 2022-05-13 ==
* 21:34 hashar: Restoring job mediawiki-core-doxygen-docker-publish deleted by Krinkle at 17:41
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2020-03-25 ==
== 2022-05-12 ==
* 22:51 James_F: Zuul: Install CI for mediawiki/extensions/UserProtect
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 17:44 greg-g: Added Shay_Nowick to deployment-prep
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2020-03-24 ==
== 2022-05-11 ==
* 22:56 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246921|T246921]]
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 16:16 James_F: Docker: Publishing new PHP images with GD installed [[phab:T236261|T236261]]
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
* 15:48 greg-g: added dkinzler@wikimedia.org to wmf-deployments (cf: [[phab:T248324|T248324]])


== 2020-03-23 ==
== 2022-05-10 ==
* 21:44 James_F: Zuul: Run all Kartographer tests alongside each patch to JsonConfig [[phab:T163274|T163274]]
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 18:43 James_F: jjb: Updated npm-node-6-docker for [[phab:T248326|T248326]] as an initial test
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 18:39 James_F: Zuul: [mediawiki/extensions/AControlImageLink] Run quibble
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 18:09 brennen: attempting to update docker-pkg files on contint1001, take 2
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 18:06 brennen: Updating docker-pkg files on contint1001 per James_F
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 17:58 James_F: Docker: Upgrade npm to v3.10.10 and cascade
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377
* 15:43 James_F: Zuul: [mediawiki/extensions/ConfigManager] Add initial CI and dependency on BlueSpiceFoundation
* 15:38 James_F: Zuul: [labs/tools/bookreader] Add l10n bot pipeline [[phab:T241595|T241595]]


== 2020-03-20 ==
== 2022-05-09 ==
* 22:43 Krinkle: Click "Disable Publishing" for extension-TitleBlacklist, [[phab:T243175|T243175]], [[phab:T143162|T143162]]
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 22:22 James_F: Zuul: Unanchor the train-deploy-notes job regex filter 582111 [[phab:T243330|T243330]]
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 18:07 James_F: jjb: Manually disabled wikimedia-portals-build cronjob. [[phab:T213806|T213806]]
* 18:06 James_F: jjb: Manually disabled DockerJessie boxes for [[phab:T239981|T239981]]
* 18:03 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T248178|T248178]]
* 00:57 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581810


== 2020-03-19 ==
== 2022-05-08 ==
* 17:46 mutante: releases/releases-jenkins - closed firewall hole to port 80 for caching servers - kept it open just for envoy from the backends - ATS speaks https to them meanwhile
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 14:30 paladox: made labs-striker self owned
* 14:09 James_F: Zuul: Added QEDK to CI whitelist


== 2020-03-18 ==
== 2022-05-06 ==
* 19:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581047
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]
* 18:41 James_F: Zuul: Add EventStreamConfig to gated extensions [[phab:T244521|T244521]]
* 18:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/581005
* 16:50 James_F: Zuul: [mediawiki/services/parsoid] Drop pipeline
* 16:50 James_F: jjb: Dropped parsoid-pipeline-publish and trigger-parsoid-pipeline-publish
* 16:47 James_F: Zuul: Run Math tests for every change in Wikibase [[phab:T201496|T201496]]
* 03:31 Amir1: manually creating empty wb_terms
* 03:16 Amir1: dropping wb_terms table from wikidatawiki in beta cluster ([[phab:T219123|T219123]] [[phab:T219175|T219175]] [[phab:T208425|T208425]])
* 00:37 James_F: jjb: Updated parsoidsvc-parsertests-docker for JUnit compression.


== 2020-03-17 ==
== 2022-05-05 ==
* 19:49 James_F: Deleted deployment-dumps-puppetmaster02 for [[phab:T241719|T241719]]
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 19:48 James_F: Deleted deployment-mediawiki-parsoid10, no longer used.
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 19:45 James_F: Restarting deployment-prep's deployment-mediawiki-07 and deployment-mediawiki-09
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 18:59 thcipriani: re-enabling puppet agent on deployment-mediawiki-07 The last Puppet run was at Mon Feb 24 18:26:55 UTC 2020 (31711 minutes ago). Puppet is disabled. reason not specified -- nobody on the machine
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 16:31 James_F: Zuul: Run PHP74 for CirrusSearch again
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 16:18 James_F: Zuul: Run PHP74 for nonselenium jobs
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins  builds from https://integration.wikimedia.org/ci/view/Beta/


== 2020-03-13 ==
== 2022-05-04 ==
* 19:38 James_F: jjb: Speculatively pushing mediawiki-fresnel-patch-docker with libgtk-3.0 for manual testing of [[phab:T247599|T247599]]
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 18:46 James_F: jjb: Make mwext-doxygen-publish publish to DOC_PROJECT not DOC_BASENAME [[phab:T246042|T246042]]
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 18:18 James_F: Docker: Pushing quibble-stretch 0.0.40-s1 and cascade.
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2020-03-12 ==
== 2022-05-03 ==
* 17:21 James_F: Zuul: Make PHP74 tests voting for almost everything [[phab:T233012|T233012]]
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 16:59 James_F: Zuul: Make PHP74 tests voting for extensions [[phab:T233012|T233012]]
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 16:53 James_F: Zuul: [MediaWiki] Switch jsduck jobs over to node10 versions [[phab:T247536|T247536]]
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 16:43 James_F: Zuul: [MediaWiki] Run mediawiki-core-javascript-docker experimentally
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 13:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/579263
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 09:38 hashar: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ broken cause it can not ssh to  deployment-parsoid11.deployment-prep.eqiad.wmflabs # [[phab:T246854|T246854]]
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 08:17 hashar: integration: removing logstash Jenkins plugin from https://integration.wikimedia.org/ci/ {{!}} [[phab:T247036|T247036]] [[phab:T78705|T78705]]
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 00:34 James_F: Shutting down deployment-mediawiki-parsoid10 ahead of its removal [[phab:T246854|T246854]]
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2020-03-11 ==
== 2022-05-02 ==
* 22:23 James_F: Zuul: [MediaWiki] Add PHP74 tests as voting for master [[phab:T233012|T233012]]
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 18:47 James_F: Zuul: Re-enabling PHP74 for all skins [[phab:T247214|T247214]]
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]
* 17:30 James_F: Soft-restarting Jenkins to drop unused plugin.
* 16:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/578983
* 00:04 James_F: Restarting release jenkins for upgrade.


== 2020-03-10 ==
== 2022-04-29 ==
* 23:52 James_F: Restarting jenkins for upgrade.
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 14:58 James_F: Add CI for operations/software/atskafka [[phab:T237993|T237993]]
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]


== 2020-03-09 ==
== 2022-04-28 ==
* 13:39 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC


== 2020-03-05 ==
== 2022-04-27 ==
* 17:08 Krinkle: Remove obsolete filters from "scap canary" dashboard, including obsolete "hhvm" query
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 15:25 James_F: Zuul: Add mediawiki-i18n-check template to mediawiki/services/mobileapps for [[phab:T246979|T246979]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029
* 01:28 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246930|T246930]]
* 00:27 greg-g: removed cmcmahon, Coren, khorn, manybubbles, mobrovac, laner from users of integration
* 00:26 greg-g: added Jforrester as user and projectadmin to integration
* 00:23 greg-g: made Krenair a projectadmin in integration


== 2020-03-07 ==
== 2022-04-26 ==
* 13:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/577776
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2020-03-06 ==
== 2022-04-25 ==
* 22:42 James_F: Raised executor count on contint1001 from 3 to 12 for [[phab:T247109|T247109]]
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 19:40 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/577653
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI
* 19:33 marxarelli: creating 2 jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/577653
* 19:05 James_F: Manually pushing trigger-helloworldoid-pipeline-test over to contint1001 to test [[phab:T247109|T247109]]
* 18:56 marxarelli: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/577626
* 18:52 marxarelli: creating 2 jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/577626
* 15:42 hashar: contint1001: docker image prune -f
* 15:14 Reedy: unstuck beta auto deploy jobs
* 14:37 Urbanecm: Live debugging [[phab:T247078|T247078]] on deployment-mediawiki-07 ended
* 13:50 Urbanecm: Live debugging [[phab:T247078|T247078]] on deployment-mediawiki-07


== 2020-03-05 ==
== 2022-04-20 ==
* 23:11 Krenair: [[phab:T236576|T236576]] Shut down integration-puppetmaster01 in favour of integration-puppetmaster-02. /root and /home archives can be found in /root on the new system. All integration instances moved to use the new master. Old master to be deleted on 12th March in the absence of objections
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service


== 2020-03-04 ==
== 2022-04-18 ==
* 22:15 James_F: Zuul: Archive CI for LiveTranslate [[phab:T196452|T196452]]
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 22:12 James_F: Zuul: Add CI for mediawiki/extensions/PushNotifications
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 20:59 hashar: deployment-puppetmaster04: /var/log/syslog /var/log/debug and /var/log/daemon.log are 5GB  filling /
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 20:59 James_F: Deleting deployment-parsoid09 for [[phab:T246395|T246395]]
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.
* 19:53 hauskatze: GitHub: extension-LiveTranslate mirror deleted refs. [[phab:T196452|T196452]]
* 19:34 James_F: Shutting off deployment-parsoid09 for [[phab:T246395|T246395]]
* 19:23 James_F: Rebooting deployment-puppetmaster04
* 15:12 thcipriani: restarting zuul
* 01:34 James_F: Beta Cluster: Created deployment-parsoid11 in Horizon [[phab:T246854|T246854]] to test 576493.
* 01:23 Krenair: Made James a deployment-prep projectadmin


== 2020-03-03 ==
== 2022-04-14 ==
* 22:51 James_F: Zuul: Add gabrielchihonglee to CI allow list
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 22:43 James_F: Docker: Publish mediawiki-phan-testrun 0.1.4
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 22:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/575575
* 18:22 Daimona: Manually fixing eswikibooks DB rows with P10600 refs [[phab:T246539|T246539]]
* 15:23 Daimona: mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=eswikibooks {{!}} [[phab:T246539|T246539]]


== 2020-03-02 ==
== 2022-04-12 ==
* 21:14 Krenair: restarted php7.2-fpm on deployment-mediawiki-07
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 18:22 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T246202|T246202]]
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141


== 2020-02-29 ==
== 2022-04-08 ==
* 14:34 Reedy: !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/575689
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 11:02 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/575563


== 2020-02-28 ==
== 2022-04-07 ==
* 20:00 brennen: finished deleting wmf/1.34.0-wmf.* branches for [[phab:T244368|T244368]]
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 19:37 brennen: in process of deleting many old branches for [[phab:T244368|T244368]]
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]
* 12:40 hauskatze: Migrated https://github.com/wikimedia/mediawiki-extensions-NearbyPages to Gerrit; GitHub repo now watches the Gerrit one
* 00:18 James_F: jjb: Deleting wikidata-query-gui-build


== 2020-02-27 ==
== 2022-04-06 ==
* 18:11 James_F: Reset mediawiki-quibble-composer-mysql-php72-docker to nominal [[phab:T246358|T246358]]
* 20:03 thcipriani: rebooting phabricator
* 17:59 James_F: Updating mediawiki-quibble-composer-mysql-php72-docker to test probably-breaking requirement that error logs are empty [[phab:T246358|T246358]]
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]
* 15:53 James_F: Docker: Publish mediawiki-phan-testrun:0.1.3 for [[phab:T226117|T226117]]
* 00:56 Krenair: Added edtadros to deployment-prep per request on IRC and https://en.wikipedia.org/w/index.php?title=User:ETadros_(WMF)/sandbox&diff=942815687&oldid=901073254 - wants access to eventlog05 in particular


== 2020-02-26 ==
== 2022-04-04 ==
* 21:45 James_F: Docker: Publishing mediawiki-phan-testrun:0.1.0 for [[phab:T226117|T226117]]
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 17:12 James_F: Zuul: Set up CI for mediawiki/extensions/MediaModeration [[phab:T246243|T246243]]
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 00:17 James_F: Zuul: Add a job to test new phan changes on MW core [[phab:T226117|T226117]]
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul


== 2020-02-25 ==
== 2022-04-02 ==
* 23:18 Krenair: Added mayakpwiki to deployment-prep, user is already in the nda group and is working for WMF Product Analytics. Wants access to eventlog05
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
* 23:04 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/574890
* 21:00 hauskatze: gerrit: replication start mediawiki/extensions/MediaModeration --wait # New repo
* 20:43 James_F: Running a manual sync for Beta Cluster.
* 20:17 James_F: Beta jobs on Jenkins backed up; doing the restart dance.
* 19:07 James_F: Ran `sudo -u doc-uploader git -C /srv/docroot pull` on doc1001.
* 16:52 James_F: Zuul: [mediawiki/extensions/Popups] Publish PHP documentation [[phab:T242779|T242779]]
* 13:34 godog: set number of replicas for es indices to 0 on deployment-logstash03


== 2020-02-24 ==
== 2022-03-31 ==
* 20:59 James_F: doc1001: `sudo -u doc-uploader rm /srv/docroot/org/wikimedia/doc/MinervaNeue/master` except PHP bits from Doxygen; everything else is at mediawiki-skins-MinervaNeue/ now
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
* 20:45 James_F: Zuul: [mediawiki/skins/Vector] Publish PHP documentation [[phab:T242779|T242779]]
* 18:12 brennen: restarted jenkins on releases1001 and releases2001
* 15:23 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/574485
* 13:50 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 13:44 Urbanecm: deployment-prep: Run scap sync manually, it doesn't seem it was invoked by jenkins for some reason


== 2020-02-21 ==
== 2022-03-29 ==
* 23:27 James_F: Zuul: Allow l10n on analytics/wikistats2 for [[phab:T245805|T245805]]
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 19:38 thcipriani: remove a bunch of extensions from beta that were on disk but not in https://gerrit.wikimedia.org/rg/mediawiki/extensions/
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 01:02 James_F: Zuul: [operations/mediawiki-config] Run tox-docker always
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
* 00:59 James_F: Zuul: [operations/mediawiki-config] Run tox-docker as experimental


== 2020-02-20 ==
== 2022-03-28 ==
* 22:08 thcipriani: killing a whole bunch of backed-up beta-scap-eqiad jobs
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 21:50 James_F: deployment-deploy01 back online; Jenkins backlog clearing for beta cluster jobs.
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 21:47 James_F: Taking deployment-deploy01 temporarily offline
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 19:44 hauskatze: GitHub: RENAMED https://github.com/wikimedia/operations-software-atskafka to https://github.com/wikimedia/atskafka # [[phab:T245731|T245731]]
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 19:43 hauskatze: GitHub: Created https://github.com/wikimedia/operations-software-atskafka # [[phab:T245731|T245731]]
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 19:42 hauskatze: replication start operations/software/atskafka --wait
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]
* 10:04 Reedy: postmerge jobs for mediawiki-config look stuck
* 10:03 Reedy: postmerge jobs look stuck


== 2020-02-19 ==
== 2022-03-27 ==
* 18:58 thcipriani: unblocking beta-scap-eqiad job (stuck for 18 hrs)
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]
* 16:45 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/572488


== 2020-02-18 ==
== 2022-03-26 ==
* 22:01 brennen: adding dpifke to Gerrit Managers
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 21:23 James_F: Zuul: [mediawiki/extensions/Teahouse] Archive repo [[phab:T176349|T176349]]


== 2020-02-17 ==
== 2022-03-25 ==
* 12:34 hauskatze: extension-Teahouse@phab: Mirror deactivated # [[phab:T176349|T176349]]
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 12:34 hauskatze: extension-Teahouse@github: Mirror deleted # [[phab:T176349|T176349]]
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord


== 2020-02-16 ==
== 2022-03-24 ==
* 21:24 hauskatze: AbuseFilterBypass@github: Deleted 'dev' branch {{!}} [[phab:T245363|T245363]]
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 21:24 hauskatze: AbuseFilterBypass@github: Switch default branch of mirror to 'master' {{!}} [[phab:T245363|T245363]]
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck


== 2020-02-15 ==
== 2022-03-23 ==
* 23:52 legoktm: rebuilding tox docker image for https://gerrit.wikimedia.org/r/572453
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 22:21 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/572423
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 21:33 hauskatze: GitHub: Activated Travis CI at https://travis-ci.org/wikimedia/texvcjs for https://github.com/wikimedia/texvcjs per https://github.com/wikimedia/texvcjs/pull/34 and [[phab:T245344|T245344]]
* 20:25 Krenair: [[phab:T243226|T243226]] Shut off deployment-puppetdb02 and deployment-puppetmaster03 - will leave for a week before deletion


== 2020-02-14 ==
== 2022-03-22 ==
* 21:53 andrewbogott: moving deployment-puppetdb02 and deployment-puppetmaster03 off of cloudvirt1014 (which will be drained next week anyway)
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 20:33 brennen: Updating dev-images docker-pkg files on contint1001 to attempt a rebuild for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945
* 20:05 thcipriani: deployment-mwmaint01:rm lingering .~tmp~ directories from failed rsync's taking up space
* 19:31 thcipriani: deployment-snapshot01:sudo apt-get clean cleared 1.2G of space
* 17:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/571849
* 17:23 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/571849
* 17:20 thcipriani: updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/571849


== 2020-02-13 ==
== 2022-03-21 ==
* 22:19 James_F: Zuul: Add *.microsoft.com in email_whitelist
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 17:10 thcipriani: truncate deployment-puppetmaster04:/var/log/daemon.log -> 5G, 100% disk usage
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]
* 15:58 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828


== 2020-02-12 ==
== 2022-03-18 ==
* 21:36 thcipriani: reloading zuul in an attempt to troubleshoot [[phab:T244690|T244690]]
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 20:22 James_F: Depooling integration-agent-docker-1001; it's being odd [[phab:T245054|T245054]]
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 17:11 brennen: restarting jenkins for updates
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
* 01:24 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T228910|T228910]]


== 2020-02-11 ==
== 2022-03-17 ==
* UKInEU: ssh -p 29418 paladox@gerrit.wikimedia.org replication start wikibase/vuejs-components --wait
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 21:27 hauskatze: github: Created https://github.com/wikimedia/wikibase-vuejs-components {{!}} [[phab:T240224|T240224]]
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 18:24 marxarelli: 1.35.0-wmf.19 was branched at {{Gerrit|3ac2e9ff045dc081317c9184dc83cf863fd518be}} for [[phab:T233867|T233867]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 16:44 James_F: Zuul: [mediawiki/extensions/Echo] Add experimental Postgres job
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 16:17 James_F: Terminated two multi-day running "parsoid-pipeline-publish" jobs
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]
* 09:46 Daimona: foreachwiki extensions/AbuseFilter/maintenance/fixOldLogEntries.php {{!}} [[phab:T228655|T228655]]
* 09:43 Daimona: mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=enwiki {{!}} [[phab:T228655|T228655]]


== 2020-02-10 ==
== 2022-03-16 ==
* 23:42 James_F: Zuul: [parsoid/deploy] Make PHP tests voting [[phab:T239642|T239642]]
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 23:28 legoktm: restarting zuul
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 22:24 James_F: Zuul: Add tox for operations/homer/mock-private [[phab:T244690|T244690]]
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 19:48 James_F: Add mahveotm to CI whitelist
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep
* 16:20 Daimona: mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=dewiki {{!}} [[phab:T228655|T228655]]
* 09:42 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571254
* 09:39 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/542071
* 09:25 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/571251
* 09:11 legoktm: building php74-compile image https://gerrit.wikimedia.org/r/571247
* 09:02 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571242
* 08:44 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/539724
* 08:30 legoktm: reloading zuul for https://gerrit.wikimedia.org/r/571236
* 08:17 legoktm: updating tox-mysqld docker-pkg image https://gerrit.wikimedia.org/r/571234
* 07:43 legoktm: updating tox docker-pkg image for gettext https://gerrit.wikimedia.org/r/571019
* 07:20 legoktm: updating tox docker-pkg image (try #2) https://gerrit.wikimedia.org/r/571127
* 07:00 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/571017


== 2020-02-07 ==
== 2022-03-15 ==
* 23:20 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T244382|T244382]]
* 18:26 brennen: gitlab: removed most existing /people groups
* 23:20 dpifke: puppetdb on deployment-puppetdb03 was killed by kernel OOM at Feb  7 09:50:29, per syslog. I just ran `systemctl start puppetdb` on that host, to fix puppet issues in beta.
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 19:10 James_F: Zuul: [labs/tools/video-cut-tool-back-end] Add node10 CI [[phab:T244079|T244079]]
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 19:01 James_F: Zuul: [labs/tools/VideoCutTool] Switch from tox to node [[phab:T244079|T244079]]
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 16:57 halfak: deploying ores {{Gerrit|a6f4f14}}
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2020-02-06 ==
== 2022-03-14 ==
* 20:27 James_F: Zuul: Run WikibaseCirrusSearch tests for every change in CirrusSearch [[phab:T221135|T221135]]
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2020-02-05 ==
== 2022-03-11 ==
* 23:39 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/568140
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
* 16:56 thcipriani: Updating docker-pkg files on contint100 to deploy  https://gerrit.wikimedia.org/r/566915
* 16:40 thcipriani: restarting zuul


== 2020-02-04 ==
== 2022-03-10 ==
* 23:28 Krinkle: Puppet agent has not been able to run on deployment-mediawiki-07 for 2.5 days since 2 Feb 18:27
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 01:56 James_F: Zuul: [wikimedia/portals] Add service-pipeline configuration [[phab:T238747|T238747]]
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 01:42 James_F: Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker always [[phab:T243975|T243975]]
* 20:03 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2020-02-03 ==
== 2022-03-09 ==
* 22:53 James_F: Zuul: Add CI for labs/tools/VideoCutTool [[phab:T244079|T244079]]
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 22:36 James_F: Zuul: New email address for Ricordisamoa
* 21:44 James_F: Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker experimentally [[phab:T243975|T243975]]
* 16:43 hauskatze: Gerrit: replication start mediawiki/extensions/AControlImageLink --wait # new repo
* 16:28 James_F: Zuul: Publish `npm run doc` for Vector [[phab:T242674|T242674]]


== 2020-02-02 ==
== 2022-03-08 ==
* afk: deployment-deploy01$ while read host; do ssh-keyscan -H "$host"; done < <(grep -vP '^#' /etc/dsh/group/mediawiki-installation {{!}} grep -vP '^$' ) {{!}} sudo -u jenkins-deploy tee /mnt/home/jenkins-deploy/.ssh/known_hosts
* 20:31 brennen: requiring 2fa for all users under /repos


== 2020-01-31 ==
== 2022-03-07 ==
* 11:47 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php enwiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat  # make myself bureaucrat to make colleagues sysops
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]
* 00:48 James_F: Restarting Jenkins.


== 2020-01-29 ==
== 2022-03-04 ==
* 23:55 James_F: zuul: [data-values/value-view] Migrate to node10, now that that passes [[phab:T228453|T228453]]
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 20:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/568600
* 19:13 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/768068
* 00:05 James_F: layout: [mediawiki/extension/NSFileRepo] Restore tests now they're fixed [[phab:T196480|T196480]]


== 2020-01-26 ==
== 2022-03-03 ==
* 23:23 James_F: Zuul: Mark mediawiki/extensions/BlueSpiceMenues as archived [[phab:T240788|T240788]]
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 22:20 James_F: Zuul: Mark mediawiki/extensions/SemanticTasks as archived [[phab:T221549|T221549]]
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 18:51 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php ep[campus,online,coordinator,instructor] {{!}} [[phab:T241504|T241504]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 11:02 hauskatze: github - deleted mirror for extension-BlueSpiceMenues; refs. [[phab:T240788|T240788]]
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2020-01-24 ==
== 2022-03-02 ==
* 21:15 halfak: deploying ores {{Gerrit|283f627}}
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 01:52 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/566911
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 01:45 thcipriani: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/566910
* 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
* 00:08 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/566895
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04
* 00:02 James_F: Taking deployment-deploy01 offline to fix beta config update deadlock.


== 2020-01-23 ==
== 2022-02-28 ==
* 23:43 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/562365 and ttps://gerrit.wikimedia.org/r/562364
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 21:59 James_F: Drop REL1_32 testing, jobs, and pipelines, EOLed  [[phab:T242981|T242981]]
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 21:50 halfak: deploying ores {{Gerrit|039251f}} (reverting to last good state)
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin
* 17:17 James_F: layout: [mediawiki/extensions/TagContent] Archive repo [[phab:T243093|T243093]]
* 15:31 halfak: deploying ores {{Gerrit|283f627a}}


== 2020-01-22 ==
== 2022-02-26 ==
* 20:19 James_F: Zuul: Adding TextExtracts as a dependency for TwitterCards [[phab:T242171|T242171]]
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 19:37 Krinkle: Restore php-apcu version on deployment-mediawiki-07 to …wmf1, ref [[phab:T236800|T236800]]
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 19:09 Krinkle: Installing newer version of php-apcu on deployment-mediawiki-07, ref [[phab:T236800|T236800]]
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases


== 2020-01-21 ==
== 2022-02-24 ==
* 22:10 Krenair: deployment-prep added jbond as admin, [[phab:T243226|T243226]]
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 22:09 mutante: gerrit - rsyncing 'git' and 'plugin' data dirs and /var/lib/gerrit2/review_site/ from gerrit1001 to gerrit1002 WITH --delete [[phab:T239151|T239151]]
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 21:48 mutante: gerrit - rsyncing git data from gerrit1001 to gerrit1002 ([[phab:T239151|T239151]])
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 19:16 marxarelli: updating 105 jobs affected by https://gerrit.wikimedia.org/r/c/integration/config/+/566332
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2020-01-20 ==
== 2022-02-23 ==
* 21:24 Krinkle: fix Jenkins Console Section patterns to account for change in Quibble output (no more colors and now "cmd: << Start/Finish x" instead of "cmd:x finished"
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 10:48 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/566001/ (node10 wikidata query gui)
* 20:10 dancy: Updating scap in beta
* 10:10 addshore: the last zuul reload also pulled in "jjb: [java8-sonar-scanner] version bump to 0.5.4" .. [[phab:T238004|T238004]]
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 10:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/565985/
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2020-01-17 ==
== 2022-02-22 ==
* 19:52 James_F: Docker: Publishing java8-sonar-scanner:0.5.4 on contint1001 [[phab:T238004|T238004]]
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 19:25 James_F: Zuul: [operations/debs/hhvm] Archive [[phab:T237038|T237038]]
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])
* 19:01 James_F: Zuul: [AbuseFilterBypass] Install quibble for initial CI
* 18:53 James_F: Zuul: [skin-quibble] Disable PHP74 in gate for now [[phab:T242980|T242980]]
* 18:30 James_F: Zuul: [WebAuthn] Add phan dependency on OATHAuth
* 18:27 James_F: Zuul: [WikiSEO] Add phan and seccheck jobs
* 18:24 James_F: Zuul: [ShortDescription] Add phan and seccheck jobs
* 18:21 James_F: Zuul: [NativeSvgHandler] Add phan and seccheck jobs
* 18:19 James_F: Zuul: [Sofa] Add phan and seccheck jobs
* 01:31 Reedy: apt-get clean and purged some old kernels from deployment-logstash2 to free up ~1.7G space


== 2020-01-16 ==
== 2022-02-21 ==
* 20:09 James_F: Zuul: Add Aron Manning to CI whitelist
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 16:37 longma: bringing integration-slave-docker-1003 back online
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 16:30 longma: restarting integration-agent-docker-1003
* 07:27 hashar: Refreshing all Jenkins jobs
* 16:24 longma: s/slave/agent
* 16:22 longma: bringing integration-slave-docker-1006 back online
* 16:15 longma: integration-slave-docker-1006:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '<nowiki>{</nowiki>print $3<nowiki>}</nowiki>')
* 14:28 Reedy: killed beta job to unblock beta-scap et al
* 09:31 Amir1: "delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" ([[phab:T219301|T219301]] [[phab:T225054|T225054]])


== 2020-01-15 ==
== 2022-02-20 ==
* 23:00 Reedy: deleted `contint2001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like [[phab:T134062|T134062]]
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
* 22:43 Reedy: deleted `contint1001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like [[phab:T134062|T134062]]
* 15:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/565042
* 01:15 marxarelli: deploy zuul layout change  https://gerrit.wikimedia.org/r/c/integration/config/+/564813


== 2020-01-14 ==
== 2022-02-19 ==
* 21:44 James_F: Zuul: Stop running JS tests for Parsoid and Parsoid-deploy [[phab:T242782|T242782]]
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 19:38 Krinkle: No syslog entries from php-fpm or puppet-agent on deployment-mediawiki-07, [[phab:T242659|T242659]], rebooting server
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 18:02 James_F: Zuul: Add JeremyNguyen to Jenkins whitelist [[phab:T235286|T235286]]
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]
* 17:54 James_F: Zuul: [mediawiki/core] Stop publishing docker images for now [[phab:T242775|T242775]]
* 15:33 James_F: Depooling integration-agent-docker-1003 due to issues.


== 2020-01-13 ==
== 2022-02-18 ==
* 19:34 James_F: Zuul: [mediawiki/extensions/NativeSvgHandler] Add basic CI via quibble
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 16:50 James_F: Docker: Publishing java7-sonar-scanner:0.5.3 on contint1001
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}


== 2020-01-10 ==
== 2022-02-17 ==
* 23:22 James_F: Zuul: Restore extension-coverage to projects with …-codehealth [[phab:T242444|T242444]]
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 18:35 bd808: Restarted zuul on contint1001 at 18:29Z; no logs since 2020-01-10 17:55:28,452
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 11:12 hashar: Bringing deployment-deploy03 back
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome


== 2020-01-09 ==
== 2022-02-16 ==
* 17:26 James_F: Zuul: Run sonar scanner analysis in codehealth pipeline for select Java projects [[phab:T238004|T238004]]
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 17:04 James_F: Docker: Publishing java7-sonar-scanner:0.5.2
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 16:29 James_F: Docker: Publish PHP 7.2.26-based images for [[phab:T241223|T241223]]
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}


== 2020-01-08 ==
== 2022-02-15 ==
* 20:56 James_F: Docker: Publishing Quibble 0.0.40 images on contint1001 [[phab:T192167|T192167]] [[phab:T220586|T220586]] [[phab:T236222|T236222]] [[phab:T236680|T236680]]
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 20:42 James_F: Terminating some jobs Zuul doesn't know about.
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 20:41 mutante: contint1001 - restartted zuul service
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 20:39 James_F: Safe-restarting Jenkins to see if it fixes the lack of progress
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 19:57 thcipriani: root@integration-cumin:~# cumin -b1 'name:integration-agent-docker' 'sleep 1; git -C /srv/git/mediawiki/core.git fetch'
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 19:25 James_F: Tagged Quibble 0.0.40 @ {{Gerrit|ba365ab36fd84a87fcfb10e0910255e8b44432f1}} [[phab:T192167|T192167]] [[phab:T220586|T220586]] [[phab:T236222|T236222]] [[phab:T236680|T236680]]
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}
* 16:26 halfak: deploiying ores {{Gerrit|039251f}}


== 2020-01-07 ==
== 2022-02-14 ==
* 18:18 James_F: Whitelist: Add RetroCraft and Ostrzyciel
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 18:16 James_F: Zuul: [mediawiki/extensions/SparkPost] Run phan
* 16:16 Amir1: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 09:02 twentyafterfour: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/562417
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon


== 2020-01-06 ==
== 2022-02-12 ==
* 20:43 James_F: Deleting https://doc.wikimedia.org/Wikibase/wmf/1.34.0-wmf.4 for [[phab:T240206|T240206]]
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 20:43 James_F: Deleting https://doc.wikimedia.org/Wikibase/wmf/1.33.0-wmf.23 for [[phab:T240206|T240206]]
 
* 20:11 James_F: Zuul: [SoftRedirector] Add phan and seccheck
== 2022-02-10 ==
* 19:45 James_F: Zuul: Add CI for new event schema repos [[phab:T240985|T240985]]
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 17:24 James_F: Zuul: Added ptrcnull to the CI whitelist
 
* 17:05 James_F: Zuul: [labs/tools/indic-wscontest] Provide CI with tox-docker [[phab:T241537|T241537]]
== 2022-02-09 ==
* 15:22 taavi: deleted shutoff deployment-mx02
 
== 2022-02-08 ==
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
 
== 2022-02-07 ==
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
 
== 2022-02-04 ==
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
 
== 2022-02-03 ==
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
 
== 2022-02-02 ==
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta
 
== 2022-02-01 ==
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]
 
== 2022-01-31 ==
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]
 
== 2022-01-28 ==
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`
 
== 2022-01-27 ==
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2  # [[phab:T299431|T299431]]
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins
 
== 2022-01-26 ==
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]
 
== 2022-01-25 ==
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/
 
== 2022-01-24 ==
* 21:44 Reedy: unstick beta ci jobs
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>
 
== 2022-01-22 ==
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
 
== 2022-01-21 ==
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
 
== 2022-01-20 ==
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org
 
== 2022-01-19 ==
* 18:20 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]
 
== 2022-01-18 ==
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0  with proper PHP version in the images # [[phab:T299389|T299389]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0
 
== 2022-01-17 ==
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
 
== 2022-01-14 ==
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta
 
== 2022-01-13 ==
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins
 
== 2022-01-12 ==
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
 
== 2022-01-11 ==
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
 
== 2022-01-10 ==
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]
 
== 2022-01-05 ==
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/
 
== 2022-01-04 ==
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383
 
== 2022-01-03 ==
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001.  Restarted CI Jenkins.
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


== 2020-01-02 ==
* 18:19 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php translate-proofr {{!}} [[phab:T241504|T241504]]
* 18:16 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php afttest-hide {{!}} [[phab:T241504|T241504]]
* 18:15 hauskatze: maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php afttest {{!}} [[phab:T241504|T241504]]
* Away: Archive RecentActivityFeed extension [[phab:T241032|T241032]]
* Away: Update Revi's e-mail on the whitelist
* Away: Add Safan41 to the whitelist [[phab:T235286|T235286]]
* Away: Add BrandonXLF to the whitelist [[phab:T235286|T235286]]
* 10:40 dcausse: created missing elastic indices: [[phab:T241487|T241487]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 20:51, 16 August 2022

2022-08-16

  • 20:51 RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350
  • 20:30 hashar: Repooled integration-agent-docker-1028 , it was mysteriously unreachable T315372
  • 19:18 Krinkle: mediawiki/extensions/EventLogging$ git remote-wildcard-br-d 'wmf/1.35*' 'wmf/1.36*' 'wmf/1.37*' 'wmf/1.38*'
  • 19:17 Krinkle: mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref T303828
  • 19:16 TheresNoTime: manually running `/usr/local/bin/wmf-beta-update-databases.py` on `deployment-deploy03`
  • 17:16 TheresNoTime: soft-rebooting deployment-mediawiki12

2022-08-12

  • 17:47 dancy: Restarting zuul
  • 17:42 dancy: Restarting Jenkins in an attempt to get CI jobs running again
  • 00:54 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service . Certificate was close to expiry

2022-08-11

  • 21:11 mutante: restarted phd service on phab2001
  • 19:12 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16
  • 12:26 jnuche: Reenabled CI beta sync jobs after cluster incident
  • 11:48 jnuche: Temporarily disabled CI beta sync jobs until issue in cluster is resolved
  • 10:25 zabe: take deployment-prep out of read-only mode

2022-08-10

2022-08-09

  • 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
  • 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
  • 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
  • 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for T293924
  • 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for T293851

2022-08-08

  • 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
  • 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for T309872
  • 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for T314630
  • 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166

2022-08-05

  • 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for T296967
  • 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
  • 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for T314630
  • 14:48 James_F: Zuul: Add WelpThatWorked to allow list
  • 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin

2022-08-04

2022-08-03

  • 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
  • 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
  • 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
  • 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
  • 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for T309872
  • 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
  • 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too

2022-08-02

2022-08-01

  • 23:16 James_F: Zuul: [design/codex] Switch to node16
  • 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
  • 22:53 TheresNoTime: remove stuck beta deployment jobs
  • 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
  • 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
  • 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
  • 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
  • 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
  • 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for T314306
  • 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
  • 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
  • 09:14 TheresNoTime: clearing stuck beta CI jobs

2022-07-29

  • 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for T313919
  • 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes

2022-07-28

2022-07-27

  • 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch T300463
  • 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for T274965
  • 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
  • 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
  • 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref T143162, T193565

2022-07-25

2022-07-23

2022-07-21

  • 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster

2022-07-20

  • 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
  • 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline

2022-07-19

  • 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
  • 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # T310905

2022-07-18

2022-07-17

2022-07-16

  • 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt

2022-07-15

2022-07-14

  • 18:50 James_F: Docker: Building node16 images for CI for T313075
  • 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for T203215
  • 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
  • 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for T268085
  • 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI

2022-07-13

2022-07-12

  • 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki (T312865)

2022-07-11

2022-07-10

  • 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet

2022-07-09

  • 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
  • 17:25 ori: Cherry-picked Ief73cc553 (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
  • 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
  • 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
  • 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump # ref T312689

2022-07-07

  • 22:42 TheresNoTime: clear stuck beta deployment jobs (again), T72597
  • 21:10 TheresNoTime: clear stuck beta deployment jobs, T72597
  • 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
  • 12:22 hashar: integration: rebooting `integration-agent-docker-1039` T312534

2022-07-05

2022-06-30

  • 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at T72597
  • 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
  • 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053

2022-06-29

  • 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
  • 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')

2022-06-28

2022-06-27

2022-06-24

  • 20:52 taavi: added `denisse` as a member

2022-06-23

2022-06-22

  • 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
  • 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
  • 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
  • 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
  • 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
  • 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
  • 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180

2022-06-21

  • 22:01 brennen: gitlab-runners: re-registering all shared runners
  • 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster

2022-06-20

  • 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
  • 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
  • 14:07 ori: restarted acme-chief on deployment-acme-chief03

2022-06-17

  • 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
  • 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack

2022-06-16

  • 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
  • 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
  • 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]

2022-06-15

  • 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
  • 16:31 hashar: integration-agent-docker-1035: docker image prune
  • 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster

2022-06-14

  • 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
  • 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
  • 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
  • 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production

2022-06-13

  • 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
  • 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore

2022-06-12

2022-06-10

  • 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
  • 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
  • 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
  • 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check

2022-06-09

  • 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
  • 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration

2022-06-08

  • 17:14 hashar: Reloaded Zuul for I393422
  • 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for T237033
  • 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134

2022-06-06

  • 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711

2022-06-02

  • 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
  • 11:26 hashar: Restarting Jenkins on contint2001
  • 11:19 hashar: Restarting Jenkins on releases1002

2022-05-31

  • 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
  • 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
  • 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
  • 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster

2022-05-30

  • 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
  • 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142

2022-05-28

  • 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437

2022-05-27

  • 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
  • 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
  • 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
  • 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
  • 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
  • 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
  • 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
  • 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep

2022-05-26

  • 18:33 dancy: Updated Jenkins beta-* job configs
  • 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 2c7b5825
  • 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch

2022-05-25

  • 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] T309230
  • 15:46 dancy: Restarted apache2 on gerrit1001

2022-05-24

2022-05-23

  • 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
  • 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
  • 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
  • 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
  • 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
  • 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
  • 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982

2022-05-22

2022-05-21

2022-05-20

2022-05-19

2022-05-18

  • 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
  • 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
  • 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
  • 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071

2022-05-17

  • 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558

2022-05-16

2022-05-14

  • 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
  • 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
  • 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request

2022-05-13

2022-05-12

  • 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
  • 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
  • 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
  • 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
  • 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)

2022-05-11

  • 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
  • 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster

2022-05-10

2022-05-09

2022-05-08

  • 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057

2022-05-06

  • 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071

2022-05-05

2022-05-04

2022-05-03

2022-05-02

2022-04-29

2022-04-28

2022-04-27

2022-04-26

  • 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
  • 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
  • 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
  • 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
  • 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
  • 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204

2022-04-25

2022-04-20

  • 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

2022-04-18

  • 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
  • 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
  • 16:48 brennen: phabricator: adding self to acl*blog-admins
  • 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.

2022-04-14

2022-04-12

2022-04-08

2022-04-07

  • 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
  • 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461

2022-04-06

  • 20:03 thcipriani: rebooting phabricator
  • 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596

2022-04-04

2022-04-02

2022-03-31

2022-03-29

  • 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
  • 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
  • 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771

2022-03-28

2022-03-27

  • 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736

2022-03-26

  • 02:37 Reedy: beta-update-databases-eqiad is back to @hourly

2022-03-25

  • 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
  • 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
  • 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
  • 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
  • 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord

2022-03-24

  • 20:00 hashar: reloading Zuul for Id844e1 # T299320
  • 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
  • 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
  • 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck

2022-03-23

  • 17:35 brennen: restarting phabricator for T304540, brief downtime expected
  • 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster

2022-03-22

2022-03-21

  • 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
  • 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203

2022-03-18

  • 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
  • 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
  • 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)

2022-03-17

2022-03-16

2022-03-15

2022-03-14

  • 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
  • 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
  • 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
  • 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
  • 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
  • 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
  • 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
  • 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
  • 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
  • 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
  • 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
  • 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079

2022-03-11

  • 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki

2022-03-10

2022-03-09

2022-03-08

  • 20:31 brennen: requiring 2fa for all users under /repos

2022-03-07

  • 10:53 zabe: restarted apache on deployment-mediawiki11 # T302699

2022-03-04

2022-03-03

2022-03-02

  • 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
  • 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
  • 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
  • 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04

2022-02-28

  • 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
  • 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
  • 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin

2022-02-26

  • 20:05 zabe: apply T302658 on deployment-prep centralauth databases
  • 13:24 zabe: apply T302660 on deployment-prep centralauth databases
  • 13:19 zabe: apply T302659 on deployment-prep centralauth databases

2022-02-24

  • 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
  • 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
  • 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
  • 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
  • 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
  • 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
  • 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
  • 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1

2022-02-23

  • 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
  • 20:10 dancy: Updating scap in beta
  • 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
  • 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
  • 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
  • 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774

2022-02-22

  • 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
  • 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

  • 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
  • 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
  • 07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

  • 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

  • 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
  • 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
  • 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

2022-02-17

  • 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
  • 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
  • 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
  • 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
  • 11:12 hashar: Bringing deployment-deploy03 back
  • 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

  • 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
  • 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
  • 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
  • 10:09 hashar: Reloading Zuul for I997fee
  • 09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

  • 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
  • 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
  • 16:24 hashar: Restarting CI Jenkins for plugins updates
  • 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
  • 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
  • 15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

  • 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
  • 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
  • 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
  • 14:26 jnuche: Jenkins upgrade complete T301361
  • 13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

  • 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

2022-02-09

  • 15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

  • 17:34 taavi: remove scap from deployment-kafka-main/jumbo
  • 16:23 taavi: hard reboot misbehaving deployment-echostore01
  • 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

2022-02-04

2022-02-03

  • 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
  • 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
  • 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

  • 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
  • 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

2022-01-31

  • 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
  • 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

  • 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
  • 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

  • 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
  • 19:34 Amir1: Reloading Zuul to deploy 757464
  • 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
  • 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
  • 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
  • 00:24 thcipriani: restarting jenkins

2022-01-26

  • 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
  • 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
  • 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
  • 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
  • 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

  • 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
  • 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
  • 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
  • 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
  • 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
  • 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
  • 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
  • 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
  • 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
  • 09:22 Reedy: unblocked beta again
  • 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

  • 21:44 Reedy: unstick beta ci jobs
  • 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
  • 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

  • 13:40 taavi: apply T299827 on deployment-prep centralauth database
  • 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

  • 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
  • 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

  • 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
  • 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
  • 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
  • 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
  • 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
  • 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
  • 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
  • 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
  • 12:10 hashar: contint2001 : docker container prune && docker image prune
  • 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
  • 12:04 hashar: contint1001 `docker image prune`
  • 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

2022-01-18

  • 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
  • 18:01 taavi: added ryankemper as a member of the deployment-prep project
  • 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
  • 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
  • 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
  • 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

  • 16:28 hashar: Building Quibble 1.3.0 Docker images
  • 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
  • 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

  • 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
  • 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
  • 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
  • 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

  • 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
  • 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
  • 16:27 dancy: testing scap prep auto on deployment-deploy03
  • 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
  • 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
  • 10:52 hashar: Restarting Jenkins CI for plugins update
  • 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
  • 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

2022-01-11

  • 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

  • 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
  • 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

2022-01-04

2022-01-03

  • 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
  • 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=592771"