You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(hashar: Gerrit: deleting mediawiki/services/service-template-node and mediawiki/services/service-runner (unused) # T231953)
imported>Stashbot
(TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye)
(715 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== 2019-09-10 ==
== 2022-06-28 ==
* 11:57 hashar: Gerrit: deleting mediawiki/services/service-template-node and mediawiki/services/service-runner (unused) # [[phab:T231953|T231953]]
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 08:03 hashar: Updated Phan jobs for [[phab:T232413|T232413]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/535507 phan: stop installing MediaWiki
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]


== 2019-09-09 ==
== 2022-06-27 ==
* 15:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535198  Shibboleth disable selenium tests
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 12:26 hashar: Reloading Zuul to add multiple extension dependencies
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
* 12:15 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535161 " FundraiserLandingPage requires EventLogging"


== 2019-09-06 ==
== 2022-06-24 ==
* 19:59 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/534872
* 20:52 taavi: added `denisse` as a member
* 19:51 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534862/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534863/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534864/
* 19:37 hashar: Reloading zuul for {{Gerrit|Id07a109accc88b138193463a8e7335a60e501910}} Add UniversalLanguageSelector to ExternalGuidance's dependencies


== 2019-09-05 ==
== 2022-06-23 ==
* 19:36 James_F: Zuul: Drop debian-glue-non-voting from cassandra-table-properties
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021


== 2019-09-04 ==
== 2022-06-22 ==
* 18:48 James_F: Zuul: Move quibble and mwgate basic test jobs from php70 to php72
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 16:49 James_F: Zuul: Replacing basic quibble HHVM test with PHP72 https://gerrit.wikimedia.org/r/533973
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 16:25 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534481/
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 12:20 hashar: Dropped Phan jobs for extensions from release branches  [[phab:T226945|T226945]]
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2019-09-03 ==
== 2022-06-21 ==
* 21:59 James_F: Zuul: Move ORES and CirrusSearch to codehealth [[phab:T224990|T224990]]
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 20:47 James_F: Zuul: Suspend trigger-mediawiki-pipeline-dev for now [[phab:T218360|T218360]]
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 15:57 James_F: Zuul: Enable the publish pipeline for mediawiki/services/kartotherian  [[phab:T223275|T223275]]
* 15:36 James_F: Zuul: Add phan dependencies for MachineVision
* 14:34 James_F: Zuul: And wmf/ branches for VisualEditor core.
* 14:21 James_F: Zuul: Drop EducationProgram from dependencies
* 09:17 hashar: Reloaded Zuul for  https://gerrit.wikimedia.org/r/533916 "Apply REL based pipelines"
* 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533905/ "Remove wmf pipelines from non mediawiki repos"
* 08:32 hashar: err
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533909/ "zuul: add pipelines for MediaWiki releases"


== 2019-09-02 ==
== 2022-06-20 ==
* 17:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533885/ zuul: introduce test-wmf pipeline
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 17:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533880/ "zuul: rename gate-and-submit-swat to use -wmf"
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533536/ "zuul: drop skip-if for fundraising/REL branch"
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 08:22 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533532/ "zuul: pipelines for fundraising branches"


== 2019-08-31 ==
== 2022-06-17 ==
* 16:20 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/533628/
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 12:28 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 12:22 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} T 231696
* 12:18 hauskatze: github: created https://github.com/wikimedia/operations-debs-prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]


== 2019-08-30 ==
== 2022-06-16 ==
* 18:28 James_F: Zuul: Move Popups and MobileFrontend to codehealth
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 15:40 Krinkle: Updated "Prod Error" task form to have Title at the top and Trace below Description.
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 10:53 hashar: deployment-mediawiki-07: nuke /mnt/mediawiki  , last touched in April 2018 and using space on /
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]


== 2019-08-29 ==
== 2022-06-15 ==
* 23:23 James_F: Zuul: Move all archived repos to their own section at the end (no-op)
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 18:38 James_F: Zuul: Adding Wikibase back into dependencies for MachineVision
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 15:47 James_F: Zuul: Drop Wikibase dependency from MachineVision
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
* 02:32 Krinkle: Delete erroneous branch 'REL1_34-5' ({{Gerrit|108cb95c57a6e02b8}}) from mediawiki/extensions/SoftRedirector. ref https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/


== 2019-08-28 ==
== 2022-06-14 ==
* 23:00 James_F: Zuul: add tests for the labs/tools/maintain-kubeusers repo [[phab:T228499|T228499]]
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 23:00 James_F: Zuul: [TimedMediaHandler] Run phan job [[phab:T224766|T224766]] (merged but not deployed?)
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 21:08 thcipriani: restarting zuul server since it's out-of-sync with reality
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 16:11 hashar: Upgraded Jenkins for security update (2.176.3)
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production
* 16:08 hashar: integration:  docker container prune -f ; docker image prune -f
* 00:07 James_F: JJB: Delete mwext-npm-doc-publish, no longer used
* 00:07 James_F: Zuul: Migrate MobileFrontend to extension-javascript-documentation [[phab:T230831|T230831]]


== 2019-08-27 ==
== 2022-06-13 ==
* 20:25 James_F: Zuul: TwnMainPage: switch to extension-quibble-composer-nohhvm [[phab:T231289|T231289]]
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore


== 2019-08-26 ==
== 2022-06-12 ==
* 22:31 James_F: Zuul: Move Popups to extension-javascript-documentation
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 22:12 James_F: Zuul: Move MinervaNeue to extension-javascript-documentation
* 18:23 James_F: Zuul: [MinervaNeue, MobileFrontend, Popups] Add extension-javascript-documentation template
* 15:35 hashar: deployment-mwmaint01: /var/cache/hhvm/cli.hhbc.sq3 (1.3GB) - [[phab:T161598|T161598]]
* 13:51 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/532379


== 2019-08-25 ==
== 2022-06-10 ==
* 21:27 Reedy: centralauth.globalblocks `ALTER TABLE /*_*/globalblocks MODIFY gb_reason varbinary(767) NOT NULL;
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 16:43 Urbanecm: Revert 'Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]'
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 16:33 Urbanecm: Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check


== 2019-08-23 ==
== 2022-06-09 ==
* 11:12 Urbanecm: deployment-prep: run scap sync to ensure everything is deployed ([[phab:T231058|T231058]])
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 07:12 rxy: Applied SQL queries per [[phab:T231058#5433197]]
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration


== 2019-08-21 ==
== 2022-06-08 ==
* 15:49 James_F: Zuul: Add phan for LiquidThreads
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 08:41 hashar: Build docker-registry.discovery.wmnet/releng/quibble-coverage:0.0.34-4  # [[phab:T230423|T230423]]
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134


== 2019-08-20 ==
== 2022-06-06 ==
* 15:55 James_F: Zuul: Update Shreyas Minocha's whitelist e-mail
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 15:45 urandom: creating Parsoid/PHP storage schema in deployment-prep -- [[phab:T230792|T230792]]
* 09:39 James_F: Zuul: [kartotherian] Disable pipeline on gate as it doesn't pass [[phab:T223275|T223275]]


== 2019-08-19 ==
== 2022-06-02 ==
* 18:43 thcipriani: git -C /srv/mediawiki-staging/php-master/extensions/ConfirmEdit checkout maintenance/GenerateFancyCaptchas.php
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 11:26 hashar: Restarting Jenkins on contint2001
* 11:19 hashar: Restarting Jenkins on releases1002


== 2019-08-15 ==
== 2022-05-31 ==
* 12:13 James_F: Zuul: Make GrowthExperiments depend on Echo for non-Phan too.
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 10:50 James_F: Zuul: Make GrowthExperiments depend on Echo
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 09:05 James_F: Zuul: Kartotherian pipeline enablement [[phab:T223275|T223275]]
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2019-08-14 ==
== 2022-05-30 ==
* 18:40 hauskatze: github: Created https://github.com/wikimedia/operations-debs-prometheus-ipsec-exporter && forced replication to mirror changes # [[phab:T230443|T230443]]
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 18:29 hauskatze: Created new gerrit project: `operations/debs/prometheus-ipsec-exporter.git` # [[phab:T230443|T230443]]
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]
* 14:42 James_F: Move all BlueSpice extensions to not run HHVM [[phab:T230482|T230482]]


== 2019-08-13 ==
== 2022-05-28 ==
* 22:56 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/530012
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]


== 2019-08-08 ==
== 2022-05-27 ==
* 17:14 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/529108
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 03:18 thcipriani: cancelled stuck beta-scap-eqiad job to unblock postmerge
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2019-08-07 ==
== 2022-05-26 ==
* 16:00 thcipriani: restarting jenkins for updates
* 18:33 dancy: Updated Jenkins beta-* job configs
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch


== 2019-08-06 ==
== 2022-05-25 ==
* 21:24 James_F: Zuul: [TheWikipediaLibrary] Enable plan
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 21:05 James_F: Zuul: [TheWikipediaLibrary] Enable phan-seccheck
* 15:46 dancy: Restarted apache2 on gerrit1001
* 21:01 James_F: Zuul: [TheWikipediaLibrary] Add phan dependencies
* 19:26 paladox: ssh -p 29418 gerrit.wikimedia.org replication start --url github mediawiki/core --wait --now
* 17:09 accraze: deploying ores {{Gerrit|d08fa628aacb82529dbb4be357b68dd55c15fdee}}
* 11:42 paladox: ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --wait


== 2019-08-05 ==
== 2022-05-24 ==
* 22:24 accraze: deploying ores {{Gerrit|4270244d4a54c520f581dd33b312aa52f9a4c736}}
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 19:22 James_F: Zuul: [mediawiki/services/chromium-render] Enable pipeline testing and publishing [[phab:T217114|T217114]]
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update


== 2019-08-04 ==
== 2022-05-23 ==
* 12:11 hauskatze: gerrit: ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/OATHAuth --wait # [[phab:T229756|T229756]]
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2019-08-02 ==
== 2022-05-22 ==
* 17:30 James_F: Zuul: [operations/mediawiki-config] Stop running php70 jobs"
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/


== 2019-08-01 ==
== 2022-05-21 ==
* 20:19 Krinkle: Resolve Jenkins admin warning about legacy token by revoking it from the only user id "<unknown>" that seems to have one
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 19:00 James_F: Zuul: [BlueSpiceExtendedSearch] Disable HHVM tests, this doesn't work there
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]


== 2019-07-31 ==
== 2022-05-20 ==
* 19:20 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$  mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki ([[phab:T225056|T225056]])
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
* 18:27 James_F: Zuul: Move three repos to generic tox-docker job
* 17:50 James_F: Zuul: Move cassandra-table-properties to use cassandra-table-properties-tox-docker
* 16:27 James_F: Zuul: Archive SemanticTitle [[phab:T229422|T229422]]
* 15:55 James_F: Docker: Pushing operations-puppet:0.5.4
* 15:46 James_F: Docker: Pushing quibble-coverage:0.0.34-3
* 14:21 andrewbogott: moving deployment-sca02 to cloudvirt1030
* 12:59 andrewbogott: moving deployment-elastic05, deployment-kafka-main-2, deployment-mx02, deployment-webperf11 to new cloudvirts


== 2019-07-30 ==
== 2022-05-19 ==
* 21:17 James_F: Docker: Push operations-puppet:0.5.3
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 16:17 James_F: Docker: Pushing quibble-coverage:0.0.34-2
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already
* 15:39 cdanis: manually set dbctl etcd key to dummy value: cdanis@deployment-etcd-01:~$ etcdctl -u root -C http://127.0.0.1:2378 set /conftool/v1/mediawiki-config/eqiad/dbconfig
* 14:44 James_F: Docker: Pushing quibble-coverage:0.0.34-1


== 2019-07-29 ==
== 2022-05-18 ==
* 18:23 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Make PHP72 job voting, drop HHVM one
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:13 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Provide experimental PHP72 job to replace HHVM one
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]


== 2019-07-27 ==
== 2022-05-17 ==
* 20:38 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-SemanticHighcharts refs. [[phab:T199421|T199421]]
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]


== 2019-07-26 ==
== 2022-05-16 ==
* 21:10 legoktm: deploying https://gerrit.wikimedia.org/r/525871
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 18:38 James_F: Zuul: operations/software/cassandra-table-properties initial jobs
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 16:02 hauskatze: Deleted https://github.com/wikimedia/mediawiki-extensions-Copyvio {{!}} refs. [[phab:T228087|T228087]]
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809
* 16:00 James_F: Zuul: [Copyvio] Mark as archived for CI [[phab:T228087|T228087]]
* 15:49 Krinkle: Fix Jenkins Console output section "Selenium" to account for Quibble 0.34 changes (multiple Selenium sections, one implicitly closing the other)
* 15:45 James_F: doc1001: sudo -u doc-uploader rm -rf /srv/docroot/org/wikimedia/doc/cover-extensions/Copyvio/ # [[phab:T228087|T228087]]
* 04:43 James_F: Zuul: [mediawiki/libs/LangConv] Make this PHP7.2+ only
* 04:36 James_F: Zuul: Add testing for mediawiki/libs/LangConv


== 2019-07-25 ==
== 2022-05-14 ==
* 23:12 hashar: Purging all docker images on Jenkins instances: sudo cumin --trace --force 'name:docker' 'docker images{{!}} awk "{ print \$3}"{{!}}xargs docker rmi'
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 22:52 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for  https://gerrit.wikimedia.org/r/525673
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 22:30 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525673
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
* 21:28 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 (second attempt) for [[phab:T229020|T229020]]  using https://gerrit.wikimedia.org/r/#/c/integration/config/+/525570/
* 21:00 James_F: Zuul: Add phan to EventBus
* 18:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525608 - [[phab:T227004|T227004]])
* 17:38 greg-g: Added Jforrester to the Release Engineering group in Gerrit
* 17:17 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for [[phab:T229020|T229020]]
* 16:56 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525591
* 16:54 hashar: Tagged Quibble 0.0.34 {{Gerrit|ff8d86ffca97d17521f96d534f1e43780e6a22d5}} # [[phab:T229020|T229020]]
* 16:41 James_F: Zuul: [analytics-wikistats2] Drop node 6, switch to node10-browser-docker [[phab:T211784|T211784]]
* 16:20 hashar: Rolling back Quibble based jobs to 0.0.32 :-(
* 15:38 hashar: Upgrading Jenkins jobs to Quibble 0.0.33
* 15:09 hashar: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/525559
* 14:52 hashar: Tagged Quibble 0.0.33 {{Gerrit|b2f9e36}}  {{!}} [[phab:T193824|T193824]] [[phab:T87781|T87781]] [[phab:T199116|T199116]]


== 2019-07-24 ==
== 2022-05-13 ==
* 20:00 James_F: Zuul: Mark VIKI, VikiSemanticTitle, and VikiTitleIcon as broken.
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 10:55 hauskatze: Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 10:54 hauskatze: deployment-prep Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2019-07-23 ==
== 2022-05-12 ==
* 17:01 James_F: Zuul: [trigger-mediawiki-pipeline-dev] Only trigger on master
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 16:00 hashar: gerrit: adding Jakob_WMDE to wmf-deployment group ( https://gerrit.wikimedia.org/r/#/admin/groups/21,members ) # [[phab:T227193|T227193]]
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 08:00 hashar: Restore puppet class contint::packages::javascript on permanent slave ( reverting https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524221/ )  # [[phab:T228639|T228639]]
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2019-07-22 ==
== 2022-05-11 ==
* 22:43 James_F: Zuul: [wikimedia/iegreview] Adding mediawiki-i18n-check-docker template
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 10:49 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master/extensions$ sudo rm -rf Wikidata
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster


== 2019-07-19 ==
== 2022-05-10 ==
* 19:11 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/524565
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377


== 2019-07-18 ==
== 2022-05-09 ==
* 22:18 James_F: Zuul: Archive maps/kartotherian and maps/tilerator
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 21:17 mutante: rebooting gerrit server
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 19:18 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.1
* 19:03 James_F: Zuul: [analytics-mediawiki-storage] Drop node 6, switch to node10-browser-docker
* 18:25 mutante: contint2001 - puppet run reinstalled python-dateutil and ruby-safe-yeaml, stopped zuul service but no more failures now
* 18:25 mutante: contint2001 /etc/init.d/zuul-merger stop; apt install zuul (this time it worked to configure zuul after also stopping zuul-merge) ; apt-get autoremove (removed unused python packages); puppet agent -tv
* 18:22 mutante: contint2001 /usr/sbin/dpkg-reconfigure: zuul is broken or not fully installed
* 18:22 mutante: contint2001 systemctl stop zuul ; apt install zuul ... still fails
* 18:10 James_F: Zuul: [wikimedia/services/kartotherian] Switch to custom job [[phab:T228170|T228170]]
* 18:06 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.0
* 13:51 hashar: integration: in hiera project config, remove  classes: [ contint::packages::apt ]  {{!}} [[phab:T225735|T225735]]
* 13:48 hashar: integration: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523150/  for apt unattended_upgrades {{!}} [[phab:T225735|T225735]]
* 13:37 hashar: integration: rm /etc/apt/sources.list.d/sury-php.list  and remove php packages {{!}} https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523148/ {{!}} [[phab:T225735|T225735]]


== 2019-07-17 ==
== 2022-05-08 ==
* 16:49 James_F: Zuul: [PageForms] Add phan dependencies on AdminLinks, Cargo, PageSchemas [[phab:T228155|T228155]]
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 16:48 James_F: Zuul: Register initial tests for eventgate-wikimedia [[phab:T226668|T226668]]
* 16:08 hashar: Jenkins: updating wikidata-query-gui-build to get rid of slave-scripts/bin/npm-setup.sh which requires an obsolete/non existing tmpfs  # [[phab:T227818|T227818]]
* 09:48 hashar: Reloaded Zuul [Convert2Wiki] Archive extension
* 00:42 James_F: Docker: Deploy ci-buster and lintr:0.2.0
* 00:04 James_F: Docker: Complete rebuild finished and published; only took 2.5 hours. But now [[phab:T228196|T228196]] is fixed elsewise anyway, oh well.


== 2019-07-16 ==
== 2022-05-06 ==
* 21:58 hauskatze: GitHub: deleted `wikimedia/mediawiki-extensions-Convert2Wiki` refs. [[phab:T228198|T228198]]
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]
* 21:29 James_F: Docker: Publishing a whole new set of RelEng images for [[phab:T228196|T228196]]
* 20:25 James_F: Docker: Running a general rebuild for all missing RelEng images [[phab:T228196|T228196]]
* 16:49 James_F: Zuul: [ContentTranslation] Adding CentralAuth to phan dependencies
* 16:48 James_F: Zuul: Adding first tasks for mediawiki/services/kartotherian [[phab:T228170|T228170]]
* 15:19 James_F: Ran manual scap pull on deployment-deploy01
* 15:10 James_F: beta-scap-eqiad failing because mw-config repo is out of date?
* 14:57 James_F: Taking deployment-deploy01 offline
* 14:50 James_F: Jenkins deadlock? Beta jenkins jobs haven't successfully run for ~20 hours.


== 2019-07-15 ==
== 2022-05-05 ==
* 22:40 James_F: Zuul: [CentralAuth] Add phan [[phab:T189371|T189371]]
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 18:05 James_F: Zuul: [MinervaNeue] Make phan voting [[phab:T133664|T133664]]
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 17:00 James_F: Zuul: [GrowthExperiments] Add MobileFrontend to phan dependencies
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 16:49 James_F: Zuul: [MinervaNeue] Add phan as non-voting [[phab:T133664|T133664]]
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 16:04 Urbanecm: Run urbanecm@deployment-deploy01:~$ mwscript migrateUserGroup.php --wiki=commonswiki Image-reviewer image-reviewer ([[phab:T216406|T216406]])
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 15:59 James_F: Zuul: Adding Echo and MobileFrontend dependencies to skins/MinervaNeue [[phab:T133664|T133664]]
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 15:38 James_F: Zuul: [SecurePoll] Make seccheck voting [[phab:T205972|T205972]]
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 15:37 James_F: Zuul: [CentralAuth] Add seccheck [[phab:T189227|T189227]]
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 13:30 hashar: Deleting compiler1003.puppet-diffs.eqiad.wmflabs .. insanely slow for some reason (was on cloudvirt1002) # [[phab:T221969|T221969]] )
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 12:55 hashar: Creating compiler1003.puppet-diffs.eqiad.wmflabs [172.16.2.46] m1.large / 4vCPUs # [[phab:T221969|T221969]]
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins builds from https://integration.wikimedia.org/ci/view/Beta/


== 2019-07-14 ==
== 2022-05-04 ==
* 00:55 Krinkle: krinkle@doc1001: Remove https://doc.wikimedia.org/cover-extensions/$DOC_BASENAME/ - a failed build for GrowthExperiments
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2019-07-12 ==
== 2022-05-03 ==
* 23:07 James_F: Zuul: Instead of hacking around for WebAuthn, define bespoke quibble php72+ template
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 22:10 James_F: Zuul: Make seccheck voting on WebAuthn [[phab:T226737|T226737]]
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 19:28 James_F: Zuul: Add MobileFrontend and MinervaNeue to Wikibase's dependencies
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 16:56 James_F: Zuul: Add Flow as phan dependency to GrowthExperiments
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 00:32 James_F: Zuul: Migrate from php70-phan-seccheck to php72-phan-seccheck [[phab:T226420|T226420]]
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2019-07-11 ==
== 2022-05-02 ==
* 23:33 James_F: Pushing docker mediawiki-phan-seccheck:0.4.0
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 21:34 James_F: Pushing new php-based docker images with php-bcmath extension for [[phab:T227043|T227043]]
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]
* 20:30 mutante: add project member cdanis
* 18:57 Krinkle: Fix "setup quibble prepare" Console Section pattern in Jenkins to account for changes in 'mediawiki-quibble-composertest-php72-docker' job
* 18:22 James_F: Zuul: Archive ZeroBanner and ZeroPortal; drop from dependencies
* 17:50 James_F: jforrester@doc1001:/srv/docroot/org/wikimedia/doc/cover-extensions$ rm -rf WikibaseQuality/ # [[phab:T205065|T205065]]
* 16:15 James_F: Zuul: Archiving WikibaseQuality
* 09:48 hashar: jenkins: add more log details to hudson.model.Queue (FINER > FINEST) https://integration.wikimedia.org/ci/log/Jenkins%20Queue/configure  # [[phab:T72597|T72597]]
* 09:06 hashar: beta cluster jobs are dead locked. Taking a thread dump in case it helps figure out what is going on. [[phab:T72597|T72597]]


== 2019-07-10 ==
== 2022-04-29 ==
* 21:33 James_F: Zuul: Fix gerrit-docker pointer
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 18:27 James_F: Zuul: Adding phan to StructuredDiscussions [[phab:T224759|T224759]]
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]
* 17:55 James_F: Zuul: Disabling mwgate-rake for mediawiki/extensions/CentralNotice
* 13:30 hashar: on zuul-merger: rm /srv/zuul/git/operations/software/gerrit  due to [[phab:T157818|T157818]]  # [[phab:T189549|T189549]]


== 2019-07-09 ==
== 2022-04-28 ==
* 18:47 James_F: Zuul: Move mediawiki-quibble-composertest from php70 to php72
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 18:18 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move back to php72 as primary
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:08 James_F: Push docker composer-package-php72:0.2.2-s2
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 17:21 James_F: Push docker mediawiki-phan-seccheck:0.3.6-2
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC
* 16:18 James_F: Push docker image +mediawiki-phan-seccheck:0.3.6-1
* 15:53 James_F: Pushing docker images php-ast:1.0.1-2 and mediawiki-phan-seccheck:0.3.6
* 15:29 thcipriani: contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to [[phab:T157818|T157818]]
* 15:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/520437
* 14:08 James_F: Pushing mediawiki-phan-seccheck:0.3.6 docker image.
* 09:48 hashar: contint1001: removing /var/lib/docker , no more needed since we now use /mnt/docker # [[phab:T207707|T207707]]
* 00:04 James_F: Pushing php-ast:1.0.1-1 and related composer-package* docker image updates


== 2019-07-08 ==
== 2022-04-27 ==
* 23:54 James_F: Zuul: Move wikimania-scholarships from php56 to php72
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 23:35 James_F: Dropping mwext-MobileFrontend-npm-run-lint-modules-docker from Zuul [[phab:T224997|T224997]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029
* 21:39 thcipriani: reenable jenkins jobs service-pipeline-test and service-pipeline-test-and-publish following docker mount move
* 21:27 thcipriani: docker-pkg failed building docker-registry.discovery.wmnet/releng/composer-php56:0.1.5
* 21:18 thcipriani: fab deploy_docker to pull down ci docker-pkg images (I hope)
* 21:15 thcipriani: restart docker on contint1001 to pickup config change for https://gerrit.wikimedia.org/r/520738/
* 21:14 thcipriani: disabled jenkins jobs service-pipeline-test and service-pipeline-test-and-publish pre-contint1001 docker restart
* 13:38 hashar: Build docker-registry.wikimedia.org/releng/bazel:0.1.0  # [[phab:T189549|T189549]]


== 2019-07-06 ==
== 2022-04-26 ==
* 11:20 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --wiki=deploymentwiki {{!}} [[phab:T209565|T209565]]
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 11:09 hauskatze: beta cluster: maurelio@deployment-deploy01 foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --dry-run {{!}} [[phab:T209565|T209565]]
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2019-07-05 ==
== 2022-04-25 ==
* 21:17 James_F: Adding node10-docker for react.ooui
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 19:08 James_F: Pushing java8-sonar-scanner:0.4.11
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI


== 2019-07-03 ==
== 2022-04-20 ==
* 21:01 accraze: deploying ores {{Gerrit|676f7ba43c23e7b6dfa20b3b3c03924cb49acb90}}
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service
* 13:12 Urbanecm: deployment-prep Manually ran sudo -u jenkins-deploy scap sync-file wmf-config/CommonSettings-labs.php 'Not synced yet?' to sync file that's mysteriously not synced, but present on deployment host


== 2019-07-02 ==
== 2022-04-18 ==
* 20:34 James_F: Zuul: Adding quibble to the PopupAlerts, UnblockMe and Scribe extensions
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.


== 2019-07-01 ==
== 2022-04-14 ==
* 23:24 James_F: Pushing quibble-coverage:0.0.32-1 image
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 22:30 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 20:40 James_F: Rebuilt codehealth jjbs using node10-test with https://gerrit.wikimedia.org/r/519652
* 18:49 James_F: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental
* 18:43 James_F: Reloading Zuul for mediawiki/tools/phan/SecurityCheckPlugin php72 tests


== 2019-06-30 ==
== 2022-04-12 ==
* 17:03 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/519793
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141


== 2019-06-28 ==
== 2022-04-08 ==
* 08:58 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/519603
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 00:07 James_F: Restoring php-gmp in quibble docker images


== 2019-06-27 ==
== 2022-04-07 ==
* 21:27 James_F: Pushing new docker php base images with gmp, and all their descendants
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 17:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519455
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]
* 17:04 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519454
* 15:29 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424 (try #2)
* 15:22 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424


== 2019-06-26 ==
== 2022-04-06 ==
* 17:57 James_F: Adding the experimental helm-lint job to deployment-charts [[phab:T216049|T216049]]
* 20:03 thcipriani: rebooting phabricator
* 17:55 James_F: Pushing initial helm-linter docker image for [[phab:T216049|T216049]]
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]
* 01:29 James_F: Moving labs/tools/stewardbots's composer test to php72+ [[phab:T226585|T226585]]


== 2019-06-25 ==
== 2022-04-04 ==
* 22:00 hauskatze: Ran `replication start operations/debs/coredns --wait`
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 21:07 hauskatze: gerrit: Created new repo: operations/debs/coredns - requested by fsero on mediawiki
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 19:11 Krinkle: Update Jenkins collapsible section patterns to account for "quibble.cmd" => "quibble.commands" changing in the output
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 15:58 James_F: Adding CentralAuth to FileImporter required extensions in Zuul
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 15:51 awight: restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 15:50 awight: deployment-prep restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 14:39 hauskatze: Ran `replication start operations/debs/file-read-backwards --wait` on gerrit.wikimedia.org refs. [[phab:T226449|T226449]]
* 08:54 jnuche: redeploying Zuul
* 14:34 hauskatze: github: created https://github.com/wikimedia/operations-debs-file-read-backwards mirror refs. [[phab:T226449|T226449]]
* 14:14 hauskatze: gerrit: created `operations/debs/file-read-backwards.git` refs. [[phab:T226449|T226449]]
* 11:42 hashar: Deleting Jenkins jobs puppet-cdh-rake-docker and puppet-cdh4-rake-docker # [[phab:T226474|T226474]]


== 2019-06-24 ==
== 2022-04-02 ==
* 23:31 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
* 22:08 James_F: Zuul: Restore graphoid deploy repo tests, and comment on change-propagation ones.
* 22:02 James_F: Pushed graphoid-deploy-npm-node-6-docker into jjb
* 22:01 James_F: Re-publishing npm-test-graphoid, whoops.
* 15:17 James_F: Reloading Zuul to drop wikiba.se-composer-php56-docker [[phab:T224905|T224905]]
* 13:04 hashar: Updating git mirrored repos: integration-cumin:~$ sudo cumin --trace --force 'name:docker' 'find /srv/git -name "*.git" -exec git -C {} fetch --prune \;'
* 12:00 hashar: Updating Quibble based jobs to use version 0.0.32
* 10:09 hashar: Tag Quibble 0.0.32 @ {{Gerrit|c0fe6eb}} # [[phab:T211701|T211701]] [[phab:T218357|T218357]] [[phab:T220199|T220199]] [[phab:T223752|T223752]]
* 09:50 hashar: Building container releng/zuul-cloner:0.2.0 and switching jobs # [[phab:T224908|T224908]]
* 08:25 hashar: Removing mysql-server from the CI slaves # https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517095/ {{!}} [[phab:T225735|T225735]]


== 2019-06-22 ==
== 2022-03-31 ==
* 11:36 hauskatze: github: deleted `wikimedia/mediawiki-extensions-FirstSteps` GitHub mirror refs. [[phab:T226149|T226149]]
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
* 00:31 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/518344


== 2019-06-21 ==
== 2022-03-29 ==
* 20:52 Krinkle: Add "post-build scripts" Jenkins Console section to collapse most unrelated output and cascading failures from artefact readers etc. which are confusing when looking for "the" error
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 20:48 Krinkle: Fix Jenkins Console Section pattern for "setup quibble mw-install" to not eat all of output from phan (differentiate from phpunit, composer-test and phan). regexcomplexity++
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 16:58 James_F: Updated all four codehealth jjb jobs to use java8-sonar-scanner 0.4.10
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
* 16:55 James_F: Docker: Publishing java8-sonar-scanner 0.4.10.
* 06:37 legoktm: deployed https://gerrit.wikimedia.org/r/518205 https://gerrit.wikimedia.org/r/518206
* 06:34 legoktm: deployed https://gerrit.wikimedia.org/r/518204
* 06:04 legoktm: deleted remexhtml-phpunit-coverage-docker-publish jenkins job
* 05:51 legoktm: deployed https://gerrit.wikimedia.org/r/518195 https://gerrit.wikimedia.org/r/518196


== 2019-06-20 ==
== 2022-03-28 ==
* 22:03 James_F: Pushing new java8-sonar-scanner:0.4.9 image.
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 21:29 James_F: Pushing new java8-sonar-scanner:0.4.8 image
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 20:06 James_F: Add new mwcore-codehealth-* config to Zuul
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 20:04 James_F: Pushed new jjb mwcore-codehealth-* jobs and updated mwext-codehealth-* ones
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 19:54 James_F: Pushing new java8-sonar-scanner:0.4.7 image
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 17:45 James_F: Updated JJB definitions for mwext-php7\d-phan(-seccheck)?-docker for new docker-setup-mwext-for-phan macro.
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]
* 17:35 James_F: Updated JJB definitions for wikibase-{client{{!}}repo}-docker to use php72 not php70.
* 17:12 James_F: Updated JJB definitions for all six coverage jobs
* 15:09 hashar: Nuking castor caches for Quibble jobs  integration-castor03 : rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/*quibble*/{composer,npm}
* 14:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/518033/
* 14:16 hashar: building some container for codehealth pipeline  https://gerrit.wikimedia.org/r/518034


== 2019-06-19 ==
== 2022-03-27 ==
* 08:58 hashar: Removed Debian packages for python / tox etc from all slaves {{!}} [[phab:T225735|T225735]]  https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517092/
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]
* 08:46 hashar: Removed Debian package colordiff  from all slaves {{!}} [[phab:T225735|T225735]] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517091/
* 03:43 legoktm: deploying https://gerrit.wikimedia.org/r/517785 https://gerrit.wikimedia.org/r/517786


== 2019-06-18 ==
== 2022-03-26 ==
* 23:19 James_F: Making extension-seccheck non-voting on all 195 repos, hopefully temporarily. [[phab:T226064|T226064]]
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 22:14 Reedy: Apply https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OATHAuth/+/496451/53/sql/mysql/patch-add_generic_fields.sql to centralauth on beta [[phab:T218210|T218210]]
* 21:59 James_F: Manually pruned some old quibble-coverage Docker images from contint1001.
* 21:51 James_F: Updating docker-pkg files on contint1001 to fix quibble-stretch-php72
* 21:43 James_F: Zuul: Adding php72 variants of phan-seccheck jobs as experimental
* 21:35 James_F: Pushing docker image quibble-stretch-php72:0.3.3
* 20:40 James_F: Reload Zuul to point MediaWiki's phan jobs at php72 instead of php70 [[phab:T225456|T225456]]
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517733/
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]]
* 20:22 James_F: Reload Zuul to point Parsoid's experimental phan job at php72 instead of php70 [[phab:T225456|T225456]]
* 20:12 James_F: Reload Zuul to point extension-phan and skin-phan templates at php72 instead of php70 [[phab:T225456|T225456]]
* 19:51 hashar: building containers for php7.2 from SRE # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517684/
* 19:38 James_F: Bumped jjb mwext-codehealth-*, mediawiki-*coverage*docker*, mwext*coverage*docker* jobs to use quibble-coverage:0.0.31-4
* 19:28 James_F: Pushing quibble-coverage:0.0.31-4 docker image.
* 17:02 James_F: Updated jjb jobs for mwext-codehealth-patch and mwext-codehealth-master-non-voting.
* 16:54 James_F: Pushing new quibble-coverage docker image
* 11:32 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-PureWikiDeletion {{!}} [[phab:T225991|T225991]]


== 2019-06-15 ==
== 2022-03-25 ==
* 00:17 Krinkle: Disable "Publish/Notify" for mediawiki/libs/ObjectFactory.git in Diffusion –&nbsp;[[phab:T222409|T222409]], [[phab:T143162|T143162]]
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord


== 2019-06-14 ==
== 2022-03-24 ==
* 22:15 James_F: Dropped jjb jobs trending-edits?(-deploy)-npm-node-6-docker
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 22:14 James_F: Reloading Zuul to archive the trending-edits repos
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 21:50 James_F: Dropped docker-registry.discovery.wmnet/releng/hhvm-jessie:latest, unused
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 19:57 James_F: Ran `docker rmi docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-4` on contint1001; unused.
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
* 19:41 James_F: Zuul: Deploying Codehealth: Add Translate extension to pipeline
* 19:39 James_F: Zuul: Deploying Remove coverage ci step for extensions with codehealth
* 19:35 James_F: Zuul: Deploying Add sonar CodeHealth checks to Wikibase repos
* 18:44 James_F: Publishing image docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-5
* 16:58 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/517087
* 16:08 James_F: Updating docker-pkg to provide quibble-coverage on php7.2
* 16:04 James_F: Dropped docker-registry.discovery.wmnet/releng/quibble-stretch-php70:0.0.31-1 from contint1001, unused
* 15:50 James_F: Bumped jjb mediawiki-phpunit-coverage-patch-docker


== 2019-06-13 ==
== 2022-03-23 ==
* 22:30 James_F: Deploying quibble-stretch-php70 update.
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 22:17 James_F: Deploying quibble-coverage fix with chmod 0755 files.
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 21:49 James_F: Reverted JJB jobs mediawiki-core-code-coverage-docker and mediawiki-phpunit-coverage-patch-docker back to quibble-stretch-php70
* 21:43 James_F: Updated JJB job mediawiki-phpunit-coverage-patch-docker to point to quibble-coverage
* 21:42 James_F: Update JJB job mediawiki-core-code-coverage-docker to point to quibble-coverage
* 21:10 brennen: restarted jenkins agent on integration-trigger-01 to pick up new root configuration
* 20:54 hashar: ¡Learning to log one step at a time!
* 20:54 James_F: Updating docker-pkg files to create quibble-coverage image
* 20:40 hashar: deleting docker-registry.wikimedia.org/releng/hhvm-jessie-compile from contint1001 # [[phab:T224908|T224908]]
* 18:17 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/514016
* 17:05 brennen: changing jenkins remote root for integration-trigger-01 to /mnt/home/jenkins-deploy per [[phab:T224069|T224069]]
* 16:07 James_F: Dropped docker-registry.discovery.wmnet/releng/npm-test-mathoid:latest, unused
* 16:06 James_F: Dropped mathoid?(-deploy)-npm-node-6-docker from jjb
* 16:01 James_F: Dropped docker-registry.discovery.wmnet/releng/npm6:latest from contint1001, unused
* 15:33 Krinkle: Disable "Publish/Notify" for quibble.git in diffusion, [[phab:T143162|T143162]], [[phab:T196347|T196347]]
* 00:13 James_F: Dropped jjb jobs for (change-propagation{{!}}citoid{{!}}cxserver{{!}}graphoid)?(-deploy)-npm-node-6-docker


== 2019-06-12 ==
== 2022-03-22 ==
* 23:34 brennen: deleting and rebuilding integration-trigger-01 (again) to use jessie instead of stretch
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 23:20 James_F: Zuul: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945
* 22:44 James_F: Enabling service-pipeline-test on four services: 3d2png, chromium-render, recommendation-api, trending-edits
* 22:40 James_F: Dropped jjb jobs wikimedia-cz-tracker-npm-node-6-docker fundraising-dash-npm-node-6-docker wikimedia-portals-npm-browser-node-6-docker wikimedia-portals-deploy-npm-browser-node-6-docker
* 22:38 brennen: deleting and re-creating integration-trigger-01
* 22:22 James_F: Dropped mapdata-npm-node-6-docker and marvin-npm-node-6-docker from jjb
* 22:21 James_F: Reloading Zuul to make mpadata use generic-node10-browser-docker
* 21:54 James_F: Reloading Zuul to deploy, umm, 12 patches. 🤷🏽‍♂️
* 21:07 James_F: Updating Zuul with https://gerrit.wikimedia.org/r/516674 https://gerrit.wikimedia.org/r/516675 https://gerrit.wikimedia.org/r/516676
* 18:35 James_F: Reloaded Zuul to use node10 things for OOUI experimentally
* 18:32 James_F: Deployed new jjb job ooui-npm-special-node10-plus-composer-docker
* 16:28 Krinkle: Fix Jenkins Console section "setup quibble mw-install" pattern to not slurp the whole output on 'mediawiki-quibble-composertest' jobs
* 16:27 James_F: Reloading Zuul to add service-pipeline-test?(-and-publish) to mediawiki/services/mobileapps
* 00:49 James_F: Updated Zuul phan dependencies for ContentTranslation, adding GlobalPreferences


== 2019-06-11 ==
== 2022-03-21 ==
* 23:51 James_F: Updated Zuul phan dependencies for WikibaseLexeme, adding Wikibase [[phab:T225524|T225524]]
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 23:37 James_F: Dropped unused jjb jobs mwext-jsduck-publish mwext-VisualEditor-jsduck-docker, and mwext-VisualEditor-docker-publish
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]
* 23:34 James_F: Updating Zuul to drop extension-jsduck, no longer used (yay)
* 23:14 James_F: Reloading Zuul to have Wikibase use extension-javascript-documentation
* 22:39 James_F: PHP55 docker images deleted; needed to specify :0.… and also :latest because we're not just dropping one tag, we're dropping the double pair. All now done. [[phab:T224907|T224907]]
* 22:34 James_F: PHP55 docker images not deleted because they're still referenced (by what?).
* 22:25 James_F: Running `docker rmi` for quibble-jessie-php55 ({{Gerrit|938a4b4f6abb}}) composer-test-php55 ({{Gerrit|9731d99e7a6f}}) composer-package-php55 ({{Gerrit|906edee88f4f}}) php55-compile ({{Gerrit|01f3fa5ef542}}) composer-php55 ({{Gerrit|a14836c5fc62}}) and php55 ({{Gerrit|f8b5dc437239}})
* 22:22 James_F: Updating docker-pkg files on contint1001 to drop all php55 images [[phab:T224907|T224907]]
* 22:18 James_F: Dropped from jjb: mwgate-composer-php55-docker, composer-php55-docker, composer-package-php55-docker, php-compile-php55-docker
* 22:06 James_F: Drop unused jjb job integration-composer-check-php55-docker
* 22:06 James_F: Reloading Zuul to drop integration-composer-check-php55-docker from integration/composer
* 22:01 James_F: Reloading Zuul to add integration-composer-check-php72-docker to integration/composer
* 21:59 James_F: Push new jjb job integration-composer-check-php72-docker
* 21:56 James_F: Reloading Zuul to drop all *php55* jobs except integration-composer
* 21:49 James_F: Reloading Zuul to add composer-php56-docker to integration/docroot
* 21:45 James_F: Reloading Zuul to drop quibble*php55*
* 21:42 James_F: Reloading Zuul to add composer-php72-docker to labs/tools/heritage
* 21:07 brennen: creating integration-trigger-01 for [[phab:T224069|T224069]]
* 19:34 hashar: building docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.4.5 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/516511/
* 19:04 halfak: deployment-prep deploying ores {{Gerrit|04fbd58}}
* 19:03 mdholloway: deployment-prep created deployment-wikifeeds01, added role::beta::docker_services and service config
* 18:29 halfak: deployment-prep deploying ores {{Gerrit|86809a4}}
* 13:17 hashar: Building container docker-registry.wikimedia.org/releng/civicrm:0.1.0 # [[phab:T210287|T210287]]


== 2019-06-10 ==
== 2022-03-18 ==
* 19:50 James_F: Restarting zuul to drop quibble experimental php70-phan-docker jobs; php72 suffices
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 19:39 thcipriani: restarting jenkins
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 19:37 James_F: Restarting jenkins with `sudo service restart jenkins` on contint1001 due to out-of-space issue earlier.
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
* 19:10 Krinkle: Fix "setup quibble mw-install" collapsible section pattern in Jenkins administration, so that it doesn't match the "PHP 7.1+" warning from Phan. The pattern is meant to detect the first line of mw/maint/install.php output
* 19:01 thcipriani: cleaning contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 17:48 James_F: Reloading zuul to set zuul parameters for PHP 7.2 variants of phan jobs too
* 17:41 James_F: Reloading zuul to Add experimental mwext-php72-phan-docker jobs to quibble template
* 17:38 James_F: Pushed new mwext-php72-phan-docker with jjb
* 16:59 James_F: Deployed https://gerrit.wikimedia.org/r/516300
* 16:31 legoktm: deployed https://gerrit.wikimedia.org/r/516298
* 04:38 legoktm: deploying https://gerrit.wikimedia.org/r/516065 https://gerrit.wikimedia.org/r/516066


== 2019-06-08 ==
== 2022-03-17 ==
* 00:49 James_F: Updating docker-pkg files on contint1001 to build mediawiki-phan:0.2.0 based on php72 [[phab:T223847|T223847]]
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]


== 2019-06-07 ==
== 2022-03-16 ==
* 23:29 marxarelli: deploying zuul changes for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 23:28 marxarelli: updating jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 13:30 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/515058
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 00:10 James_F: jjb: Migrated docker-setup-mwext-for-phan to node10 quibble
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep


== 2019-06-06 ==
== 2022-03-15 ==
* 18:06 James_F: Reloading Zuul to drop EventBus dependency on CentralNotice for now [[phab:T225195|T225195]]
* 18:26 brennen: gitlab: removed most existing /people groups
* 15:12 hashar: reloading zuul for {{Gerrit|I044bedd41091b57a1a188ea243c4769a6e21babb}} [[phab:T222802|T222802]]
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 15:12 hashar: reloading Zuul for {{Gerrit|I8d6ea5b5491fe50d7144926ef8bc69d4704d4fab}} [[phab:T224301|T224301]]
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 08:13 hashar: Reloading Zuul for {{Gerrit|I764972711843645afd00e196a3bedd17730b4cbe}}  which drops  mwselenium-quibble-docker from Wikibase
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2019-06-05 ==
== 2022-03-14 ==
* 19:34 andrewbogott: moving deployment-imagescaler03 to cloudvirt1029
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 17:35 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 16:01 James_F: Reloading Zuul to deploy {{Gerrit|172a4e7886b285adb48a85dce7287dce5c3778e2}}
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 09:07 hashar: Pooled in integration-slave-docker-1058 and integration-slave-docker-1059
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2019-06-04 ==
== 2022-03-11 ==
* 20:39 James_F: Updating jjb PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [[phab:T220917|T220917]]
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
* 20:31 James_F: Updating docker-pkg files on contint1001 for quibble-stretch-php70 [[phab:T220917|T220917]]
* 17:44 James_F: Updating jjb PHP code coverage jobs to node10 quibble [[phab:T224983|T224983]]
* 17:31 James_F: Reloading Zuul to make mwext-MobileFrontend-npm-run-lint-modules-docker non-voting [[phab:T224997|T224997]]
* 16:10 hashar: Deleting  integration-slave-docker-1021 and integration-slave-docker-1049  / too small disk (20G partition) and not enough ram (2G) # [[phab:T221872|T221872]]
* 13:24 hashar: Update all selenium-daily* jobs to use NodeJS 10 instead of NodeJS 6. [[phab:T217545|T217545]]
* 13:01 hashar: Building docker-registry.discovery.wmnet/releng/node10-test-browser:0.6.0 # [[phab:T217545|T217545]]
* 11:32 hashar: Upgrading Jenkins Pipeline plugins
* 11:31 hashar: Upgrading Jenkins Warnings Next Generation Plugin # [[phab:T224745|T224745]]
* 11:30 hashar: Upgrading Jenkins BlueOcean and all its dependencies


== 2019-06-03 ==
== 2022-03-10 ==
* 19:32 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514089
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 19:29 James_F: Generated mwgate-node10-docker and deployed via jjb.
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 18:34 hashar: hswitch most Quibble jobs to node 10 [[phab:T222406|T222406]] - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ [[phab:T222406|T222406]]
* 20:03 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/768843
* 17:40 James_F: Reloading Zuul to switch most Quibble jobs to node 10 [[phab:T222406|T222406]]
* 15:12 hashar: updating Quibble jenkins jobs
* 15:58 hashar: Deleting integration-slave-docker-1055 and integration-slave-docker-1056 . CPU is way too slow [[phab:T223971|T223971]]
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 15:46 hashar: reduce number of executors on all integration slave docker, they are somehow starving on CPU and/or IO when lot of mediawiki builds are running in parallel
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 14:38 James_F: hashar and I are temporarily disabling running selenium tests in CI. See [[phab:T211784|T211784]] [[phab:T222406|T222406]] for more details.
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2019-06-01 ==
== 2022-03-09 ==
* 22:54 James_F: Reloading Zuul to re-add Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 21:14 James_F: Reloading Zuul to add phan for CentralNotice and add DannyS712 to whitelist


== 2019-05-31 ==
== 2022-03-08 ==
* 23:34 James_F: Reloading Zuul to remove Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 20:31 brennen: requiring 2fa for all users under /repos
* 22:12 James_F: Reloading Zuul to add phan for LdapAuthentication
* 21:18 James_F: Reloading Zuul to add dependencies for TimedMediaHandler, ReadingLists, JsonConfig, and FlaggedRevs.
* 21:13 James_F: Reloading Zuul to add phan for Collection, ContentTranslation, and Jade
* 19:43 James_F: Reloading Zuul to add phan for Math(s)
* 19:31 James_F: Reloading Zuul to add phan for MobileFrontend
* 19:20 James_F: Reloading Zuul to add phan for Sentry
* 18:35 James_F: Reloading Zuul to deploy CentralNotice dependency to EventBus
* 18:17 James_F: Reloading Zuul to deploy extra dependencies for MobileFrontend
* 18:10 James_F: Reloading Zuul to deploy phan for cldr and GlobalPreferences
* 18:02 James_F: Reloading Zuul to deploy {{Gerrit|I0ee1c8166}}
* 17:56 James_F: Reloading Zuul to deploy {{Gerrit|I9c115b1f6}} and {{Gerrit|I0d98f01d}}


== 2019-05-30 ==
== 2022-03-07 ==
* 23:19 James_F: Pruned releng/quibble-fresnel:0.0.31-3 {{Gerrit|8ca03484a2c3}} from contint1001 to save space
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]
* 23:02 James_F: Updating docker-pkg files on contint1001 for +quibble-fresnel (0.0.31-4)
* 21:33 Krinkle: Jenkins admin says ""Your Jenkins data directory /var/lib/jenkins (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full.""
* 21:27 greg-g: back in business (ugh)
* 21:21 greg-g: doing the zuul deadlock dance again
* 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/513168
* 02:48 legoktm: deploying https://gerrit.wikimedia.org/r/512474 https://gerrit.wikimedia.org/r/512507 https://gerrit.wikimedia.org/r/513153 https://gerrit.wikimedia.org/r/513157 https://gerrit.wikimedia.org/r/512729 https://gerrit.wikimedia.org/r/513206 https://gerrit.wikimedia.org/r/513214 https://gerrit.wikimedia.org/r/513217 https://gerrit.wikimedia.org/r/513218 https://gerrit.wikimedia.org/r/513211
* 02:35 legoktm: deployed https://gerrit.wikimedia.org/r/513129 https://gerrit.wikimedia.org/r/513071


== 2019-05-29 ==
== 2022-03-04 ==
* 21:46 greg-g: doing the executor deadlock dance: https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Jenkins_execution_lock
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 19:13 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/768068


== 2019-05-28 ==
== 2022-03-03 ==
* 12:43 paladox: delete extensions/3D repo in gerrit - [[phab:T224463|T224463]]
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 01:35 legoktm: deployed https://gerrit.wikimedia.org/r/512794
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2019-05-26 ==
== 2022-03-02 ==
* 18:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/512544
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04


== 2019-05-22 ==
== 2022-02-28 ==
* 07:21 hashar: Updating Jenkins job to have castor use rsync --delay-updates # [[phab:T203506|T203506]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/511672/
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin


== 2019-05-21 ==
== 2022-02-26 ==
* 22:05 marxarelli: generating new blubber-pipeline-* and trigger-blubber-pipeline-* jenkins jobs defined by https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 11:23 hashar: Depooling integration-slave-docker-1055 and integration-slave-docker-1056 : CPU is too slow # [[phab:T223971|T223971]]
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 08:24 hashar: Updated phan jobs to no more install mediawiki development dependencies (potentially conflicting with the extensions code) https://gerrit.wikimedia.org/r/#/c/integration/config/+/511447/ # [[phab:T223397|T223397]]
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases
* 07:29 legoktm: deployed https://gerrit.wikimedia.org/r/511627
* 07:10 legoktm: manually rebuilding mediawiki-core-code-coverage-docker with 5 hour timeout


== 2019-05-20 ==
== 2022-02-24 ==
* 13:11 hashar: updating phan jobs to use docker-registry.wikimedia.org/releng/mediawiki-phan:0.1.15 # [[phab:T219114|T219114]]
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 01:43 legoktm: deploying https://gerrit.wikimedia.org/r/511361
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2019-05-19 ==
== 2022-02-23 ==
* 15:43 hashar: Purging webperformance.integration.eqiad.wmflabs  /tmp directory (inodes full)
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 15:43 hashar: Purging all images on Docker CI slaves
* 20:10 dancy: Updating scap in beta
* 08:03 legoktm: deployed https://gerrit.wikimedia.org/r/511122
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/  # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2019-05-18 ==
== 2022-02-22 ==
* 18:14 Amir1: cherry-picking 511078 on puppetmaster
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 12:07 Reedy: reload zuul to deploy https://gerrit.wikimedia.org/r/510967
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])


== 2019-05-17 ==
== 2022-02-21 ==
* 22:55 paladox: created mediawiki/extensions/Scribe gerrit repo - [[phab:T223662|T223662]]
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 00:35 brennen: Updating dev-images docker-pkg files on contint1001
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 07:27 hashar: Refreshing all Jenkins jobs


== 2019-05-16 ==
== 2022-02-20 ==
* 09:36 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/composer-test-php72:0.1.0 # [[phab:T223428|T223428]]
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
* 07:35 hashar: integration-slave-jessie-1002: purging all php Debian packages and rerunning puppet. There is some package conflict somewhere :-(


== 2019-05-15 ==
== 2022-02-19 ==
* 11:55 hashar: Bring back https://integration.wikimedia.org/ci/computer/integration-castor03/ to restore the central cache behavior. In turn unblocking a wide range of builds
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 11:32 hashar: bringing back castor, integration-castor03 is back
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 09:35 hashar: Regenerating all CI jobs to disable castor saving entirely
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]
* 02:35 Krenair: Logged into deployment-sca0[12] as root and given them the correct nameservers to try to unbreak things. [[phab:T221654|T221654]]
* 01:00 thcipriani: /usr/local/sbin/keyholder arm


== 2019-05-14 ==
== 2022-02-18 ==
* 09:43 hashar: Upgraded all tox jobs to tox 3.10.0
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 09:19 hashar: Building docker containers releng/tox-*:0.4.0
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}


== 2019-05-13 ==
== 2022-02-17 ==
* 20:21 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/509930
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 19:47 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/502606
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* afk: updating docker-pkg images on contint1001 for https://gerrit.wikimedia.org/r/508019
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 14:48 hashar: if you build Docker containers, there is a long delay between it being build/published and it actually being available https://phabricator.wikimedia.org/T222210#5176863  known issue
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 13:49 hashar: Building docker image releng/tox:0.4.0
* 11:12 hashar: Bringing deployment-deploy03 back
* 10:37 hashar: Rolling CI config change https://gerrit.wikimedia.org/r/508512 which caused some patches to not be processed last week # https://wikitech.wikimedia.org/wiki/Incident_documentation/20190506-zuul / [[phab:T105474|T105474]]
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome


== 2019-05-10 ==
== 2022-02-16 ==
* 18:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/509483
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 18:08 Reedy: Reloading Zuul to deploy dependancies, new tests, phan
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 17:53 Reedy: Reloading Zuul to deploy 5 more phan patches
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 17:37 Reedy: Reloading Zuul to deploy 5 phan enabling patches
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 17:23 Reedy: Reloading Zuul to deploy various phan additions and one dependancy
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}
* 17:08 Reedy: Reloading Zuul to deploy patches adding phab
* 16:46 Reedy: Reloading Zuul to deploy various dependancy patches


== 2019-05-09 ==
== 2022-02-15 ==
* 08:17 elukey: remove mediawiki memcached nutcracker config from deployment-prep (should be unused) - [[phab:T214275|T214275]]
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 00:19 thcipriani: updating docker images on contint1001 for https://gerrit.wikimedia.org/r/508929
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 00:19 thcipriani: clean docker images on contint1001
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}


== 2019-05-07 ==
== 2022-02-14 ==
* 22:23 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/507871
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 10:57 hashar: Upgraded Zuul to 2.5.1-wmf8 # [[phab:T105474|T105474]] [[phab:T140297|T140297]]
* 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon


== 2019-05-06 ==
== 2022-02-12 ==
* 21:57 thcipriani: update docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508086
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 19:07 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/508012
* 18:37 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508370
* 17:44 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508036
* 13:31 hashar: Jenkins: installed Least Load plugin {{!}} [[phab:T218458|T218458]]


== 2019-05-03 ==
== 2022-02-10 ==
* 20:58 thcipriani: updating docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508006
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 20:58 thcipriani: clean docker images from contint1001
* 17:02 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/507992
* 09:22 hashar: removed zuul debian package from integration-castor03
* 08:30 hashar: Building Docker image releng/gradle:0.1.0  {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/507872/ {{!}} [[phab:T222199|T222199]]


== 2019-05-01 ==
== 2022-02-09 ==
* 17:53 halfak: deploying ores:52e9759
* 15:22 taavi: deleted shutoff deployment-mx02


== 2019-04-30 ==
== 2022-02-08 ==
* 18:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/494778
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 17:01 bearND: (beta): Update mobileapps to {{Gerrit|142ba30}}
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 02:01 hashar: Polled in integration-slave-docker-1055 and ntegration-slave-docker-1056
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
* 01:35 hashar: Deleting integration-slave-docker-1037 (bigram)  it is too slow for some reason # [[phab:T222023|T222023]]


== 2019-04-29 ==
== 2022-02-07 ==
* 14:04 godog: add dsharpe user
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550


== 2019-04-26 ==
== 2022-02-04 ==
* 13:33 Krenair: shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. [[phab:T218729|T218729]]
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622


== 2019-04-25 ==
== 2022-02-03 ==
* 16:35 Krenair: shutting down deployment-ms-fe02 and deployment-poolcounter04 [[phab:T218729|T218729]]
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]


== 2019-04-23 ==
== 2022-02-02 ==
* 15:33 paladox: merging the 2.15.13 release into stable-2.15 following https://wikitech.wikimedia.org/wiki/Gerrit#Update_our_repository
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 14:41 Krenair: Shut down deployment-ms-be03 and deployment-ms-be04 [[phab:T218729|T218729]]
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta
* 10:22 Amir1: ores:060fc37 going beta


== 2019-04-19 ==
== 2022-02-01 ==
* 19:56 mutante: phab1003 - editing /srv/deployment/phabricator/deployment-cache/.config manually to replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet to fix git cloning issue on first puppet run on new host where somehow tin.eqiad still shows up. fixes puppet run on [[phab:T221389|T221389]]
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]


== 2019-04-18 ==
== 2022-01-31 ==
* 19:25 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/504824
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]


== 2019-04-17 ==
== 2022-01-28 ==
* 17:17 andrewbogott: cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504580/ to move off of soon-to-be-shutdown dns recursors
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`


== 2019-04-16 ==
== 2022-01-27 ==
* 17:36 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf {{!}} tee [[phab:T221107|T221107]].php
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2  # [[phab:T299431|T299431]]
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins


== 2019-04-15 ==
== 2022-01-26 ==
* 10:16 Amir1: ores:8f01d40 going beta
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 08:51 hashar: castor: nuked /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwselenium-quibble-docker # [[phab:T220948|T220948]]
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2019-04-13 ==
== 2022-01-25 ==
* 21:05 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. [[phab:T91410|T91410]]
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 21:00 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul.
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 21:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503669
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 18:52 Krinkle: "Your JENKINS_HOME (/var/lib/jenkins) is almost full. "
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 18:16 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/503664
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 17:44 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/502807 (postgres php72)
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 00:06 Krenair: transferred /home from deployment-cache-upload04 to deployment-cache-upload05 and shut down old one
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 00:06 Krenair: transferred /home from deployment-cumin to deployment-cumin02 and shut down old one
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2019-04-12 ==
== 2022-01-24 ==
* 15:09 Krenair: upload traffic now through cache-upload05
* 21:44 Reedy: unstick beta ci jobs
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2019-04-11 ==
== 2022-01-22 ==
* 18:29 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/480463
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 18:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/499334
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
* 17:50 Reedy: deployed 'quibble-donationinterface-REL1_31-php70-docker' to jenkins
* 16:31 Krenair: deleting deployment-db04, unused and shut down since 28th March. [[phab:T219087|T219087]]
* 16:18 mateusbs17: deployment-chromium0[1-2] Update Proton on to {{Gerrit|8988283}} ([[phab:T213362|T213362]], [[phab:T216191|T216191]], [[phab:T212322|T212322]])


== 2019-04-10 ==
== 2022-01-21 ==
* 23:32 James_F: Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. [[phab:T220653|T220653]]
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 22:36 James_F: Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches.
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
* 20:27 paladox: create operations/software/gerrit/plugins/MassBranchCreation repository
* 14:16 hashar: contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 14:13 hashar: contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 13:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/502790
* 12:46 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502786
* 12:40 hashar: contint2001: stopped puppet and zuul-merger for debugging


== 2019-04-09 ==
== 2022-01-20 ==
* 23:08 Krinkle: Reloading Zuul to deploy https://phabricator.wikimedia.org/T220561
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 17:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502560
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 16:38 Krenair: deleting deployment-db03, unused since [[phab:T216635|T216635]] and shut down since 24th March. [[phab:T219087|T219087]]
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 16:21 bearND: (beta): Update mobileapps to {{Gerrit|3edfcad}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]]) - 3rd time is the charm
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2019-04-08 ==
== 2022-01-19 ==
* 22:46 hashar: cleaned docker images on integration-slave-docker-1030 and  integration-slave-docker-1043 :)
* 18:20 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 20:06 bearND: (beta): Update mobileapps to {{Gerrit|cdb9928}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]])
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 14:47 hashar: hard rebooting integration-slave-docker-1053 OOM / deadlocked
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]
* 13:39 hashar: Deleting integration-slave-docker-1045 again. It uses Stretch instead of Jessie
* 13:21 hashar: integration: fix cumin profiles that got renamed by {{Gerrit|9e0aa8264659799e74c3e815ea39c640e5f05393}}
* 10:15 hashar: Building Docker image  releng/quibble-stretch-php73:0.0.31-2 # [[phab:T220237|T220237]]


== 2019-04-07 ==
== 2022-01-18 ==
* 00:30 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501793
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0  with proper PHP version in the images # [[phab:T299389|T299389]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2019-04-06 ==
== 2022-01-17 ==
* 02:17 legoktm: rebuilding npm-php image again https://gerrit.wikimedia.org/r/501848
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 01:52 legoktm: rebuilding npm-php image https://gerrit.wikimedia.org/r/501847
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/501786 https://gerrit.wikimedia.org/r/501714 https://gerrit.wikimedia.org/r/501707 https://gerrit.wikimedia.org/r/501782 https://gerrit.wikimedia.org/r/501709 https://gerrit.wikimedia.org/r/500111 https://gerrit.wikimedia.org/r/500106 https://gerrit.wikimedia.org/r/500127 https://gerrit.wikimedia.org/r/500119
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
* 00:53 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/501813


== 2019-04-05 ==
== 2022-01-14 ==
* 23:53 Krinkle: Beta cluster puppetmaster is stalled behind origin/production as of 24 hours ago (57 patches behind) due to a local merge conflict
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 22:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501789
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 22:02 legoktm: rebuilding mediawiki-phan docker image https://gerrit.wikimedia.org/r/501794
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 20:11 thcipriani: updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/501465
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta
* 08:56 hashar: Reloaded Zuul for operations/software/gerrit/plugins/barricade https://gerrit.wikimedia.org/r/#/c/integration/config/+/501507/
* 08:21 legoktm: deploying https://gerrit.wikimedia.org/r/501416


== 2019-04-04 ==
== 2022-01-13 ==
* 23:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501442
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 22:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/500360
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 21:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501428
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 20:47 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/501407
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 17:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/500719 (
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2019-04-03 ==
== 2022-01-12 ==
* 06:13 hashar: gerrit: renamed group "scholarships" to "wikimedia-wikimania-scholarships". Made it owned by "Gerrit Managers" # [[phab:T218864|T218864]]
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/


== 2019-04-02 ==
== 2022-01-11 ==
* 22:24 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/PageAssessments/maintenance/purgeUnusedProjects.php --wiki=enwikivoyage {{!}} [[phab:T219935|T219935]]
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 09:46 hashar: Upgrading CI Quibble jobs to 0.0.31


== 2019-04-01 ==
== 2022-01-10 ==
* 21:42 hauskatze: Imported tool-ldap from Diffusion to Gerrit with full history {{!}} [[phab:T219703|T219703]]
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 21:33 hauskatze: Created https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/ldap {{!}} [[phab:T219703|T219703]]
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]
* 20:53 hashar: ssh contint1001.wikimedia.org sudo rm /tmp/docker-pkg-build.log
* 20:44 hashar: Building Quibble 0.0.31 containers again # [[phab:T219647|T219647]]  [[phab:T219786|T219786]]
* 20:10 hashar: gerrit: flush-caches --cache git_tags  # some tag got stalled when querying over https -  [[phab:T219786|T219786]]
* 18:50 hauskatze: Created mediawiki/extensions/ContributionCredits.git per request on mediawiki.org
* 18:00 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/#/c/integration/config/+/500504/
* 17:25 Krinkle: fresnel-node10-browser-docker failing with ENOMEM. Depooled integration-slave-docker-1049 as precaution.
* 17:04 hashar: Building CI docker images for Quibble 0.0.31 (yes it is a long day...)
* 13:50 hashar: Reverted CI Jenkins jobs to Quibble 0.0.28 # [[phab:T219647|T219647]]
* 13:11 hashar: Upgraded CI Jenkins jobs to Quibble 0.0.30 # [[phab:T219647|T219647]]
* 10:50 hashar: Manually triggering postmerge step of citoid due to [[phab:T219017|T219017]] for mvolz. On contint1001: zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/citoid --change 497315,1
* 08:08 hashar: Rebuilding Quibble Jessie containers that failed to build last week due to wikimedia-jessie container. # [[phab:T219647|T219647]]
* 08:07 hashar: Rebuilding container docker-registry.wikimedia.org/wikimedia-jessie # [[phab:T219683|T219683]]


== 2019-03-29 ==
== 2022-01-05 ==
* 19:12 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/497975/ + https://gerrit.wikimedia.org/r/498887/
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 17:30 hashar: some quibble 0.0.30 images fail to build for an unknown reason . Left to figure out after the week-end has passed..
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 17:22 hashar: Building Ci docker images for quibble 0.0.30 [[phab:T219647|T219647]]  [[phab:T219645|T219645]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 12:03 Krenair: added alaasarhan to deployment-prep [[phab:T219621|T219621]]
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2019-03-28 ==
== 2022-01-04 ==
* 18:49 Krenair: shut off deployment-db04 instance per [[phab:T219087|T219087]]
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 18:20 thcipriani: reload zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 17:36 paladox: forking plugins/quota from upstream
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383
* 17:24 Krenair: deployment-prep [[phab:T219087|T219087]] beginning master switch
* 10:45 hashar: Tagged Quibble 0.0.30 {{Gerrit|6ddc6d508cb554e6443ff72648da3ea8a3253fff}}
* 08:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/499539 (no-op) and https://gerrit.wikimedia.org/r/499717 (SecurePoll phan)


== 2019-03-27 ==
== 2022-01-03 ==
* 22:27 James_F: Altered Wikimedia GitHub settings to require 2FA; see [[phab:T198810|T198810]]
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
* 14:16 hashar: contint: refreshed all git caches manually from cumin: cumin --force 'name:docker' 'find /srv/git -name '*.git' -type d -print -exec git -C {} fetch --prune \;'
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
* 14:16 hashar: contint: added repositories to the git caches https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499482/


== 2019-03-26 ==
* 23:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499344 (part 1, 2 and 3)
* 20:36 paladox: create gerrit repo operations/software/gerrit/plugins/WikimediaBlocks [[phab:T219300|T219300]]
* 20:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499312
* 17:43 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/499262
* 17:42 Krinkle: Reloading Zuul to deploy merged-but-not-deployed patches https://gerrit.wikimedia.org/r/#/c/integration/config/+/497800/ and https://gerrit.wikimedia.org/r/#/c/498684/
== 2019-03-25 ==
* 14:46 mateusbs17: sunset deployment-maps03
== 2019-03-24 ==
* 16:06 Krenair: shut off old deployment-db03 instance per [[phab:T219087|T219087]]
* 05:39 Krenair: cleaned up old puppet certs/nodes -certcentral-testclient03 -certcentral-testdns -certcentral03 -zotero01 -eventgate-analytics -t153468-test -rd3-cptest-master01 -maps05
* 04:12 Krenair: removed php7.0-fpm package (conflicting with php7.2-fpm) and removed /etc/nginx/sites-enabled/default (conflicting with apache, puppet will remove the available copy too) from -deploy02, -jobrunner03, -mwmaint01, and -mediawiki-07 hosts to try to get puppet there happy again
== 2019-03-23 ==
* 23:44 legoktm: deploying https://gerrit.wikimedia.org/r/498681 https://gerrit.wikimedia.org/r/498682 https://gerrit.wikimedia.org/r/498683
* 23:33 legoktm: deploying https://gerrit.wikimedia.org/r/498247 https://gerrit.wikimedia.org/r/498249 https://gerrit.wikimedia.org/r/498680
* 20:52 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/498664 / [[phab:T215562|T215562]]
== 2019-03-22 ==
* 19:59 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/498466/ # [[phab:T219017|T219017]]
* 19:34 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/494548/
* 19:24 thcipriani: clean old docker-pkg image from contint1001
* 03:08 legoktm: rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T218963|T218963]]
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T215562|T215562]])
== 2019-03-21 ==
* 23:34 legoktm: rebuilding mediawiki-phan docker image for https://gerrit.wikimedia.org/r/498266
* 22:34 legoktm: onlined integration-slave-jessie-1002
* 22:33 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 21:03 legoktm: deploying https://gerrit.wikimedia.org/r/498178 https://gerrit.wikimedia.org/r/498185 https://gerrit.wikimedia.org/r/498187 https://gerrit.wikimedia.org/r/498189 https://gerrit.wikimedia.org/r/498182
* 15:22 <hashar>: pruning images/containers on integration-slave-docker-1021
* 10:13 <hashar>: deployment-deploy01: sudo rm -fR /tmp/mw-cache-master | files were from Mar 15 10:21
* 00:33 <legoktm>: deploying https://gerrit.wikimedia.org/r/497600
== 2019-03-20 ==
* 13:20 <hashar>: Scheduled update of Diffusion repository wikibase-termbox via https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ on request of Pablo_WMDE
== 2019-03-19 ==
* 22:47 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497322 / T218553
* 21:00 <paladox>: hiding operations/software/gerrit/plugins/WikimediaWebSessions from normal users
* 20:59 <paladox>: create operations/software/gerrit/plugins/WikimediaWebSessions project T218739
* 20:23 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/preferences.d/jessie_mitaka_pinning_*' # T218559
* 19:56 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list' # T218559
* 16:56 <Reedy>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/494802
* 15:48 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497437
* 14:01 <hashar>: Removed Zuul "check" pipeline | https://gerrit.wikimedia.org/r/#/c/integration/config/+/493188/ | T192217
* 02:21 <Krenair>: sudo aptly publish --architectures="all,amd64" --skip-signing repo buster-deployment-prep
* 02:09 <Krenair>: sudo aptly repo create -component="main" -distribution="buster-deployment-prep" buster-deployment-prep
== 2019-03-18 ==
* 16:39 Krenair: yet*
* 16:38 Krenair: created deployment-acme-chief01 and a client instance for further acme-chief testing + dev. used stretch, would be buster like prod but not sure that's easily available outside testlabs yes
* 10:09 hashar: contint1001: rm -fR /srv/doc1001.eqiad.wmnet
* 10:06 hashar: github: deleting https://github.com/wikimedia/wikidata-gremlin # archived [[phab:T155829|T155829]]
* 09:58 hashar: arming keyholder on integration-cumin
* 09:55 hashar: deleting shutdowned instance integration-publisher02 , we do not use it anymore since doc publishing got overhauled ( [[phab:T137890|T137890]] )  # [[phab:T218146|T218146]]
* 09:12 hashar: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
* 07:37 legoktm: deployed https://gerrit.wikimedia.org/r/496610
== 2019-03-17 ==
* 16:29 Krenair: deactivated and cleaned certs for deployment-redis3-changeprop02 and deployment-prometheus01 (which no longer appear to exist but were causing cumin to be upset)
* 16:27 Krenair: armed deployment-cumin keyholder
* 00:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496887
== 2019-03-16 ==
* 21:56 Krinkle: krinkle@contint1001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/OOJsUIAjaxLogin --change 490979,1
* 21:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497077
* 21:41 legoktm: deploying https://gerrit.wikimedia.org/r/496507 https://gerrit.wikimedia.org/r/496524 https://gerrit.wikimedia.org/r/496520 https://gerrit.wikimedia.org/r/496516 https://gerrit.wikimedia.org/r/496510
* 21:21 Krinkle: krinkle@contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/EventLogging --change 264494,4
* 21:20 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/497076
* 21:20 Krinkle: Removing doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-extensions-EventLogging (created by accident)
== 2019-03-15 ==
* 21:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496978
* 21:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496882
* 20:10 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/496688
* 18:21 thcipriani: clean old docker images from contint1001
* 17:40 thcipriani: rearm beta keyholder
* 16:54 thcipriani: reenable beta-scap-eqiad
* 16:08 thcipriani: disable beta-scap-eqiad to test new php, back shortly
== 2019-03-14 ==
* 23:48 Krinkle: Abort job quibble-vendor-mysql-hhvm-docker/39874/ for mwext-CentralAuth (stuck after 59 minutes)
* 21:25 hashar: Manually triggered tests for 12 ContentTranslation changes that had label:verified=-1 # [[phab:T216689|T216689]]
* 21:21 hashar: Updated quibble-vendor-mysql-hhvm-docker with latest libc6 hopefully fixing HHVM segfault within libpthread # [[phab:T216689|T216689]]
* 19:14 ebernhardson: restart logstash on deployment-logstash2 to re-read and re-create apifeatureusage template
* 16:34 hashar: rollback quibble-vendor-mysql-hhvm-docker job to no more capture core files, we have enough and a good lead ( reverting https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ ) # [[phab:T216689|T216689]]
* 12:31 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/4 # [[phab:T216689|T216689]]
* 12:12 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/3 # [[phab:T216689|T216689]]
* 10:49 hashar: triggering tests for all ContentTranslation pending changes # [[phab:T216689|T216689]]
* 09:54 hashar: ci: live hacked job https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/ in attempt to capture 'core' files from hhvm {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ {{!}} [[phab:T216689|T216689]]
== 2019-03-13 ==
* 21:26 thcipriani: pool new bigram CI instance integration-slave-docker-1054
* 20:58 thcipriani: deleting bigram CI instance integration-slave-docker-1046 due to corrupt disk cf: [[phab:T218245|T218245]]
* 20:44 thcipriani: marking integration-slave-docker-1046 offline
* 20:38 hashar: Added integration-slave-docker-1045 to Jenkins. The instance existed in WMCS but was not in Jenkins
* 20:13 bearND: (beta): Update mobileapps to {{Gerrit|5865552}} ({{Gerrit|7074964}} {{Gerrit|d6dc3cd}} {{Gerrit|fbc6262}})
* 19:13 hashar: integration-slave-docker-1046 is back online # [[phab:T218245|T218245]]
* 19:04 hashar: hard rebooting integration-slave-docker-1046 , not reachable over ssh # [[phab:T218245|T218245]]
* 18:48 hashar: Building containers releng/quibble-jessie-hhvm and releng/quibble-stretch-hhvm with HHVM core_dump_report enabled # [[phab:T216689|T216689]]
* 18:09 ebernhardson: restart elasticsearch on deployment-elastic* to deploy apifeature usage fix ([[phab:T183156|T183156]])
* 10:39 hashar: Triggered tests for ContentTranslation changes that had label:verified=-1 # [[phab:T217654|T217654]]
* 10:03 hashar: Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # [[phab:T217654|T217654]]
== 2019-03-12 ==
* 18:38 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/479558  / [[phab:T203506|T203506]]
* 18:23 thcipriani: bring integration-slave-jessie-1001 back online, /srv disk space now at 20% (not sure if someone cleared disk and forgot to repool)
* 16:45 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/495931
* 13:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/493139/
== 2019-03-11 ==
* 15:59 Krenair: previous !log was for deployment-prep
* 15:56 Krenair: added MSantos to projectadmin, per chat in -infrastructure he'll use it to work on maps. user is a new foundation software engineer
* 01:20 legoktm: deploying https://gerrit.wikimedia.org/r/495593
== 2019-03-10 ==
* 02:50 legoktm: deploying https://gerrit.wikimedia.org/r/495443 https://gerrit.wikimedia.org/r/493741
* 02:45 legoktm: deploying https://gerrit.wikimedia.org/r/495332 https://gerrit.wikimedia.org/r/495470
== 2019-03-08 ==
* 22:18 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490950/
* 20:20 legoktm: deploying https://gerrit.wikimedia.org/r/495302
* 20:16 legoktm: deploying https://gerrit.wikimedia.org/r/495300 https://gerrit.wikimedia.org/r/495298 https://gerrit.wikimedia.org/r/493502
* 11:36 hasharLunch: integration: deleting old java8 docker images:  sudo cumin --force 'name:docker' 'docker images{{!}}grep java8{{!}}awk "{ print $3 }"{{!}}xargs docker rmi'
* 10:04 hashar: Updating maven based jobs to latest java8 containers {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/495022/ {{!}} [[phab:T208938|T208938]]
== 2019-03-06 ==
* 18:25 hasharAway: contint1001: restart Jenkins for plugins upgrade
* 18:24 hasharAway: reloading zuul for {{Gerrit|I21d7fa2939f507441f7d130f8207541e28d762ad}}
* 18:02 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org
* 17:37 hashar: Reloading Zuul for  mediawiki/libs/Zest https://gerrit.wikimedia.org/r/#/c/integration/config/+/494609/2/zuul/layout.yaml
* 16:27 Krinkle: doc1001 had bad permissions set on /org/wikimedia/doc (chmod 755 instead of 775). Making it impossible to git pull in the way that post-merge Jenkins job on integration/docroot recommends. Fixed with `sudo -u doc-uploader chmod 775 /srv/wikimedia/org/wikimedia/doc`.
* 15:58 andrewbogott: deleting deployment-prometheus01 on Filippo's advice
* 13:31 gehel: upgrading logstash to 5.6.14 on deployment-logstash2
* 13:15 gehel: upgrading elasticsearch to 5.6.14 on deployment-logstash2
* 11:33 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mediawiki-09:~$ sudo systemctl restart php7.2-fpm # [[phab:T217323|T217323]]
== 2019-03-05 ==
* 20:10 thcipriani: reenable beta-scap-eqiad
* 19:17 thcipriani: disable beta-scap-eqiad due to [[phab:T217587|T217587]]
* 10:13 hashar: integration: fixed erroneous ssh key restriction for cumin {{!}} [[phab:T217642|T217642]]
== 2019-03-04 ==
* 17:36 hashar: Build Docker containers for chromium=v71 pin ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/494243/ )
* 15:14 herron: cherry picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/492390/ to deployment-puppetmaster03
* 13:49 hauskatze: GitHub: deleted wikimedia/mediawiki-extensions-UploadLocal {{!}} [[phab:T213011|T213011]]
* 09:59 hashar: cleaned docker images on integration-slave-docker-1037
* 07:38 legoktm: deploying https://gerrit.wikimedia.org/r/494167
== 2019-03-03 ==
* 02:26 Krinkle: tried rebooting or shutting down integration-slave-docker-1021, no response on horizon. Did pause/resume instead, which did work, after which shutdown/start worked. Jenkins agent has been relaunched and seems online again.
* 02:20 Krinkle: integration-slave-docker-1021 (ci1.medium) has jobs failing on it due to ENOMEM. Horizon shows in log: integration-slave-docker-1021 login: [4961938.696837] Out of memory: Kill process 21770 (chromium) score 841 or sacrifice child; [4961938.699176] Killed process 21770 (chromium) total-vm:3171496kB, anon-rss:1379288kB, file-rss:0kB, shmem-rss:1636kB
== 2019-03-02 ==
* 22:12 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/493959
* 20:44 hauskatze: Renamed https://github.com/wikimedia/wikimedia-github-community-health-defaults to https://github.com/wikimedia/.github
* 20:42 hauskatze: ssh -p 29418 gerrit.wikimedia.org replication start wikimedia/github-community-health-defaults --wait
* 20:40 hauskatze: github created https://github.com/wikimedia/wikimedia-github-community-health-defaults
* 20:31 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/493881
* 20:30 Krinkle: Failure on integration-slave-docker-1021 (ENOMEM) https://integration.wikimedia.org/ci/job/fresnel-node10-browser-docker/61/console
* 19:51 legoktm: deploying https://gerrit.wikimedia.org/r/493872
* 19:37 legoktm: deploying https://gerrit.wikimedia.org/r/493862
* 18:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493808
* 18:21 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493837
== 2019-03-01 ==
* 19:17 thcipriani: integration-slave-docker-1021:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '{print $3}')
* 17:02 thcipriani: integration-slave-jessie-1004 back online
* 16:58 thcipriani: integration-slave-jessie-1002 back online (disk space looked fine); rebooting integration-slave-jessie-1004 -- can't ssh to machine
* 16:11 Lucas_WMDE: delete refs/master and refs/gerrit/master on WikibaseQualityConstraints repository [[phab:T217408|T217408]]
* 15:49 hashar: wikidata/query/blazegraph  change Gerrit config to require a change-id # [[phab:T216855|T216855]]
* 14:28 hashar: Upgrading integration/jenkins-job-builder to version 2.0.2  + one custom hack 11aa5de4...a06d173e  # [[phab:T143731|T143731]]
* 14:18 hashar: integration/jenkins-job-builder : importing upstream code to new branch "upstream".  Push all upstream tags to our repository
== 2019-02-28 ==
* 18:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493261
* 03:42 duh: deploying https://gerrit.wikimedia.org/r/493357
* 03:42 duh: deploying
* 03:30 duh: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/493355
== 2019-02-27 ==
* 22:02 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/
* 19:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/492758
* 19:08 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758 (take II)
* 18:27 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758
== 2019-02-26 ==
* 22:52 ebernhardson: delete logstash logs in /var/log/logstash generated prior to 2019
* 22:51 ebernhardson: restart logstash on deployment-logstash2 while hacking around to see why apifeatureusage doesn't work
* 18:01 dcausse: deployement-prep: installing elastic 6.5.4 to deployment-elastic* machines
* 16:46 addshore: added Cparle to deployment-prep
* 16:11 hashar: Generating 1.33.0-wmf.19 deploy notes https://integration.wikimedia.org/ci/job/train-deploy-notes/9/console {{!}} [[phab:T206673|T206673]]
* 09:32 godog: remove now-merged node-exporter timer disable, cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/492632
== 2019-02-25 ==
* 23:32 twentyafterfour: root@deployment-db05# mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata # [[phab:T216067|T216067]]
* 22:14 thcipriani: docker rmi images without "latest" tag on contint1001 to free space -- should have kept all current docker-pkg images as well as images with children -- [[phab:T217094|T217094]]
* 13:39 hashar: Rebuilding some CI Docker images using PHP sury.org to switch the sury.org component from jessie to stretch ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/492666/ )
== 2019-02-24 ==
* 21:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/492561, [[phab:T216964|T216964]]
* 04:54 legoktm: rebuilding docker image for https://gerrit.wikimedia.org/r/485241
* 04:41 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/tools$ sudo -u zuul rm -rf phan
== 2019-02-23 ==
* 22:25 legoktm: deploying https://gerrit.wikimedia.org/r/492497
* 02:32 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/492427
* 01:59 Krinkle: Reloading Zuul to deploy ttps://gerrit.wikimedia.org/r/492425
== 2019-02-22 ==
* 20:59 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/492377
* 16:48 paladox: created "block-users" group for https://bugs.chromium.org/p/gerrit/issues/detail?id=10507 (reported by a wmf user)
* 13:38 Reedy: restarting zuul because nothing is being run
== 2019-02-21 ==
* 21:02 paladox: branch deploy/wmf/stable-2.16 from {{Gerrit|0e0ea0ff735da0a494347884917fc48881d7e545}} in operations/software/gerrit
* 19:49 Amir1: ores:5d937b1 is going beta
* 18:22 Amir1: ores:2d84709 going beta
* 17:45 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/483225/
* 16:35 hashar: Adjusting IP of https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ and adding it back # [[phab:T216513|T216513]]
* 14:10 Amir1: deploying ores:5d50713 to beta
== 2019-02-20 ==
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status  # [[phab:T216635|T216635]]
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status
* 17:26 hashar: For beta cluster the MySQL master database has some innodb issue [[phab:T216635|T216635]] , the MySQL slave has an issue as well [[phab:T216067|T216067]]
* 17:09 hashar: reloading zuul for {{Gerrit|Id1e3afd0afba9b388778066b9b6e8e564a25826b}}
* 17:09 hashar: contint1001: fix broken root ownership on zuul git deploy repo: sudo find /etc/zuul/wikimedia/.git -not -user zuul -exec chown zuul:zuul {} +
* 17:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490606/
* 16:57 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490559
* 16:36 hauskatze: Ran replication start mediawiki/extensions/PageViewInfo --wait on gerrit.wikimedia to populate GitHub mirror (success messages afterwards) {{!}} [[phab:T180864|T180864]]
* 00:25 greg-g: disabled beta-update-databases-eqiad in the jenkins UI - [[phab:T216067|T216067]]
== 2019-02-19 ==
* 21:23 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/491577, also deploys  https://gerrit.wikimedia.org/r/490492  / [[phab:T198495|T198495]]
* 20:28 hashar: jenkins: disable Android Emulator plugin {{!}} [[phab:T198495|T198495]]
* 20:27 hashar: gerrit: archived repository integration/jenkinsci/android-emulator-plugin {{!}} [[phab:T198495|T198495]]
* 20:23 hashar: Deleting Jenkins job apps-android-wikipedia-periodic-test {{!}} [[phab:T198495|T198495]]
* 15:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491513
* 15:46 hashar: Creating integration-slave-jessie-android m1.large it got deleted last week {{!}} [[phab:T216517|T216517]]
* 08:55 hashar: Cleaning contint1001 / partition
* 04:55 bd808: Removed stale cherry-pick of {{Gerrit|Ic7e726768701fefdee68622b08e3f2995779fe5a}} from deployment-puppetmaster03 that was blocking rebase on origin/production
* 04:50 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491405
* 04:14 Krinkle: Disabled spam account on Phab - https://phabricator.wikimedia.org/people/manage/18915/
* 03:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491403
* 00:54 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491381  / {{Gerrit|bc8e4198961cb73}} /  [[phab:T133646|T133646]]
== 2019-02-18 ==
* 22:32 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491379  / [[phab:T211784|T211784]]
* 21:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/491378 / [[phab:T133646|T133646]])
* 21:37 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/459268 / [[phab:T133646|T133646]]
* 15:38 elukey: kill/spawn deployment-aqs0[2,3] in deployment-prep with Debian Stretch
* 14:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490590
* 12:00 Krenair: [[phab:T216067|T216067]] Stopping mysql on -db04 to begin copy to -db05. Note crashed tables centralauth.globaluser and centralauth.localuser
* 11:57 elukey: kill/spawn deployment-aqs01 with Debian Stretch in deployment-prep
* 11:45 arturo: manually start deployment-db03 per Krenair request
* 11:29 hasharAway: beta: tried to start instance deployment-db03 172.16.5.23 --> ERROR  {{!}} [[phab:T216067|T216067]]
== 2019-02-17 ==
* 07:21 legoktm: deploying https://gerrit.wikimedia.org/r/491029
* 07:10 legoktm: Building image docker-registry.discovery.wmnet/releng/tox-acme-chief:0.3.4
* 06:28 legoktm: building new tox-acme-chief docker image https://gerrit.wikimedia.org/r/489725
* 01:12 Krinkle: beta-scap-eqiad (cron) failing with "sudo: a password is required"
== 2019-02-16 ==
* 19:44 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490937 / [[phab:T216275|T216275]])
* 17:23 thcipriani: installed php7.0-curl on deployment-deploy01 (why was that suddenly necessary?)
== 2019-02-15 ==
* 17:28 thcipriani: integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
== 2019-02-14 ==
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T216102|T216102]]
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T133646|T133646]]
* 13:24 thcipriani: rearm keyholder on deployment-deploy01: sudo keyholder arm, passwords on https://wikitech.wikimedia.org/wiki/Keyholder
== 2019-02-13 ==
* 21:32 marxarelli: dduvall@integration-slave-jessie-1001:/mnt/home/jenkins-deploy$ `rm -rf .gradle/ .m2/` due to full disk
* 21:21 marxarelli: bringing integration-slave-docker-1046 and integration-slave-jessie-1001 back online
* 21:20 marxarelli: dduvall@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
* 21:15 marxarelli: removing old docker images on integration-slave-docker-1046
* 21:10 marxarelli: starting migrated integration-slave-docker-1046 instance
* 21:01 marxarelli: pooling new jenkins node for integration-slave-docker-1052
* 20:46 marxarelli: pooling jenkins node for integration-slave-docker-1051
* 20:45 marxarelli: launching replacement instance integration-slave-docker-1052
* 20:35 marxarelli: launching replacement instance integration-slave-docker-1051
* 20:32 marxarelli: pooling jenkins node for integration-slave-docker-1050
* {{safesubst:SAL entry|1=20:15 marxarelli: integration-slave-docker-{1044,1046,1047} unresponsiveness due to cloudvirt failure. 1046 is being moved already by CS. deleting 1044 and 1047}}
* {{safesubst:SAL entry|1=19:57 marxarelli: seeing jenkins agent connection failures for integration-slave-docker-{1044,1046,1047}}}
* 19:48 marxarelli: pooling replacement jenkins node integration-slave-docker-1049
* 19:34 marxarelli: deleting integration-slave-jessie-android jenkins node and instance
* 19:33 marxarelli: deleting integration-slave-jessie-1003 jenkins node and instance
* 19:32 marxarelli: deleting integration-slave-docker-1033 jenkins node and instance
* 19:25 marxarelli: deleting integration-slave-docker-1017 jenkins node and instance
* 18:45 Krinkle: integration-slave-jessie-1003 seems to be consitently unable to start jobs, marking as offline manually
* 18:32 thcipriani: bringing up new integration-castor03, re-enabling castor-save* jobs
* 18:15 marxarelli: adding new jenkins node integration-slave-docker-1048
* 18:02 marxarelli: launching new integration-slave-docker-1048 instance
* 17:59 marxarelli: deleting integration-slave-docker-1038 node and deleting instance
* 17:50 marxarelli: bringing integration-slave-docker-1033 back online after clearing out old docker images
* 17:33 thcipriani: rebuilding integration-castor03
* 17:21 thcipriani: stopping rsync server on castor03
* 17:21 twentyafterfour: stopped rsync on castor03
* 17:16 twentyafterfour: disconnected castor03 from jenkins
* 16:48 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/487880/
* 14:34 thcipriani: modified castor-save-workspace-cache to exit 0 and run on blubber nodes while integration-castor03 is down
* 14:26 dcausse: deployement-prep: upgrading to elastic 5.6.14
== 2019-02-12 ==
* 23:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490130/1
== 2019-02-11 ==
* 21:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/489689
* 20:39 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/489688/
* 19:46 marxarelli: installing/enabling HTTP Request jenkins plugin on integration.wikimedia.org/ci to support https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/480689 changes
* 16:04 addshore: bring integration-slave-docker-1040 back online
* 16:04 addshore: addshore@integration-slave-docker-1040:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.52GB
* 10:47 godog: shut deployment-prometheus01, unused now
* 03:06 Reedy: graceful restart of zuul as no jobs were running
== 2019-02-10 ==
* 03:22 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489434  (Create quibble-stretch-hhvm, replacing jessie)
* 02:06 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489430
== 2019-02-08 ==
* 20:20 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config, ref [[phab:T91410|T91410]]
* 15:59 addshore: this reload also included "Switch npm-audit job to node10"? [[phab:T211784|T211784]], which did touch the zuul file
* 15:58 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/489241/
* 03:10 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config
* 00:28 Krinkle: krinkle@doc1001: sudo -u doc-uploader chmod 775 /srv/docroot/org/wikimedia/doc/
* 00:12 marxarelli: removed old docker images on contint1001 to free up space
== 2019-02-07 ==
* 23:17 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
* 23:15 thcipriani: integration-slave-docker-1033:sudo docker image prune and bring back online
* 22:28 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/467550 (
* 19:09 paladox: created integration/zuul/build gerrit repo for [[phab:T215458|T215458]]
* 19:05 paladox: created integration/zuul/wheels gerrit repo for [[phab:T215458|T215458]]
* 15:48 addshore: brought integration-slave-docker-1043 back online
* 15:48 addshore: addshore@integration-slave-docker-1043:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 14.86GB
* 08:49 hashar: cleaning docker images on integration-slave-docker-1021
== 2019-02-06 ==
* 22:34 shdubsh: Deploy node-exporter 0.17 [[phab:T213708|T213708]]
* 14:12 godog: shut off deployment-prometheus01 - [[phab:T215272|T215272]]
* 14:00 godog: switch beta-prometheus to deployment-prometheus02 - [[phab:T215272|T215272]]
== 2019-02-05 ==
* 20:07 ebernhardson: jobrunner port 9006 is firewalled, revert to 9005 and created [[phab:T215339|T215339]] to fix job queue in beta cluste
* 19:36 ebernhardson: Update profile::cpjobqueue::{jobrunner,videoscaler}_host in horizon hiera from port 9005 to 9006 to match new restrictions in gerrit.wikimedia.org/r/481866
* 16:29 addshore: [[phab:T215288|T215288]] added mirrys to deployment-prep as a user
* 15:32 addshore: [[phab:T215278|T215278]] addshore@integration-slave-docker-1037:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.59GB
== 2019-02-04 ==
* 23:13 thcipriani: integration-slave-docker-1040:sudo docker image prune and bring back online
* 23:12 thcipriani: integration-slave-docker-1038:sudo docker image prune and bring back online
* 21:48 ebernhardson: restart logstash on deployment-logstash2
* 15:25 hashar: removed Jenkins user "nodepoolmanager" as well as related authorizations {{!}} [[phab:T209361|T209361]]
== 2019-02-03 ==
* 06:15 legoktm: deployed https://gerrit.wikimedia.org/r/487627
* 05:44 legoktm: deployed https://gerrit.wikimedia.org/r/485967
* 04:36 legoktm: deploying https://gerrit.wikimedia.org/r/487534
== 2019-02-02 ==
* 22:17 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2019-01-31 ==
* 15:03 thcipriani: rearm keyholder on deployment-deploy01
* 12:05 arturo: VM instances deployment-deploy01,deployment-deploy02,deployment-fluorine02,deployment-kafka-jumbo-2,deployment-kafka-main-1,deployment-maps04,deployment-mcs01,deployment-mediawiki-09,deployment-memc04,deployment-ms-be03,deployment-ms-fe02,deployment-parsoid09,deployment-sca04,deployment-webperf12, were stopped briefly due to issue in hypervisor ([[phab:T215012|T215012]])
== 2019-01-30 ==
* 08:35 legoktm: deploying https://gerrit.wikimedia.org/r/486439 https://gerrit.wikimedia.org/r/481570 https://gerrit.wikimedia.org/r/481571
== 2019-01-29 ==
* 07:41 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * b/c full disk
== 2019-01-28 ==
* 16:33 hashar: contint1001: cleaning up disk space on /
* 13:07 addshore: bringing integration-slave-docker-1041 back online
* 13:07 addshore: addshore@integration-slave-docker-1041:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.12GB
* 09:37 Amir1: ores:ad160b0 is going beta
== 2019-01-27 ==
* 19:57 addshore: bringing integration-slave-docker-1034 back online
* 19:50 addshore: addshore@integration-slave-docker-1034:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.12GB
== 2019-01-26 ==
* 22:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486791
* 21:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486737
== 2019-01-25 ==
* 19:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486503/
* 19:07 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486501/
* 14:56 hashar: contint1001: systemctl stop zuul-merger && find /srv/zuul/git -name .git -type d -print -execdir git gc --prune=now \;
* 13:35 hashar: flake8 broken under python2.7 due to configparser==3.5.2  https://github.com/jaraco/configparser/issues/27 https://github.com/jaraco/configparser/issues/27
* 00:58 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* clean gradle cache, bring back online
== 2019-01-23 ==
* 22:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/486170/
== 2019-01-22 ==
* 23:46 thcipriani: Updating (more) tox docker image on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485971/
* 23:24 thcipriani: integration-slave-docker-1017 clean docker images and repool
* 21:06 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485895/
* 20:18 legoktm: rebuilding php, composer, phan images for apt security update https://gerrit.wikimedia.org/r/485883
* 19:59 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485882/
* 19:28 legoktm: rebuilding ci support images for apt security upgrade https://gerrit.wikimedia.org/r/485874
* 19:09 legoktm: rebuilding ci-stretch/ci-jessie images https://gerrit.wikimedia.org/r/485873
* 19:08 legoktm: manually docker pulled docker-registry.wikimedia.org/wikimedia-jessie and docker-registry.wikimedia.org/wikimedia-stretch on contint1001
* 18:56 bearND: (beta) Update mobileapps to {{Gerrit|0aac268}} (fix pronunciation detection in mobile-sections [[phab:T214338|T214338]])
* 18:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/485870/ and https://gerrit.wikimedia.org/r/#/c/integration/config/+/485331/
* 14:31 addshore: reloaded zuul for REVERT: Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:26 addshore: reloaded zuul for Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:03 addshore: reload zuul for Add ArticlePlaceholder to Wikibase Tests [integration/config] - https://gerrit.wikimedia.org/r/485753
== 2019-01-21 ==
* 19:49 hashar: integration: update sudo rule for debian-glue to keep env variable EXTRAPACKAGES. Would let us get eatmydata included {{!}} [[phab:T214328|T214328]]
* 15:40 hashar: contint1001: removing all generated doc/cover from /srv/org/wikimedia/doc {{!}} [[phab:T137890|T137890]]
== 2019-01-18 ==
* 23:22 hashar: contint1001: sudo docker image prune  # Total reclaimed space: 3.592GB
* 23:00 Krinkle: Some docker builds on integration-slave-docker-1021 failing with ENOMEM
* 23:00 mutante: contint1001 - gzipping more files in /var/log/zuul/
* 22:57 mutante: contint1001 - moved zuul logs from 2018 and gzipped zuul logs from /var/log/zuul to /srv/logs/zuul to free disk space on /
* 22:39 mutante: contint1001 - apt-get clean - disk space low
* 22:31 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482527  / [[phab:T212602|T212602]]
== 2019-01-17 ==
* 19:34 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 08:39 legoktm: deploying composer docker image - https://gerrit.wikimedia.org/r/484853
== 2019-01-16 ==
* 21:11 bearND: (beta): Update mobileapps to {{Gerrit|258d76b}} page summary changes
== 2019-01-15 ==
* 09:00 hashar: Deleting Docker images on integration-slave-docker-1021
== 2019-01-14 ==
* 22:02 bearND: (beta): Update mobileapps to {{Gerrit|f2658de}}
* 21:47 mutante: deployment-mcs01 - sudo su deploy-service; cd /srv/deployment/mobileapps/deploy-cache/revs/1182b3b8f288df0221257b929ca43fb86862c2f8/scap ; touch log  (for debugging permission problem reported by bearND)
* 14:31 hashar: Nuked Castor cache for all *tox* jobs. Some might have cached binary wheels compiled against a lib that is no more existing (eg libmysqlclient.so.18 for mysql-python). Follow up the jessie -> stretch upgrade # [[phab:T191764|T191764]]
* 14:28 hashar: Deleted Castor cache for wikimedia-cz/tracker  mysql-python got cached as  a wheel but compiled against libmysqlclient.so.18. That fails with the new tox...:0.3.0 containers which uses mariadb / libmysqlclient.so compat symlink
== 2019-01-11 ==
* 20:48 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
* 19:29 legoktm: deploying https://gerrit.wikimedia.org/r/483832
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/483824
* 04:37 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/483674
* 03:48 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483671
* 03:15 twentyafterfour: mediawiki-1.32.0 uploaded to releases.wikimedia.org (archives should be downloadable in a few minutes after puppet runs and the proxy's cache expires)
* 03:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483667 Switch unicodejs jobs to Node 10  - [[phab:T211784|T211784]]
* 02:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483666
* 02:01 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483656
* 01:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483644
* 00:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482755 - [[phab:T211784|T211784]]
== 2019-01-09 ==
* 19:59 legoktm: deploying https://gerrit.wikimedia.org/r/482766
* 18:05 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/483189 https://gerrit.wikimedia.org/r/#/c/integration/config/+/481947/
* 13:27 hasharAway: shutting down instance integration-publishing02 [172.16.4.5]  No more used {{!}} [[phab:T137890|T137890]]
* 13:26 hasharAway: deleting https://integration.wikimedia.org/ci/job/publish-on-contint1001/ no more used {{!}} [[phab:T137890|T137890]]
== 2019-01-08 ==
* 21:52 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/476600/
* 21:31 Hauskatze: github: @niedzielski updated @jdlrobson permission on Wikimedia from `read` to `admin`
* 21:30 Hauskatze: github:
* 20:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/482855/
* 19:53 mutante: deployment-prep adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 19:53 mutante: adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 14:25 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org/
* 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T 101131
* 05:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482752
* 02:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482751
== 2019-01-07 ==
* 20:29 Krinkle: Reloading Zuul to apply https://gerrit.wikimedia.org/r/482695
* 20:12 Krinkle: Delete job 'fresnel-npm6-browser-node-6-docker' and 'fresnel-node-10-docker'
* 20:08 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/482572
* 19:32 awight: [[phab:T212530|T212530]]: ORES revscoring 2.3.0
* 19:22 legoktm: building node10 docker images w/ https://gerrit.wikimedia.org/r/482573 this time
* 00:30 legoktm: deploying https://gerrit.wikimedia.org/r/482392 / create node10 mage
== 2019-01-06 ==
* 00:39 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482392
== 2019-01-03 ==
* 20:15 Hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/SparkPost
* 20:03 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-SparkPost
* 18:46 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests, part 2
* 18:32 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests
* 16:44 thcipriani: remove old docker images and bring integration-slave-docker-1021 back online
== 2019-01-02 ==
* 10:19 hashar: updating all debian-glue jobs and creating new ones with hardcoded distributions (trusty, jessie, stretch, unstable) [[phab:T210780|T210780]]
== 2019-01-01 ==
* 15:33 hashar: contint1001: deleting some extensions documentation for wmf branches: rm -fR /srv/org/wikimedia/doc/{Kartographer,MinervaNeue,MobileFrontend,Wikibase}/wmf  # [[phab:T118599|T118599]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 14:45, 28 June 2022

2022-06-28

2022-06-27

2022-06-24

  • 20:52 taavi: added `denisse` as a member

2022-06-23

2022-06-22

  • 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
  • 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
  • 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
  • 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
  • 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
  • 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
  • 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180

2022-06-21

  • 22:01 brennen: gitlab-runners: re-registering all shared runners
  • 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster

2022-06-20

  • 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
  • 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
  • 14:07 ori: restarted acme-chief on deployment-acme-chief03

2022-06-17

  • 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
  • 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack

2022-06-16

  • 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
  • 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
  • 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]

2022-06-15

  • 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
  • 16:31 hashar: integration-agent-docker-1035: docker image prune
  • 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster

2022-06-14

  • 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
  • 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
  • 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
  • 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production

2022-06-13

  • 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
  • 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore

2022-06-12

2022-06-10

  • 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
  • 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
  • 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
  • 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check

2022-06-09

  • 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
  • 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration

2022-06-08

  • 17:14 hashar: Reloaded Zuul for I393422
  • 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for T237033
  • 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134

2022-06-06

  • 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711

2022-06-02

  • 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
  • 11:26 hashar: Restarting Jenkins on contint2001
  • 11:19 hashar: Restarting Jenkins on releases1002

2022-05-31

  • 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
  • 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
  • 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
  • 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster

2022-05-30

  • 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
  • 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142

2022-05-28

  • 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437

2022-05-27

  • 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
  • 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
  • 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
  • 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
  • 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
  • 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
  • 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
  • 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep

2022-05-26

  • 18:33 dancy: Updated Jenkins beta-* job configs
  • 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 2c7b5825
  • 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch

2022-05-25

  • 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] T309230
  • 15:46 dancy: Restarted apache2 on gerrit1001

2022-05-24

2022-05-23

  • 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
  • 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
  • 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
  • 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
  • 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
  • 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
  • 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982

2022-05-22

2022-05-21

2022-05-20

2022-05-19

2022-05-18

  • 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
  • 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
  • 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
  • 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071

2022-05-17

  • 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558

2022-05-16

2022-05-14

  • 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
  • 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
  • 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request

2022-05-13

2022-05-12

  • 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
  • 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
  • 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
  • 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
  • 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)

2022-05-11

  • 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
  • 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster

2022-05-10

2022-05-09

2022-05-08

  • 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057

2022-05-06

  • 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071

2022-05-05

2022-05-04

2022-05-03

2022-05-02

2022-04-29

2022-04-28

2022-04-27

2022-04-26

  • 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
  • 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
  • 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
  • 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
  • 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
  • 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204

2022-04-25

2022-04-20

  • 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

2022-04-18

  • 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
  • 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
  • 16:48 brennen: phabricator: adding self to acl*blog-admins
  • 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.

2022-04-14

2022-04-12

2022-04-08

2022-04-07

  • 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
  • 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461

2022-04-06

  • 20:03 thcipriani: rebooting phabricator
  • 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596

2022-04-04

2022-04-02

2022-03-31

2022-03-29

  • 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
  • 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
  • 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771

2022-03-28

2022-03-27

  • 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736

2022-03-26

  • 02:37 Reedy: beta-update-databases-eqiad is back to @hourly

2022-03-25

  • 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
  • 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
  • 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
  • 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
  • 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord

2022-03-24

  • 20:00 hashar: reloading Zuul for Id844e1 # T299320
  • 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
  • 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
  • 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck

2022-03-23

  • 17:35 brennen: restarting phabricator for T304540, brief downtime expected
  • 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster

2022-03-22

2022-03-21

  • 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
  • 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203

2022-03-18

  • 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
  • 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
  • 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)

2022-03-17

2022-03-16

2022-03-15

2022-03-14

  • 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
  • 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
  • 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
  • 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
  • 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
  • 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
  • 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
  • 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
  • 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
  • 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
  • 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
  • 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079

2022-03-11

  • 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki

2022-03-10

2022-03-09

2022-03-08

  • 20:31 brennen: requiring 2fa for all users under /repos

2022-03-07

  • 10:53 zabe: restarted apache on deployment-mediawiki11 # T302699

2022-03-04

2022-03-03

2022-03-02

  • 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
  • 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
  • 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
  • 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04

2022-02-28

  • 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
  • 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
  • 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin

2022-02-26

  • 20:05 zabe: apply T302658 on deployment-prep centralauth databases
  • 13:24 zabe: apply T302660 on deployment-prep centralauth databases
  • 13:19 zabe: apply T302659 on deployment-prep centralauth databases

2022-02-24

  • 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
  • 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
  • 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
  • 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
  • 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
  • 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
  • 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
  • 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1

2022-02-23

  • 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
  • 20:10 dancy: Updating scap in beta
  • 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
  • 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
  • 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
  • 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774

2022-02-22

  • 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
  • 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

  • 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
  • 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
  • 07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

  • 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

  • 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
  • 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
  • 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

2022-02-17

  • 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
  • 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
  • 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
  • 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
  • 11:12 hashar: Bringing deployment-deploy03 back
  • 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

  • 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
  • 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
  • 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
  • 10:09 hashar: Reloading Zuul for I997fee
  • 09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

  • 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
  • 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
  • 16:24 hashar: Restarting CI Jenkins for plugins updates
  • 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
  • 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
  • 15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

  • 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
  • 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
  • 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
  • 14:26 jnuche: Jenkins upgrade complete T301361
  • 13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

  • 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

2022-02-09

  • 15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

  • 17:34 taavi: remove scap from deployment-kafka-main/jumbo
  • 16:23 taavi: hard reboot misbehaving deployment-echostore01
  • 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

2022-02-04

2022-02-03

  • 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
  • 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
  • 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

  • 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
  • 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

2022-01-31

  • 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
  • 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

  • 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
  • 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

  • 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
  • 19:34 Amir1: Reloading Zuul to deploy 757464
  • 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
  • 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
  • 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
  • 00:24 thcipriani: restarting jenkins

2022-01-26

  • 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
  • 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
  • 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
  • 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
  • 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

  • 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
  • 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
  • 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
  • 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
  • 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
  • 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
  • 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
  • 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
  • 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
  • 09:22 Reedy: unblocked beta again
  • 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

  • 21:44 Reedy: unstick beta ci jobs
  • 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
  • 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

  • 13:40 taavi: apply T299827 on deployment-prep centralauth database
  • 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

  • 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
  • 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

  • 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
  • 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
  • 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
  • 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
  • 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
  • 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
  • 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
  • 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
  • 12:10 hashar: contint2001 : docker container prune && docker image prune
  • 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
  • 12:04 hashar: contint1001 `docker image prune`
  • 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

2022-01-18

  • 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
  • 18:01 taavi: added ryankemper as a member of the deployment-prep project
  • 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
  • 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
  • 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
  • 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

  • 16:28 hashar: Building Quibble 1.3.0 Docker images
  • 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
  • 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

  • 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
  • 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
  • 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
  • 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

  • 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
  • 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
  • 16:27 dancy: testing scap prep auto on deployment-deploy03
  • 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
  • 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
  • 10:52 hashar: Restarting Jenkins CI for plugins update
  • 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
  • 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

2022-01-11

  • 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

  • 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
  • 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

2022-01-04

2022-01-03

  • 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
  • 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=591650"