You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation (T287279))
imported>Stashbot
(hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/923647)
 
(438 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== 2021-07-26 ==
== 2023-05-26 ==
* 20:45 brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation ([[phab:T287279|T287279]])
* 15:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/923647
* 18:10 dduvall: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 18:00 dduvall: creating 2 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 08:24 hashar: jjb: update jobs to Quibble 1.0.1  # https://gerrit.wikimedia.org/r/c/integration/config/+/708038


== 2021-07-23 ==
== 2023-05-25 ==
* 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner ([[phab:T287279|T287279]])
* 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # [[phab:T334521|T334521]]
* 14:08 hashar: Building Docker images for quibble 1.0.1
* 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` #  [[phab:T334521|T334521]]
* 13:45 hashar: Tag Quibble 1.0.1 @ {{Gerrit|5a2548699a}} # [[phab:T287001|T287001]]
* 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait  # [[phab:T334521|T334521]]
* 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd


== 2021-07-21 ==
== 2023-05-24 ==
* 21:06 brennen: gitlab1001: running ansible for logging typo fix ([[phab:T274462|T274462]])
* 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, [[phab:T319477|T319477]]
* 20:36 dancy: Newest scap deployed to beta cluster
* 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # [[phab:T336362|T336362]]
* 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes ([[phab:T274462|T274462]], [[phab:T275170|T275170]])
* 16:46 dancy: restarting Gerrit to fix plugins
* 16:07 dancy: Updating plugins on releases-jenkins
* 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki


== 2021-07-20 ==
== 2023-05-23 ==
* 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar ([[phab:T281156|T281156]])
* 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
* 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47  # [[phab:T287001|T287001]]
* 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` [[phab:T337172|T337172]]


== 2021-07-19 ==
== 2023-05-21 ==
* 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values
* 10:54 Lucas_WMDE: aborted stuck job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/37265/console which was blocking gate-and-submit


== 2021-07-18 ==
== 2023-05-17 ==
* 08:48 majavah: set shared_acme_certificates: <nowiki>{</nowiki><nowiki>}</nowiki> on deployment-prep shared hiera, [[phab:T276653|T276653]]
* 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # [[phab:T336840|T336840]]
* 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)


== 2021-07-16 ==
== 2023-05-16 ==
* 21:01 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/698774
* 15:11 hashar: Killing stall https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ job
* 17:44 brennen: Try #2: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 15:04 hashar: CI Jenkins: updating IRC plugin from 2.42 to 3.829.v12d4b_d1f7650  and instant-messaging plugin from 1.48 to 2.666.va_6c1e97cc252  # [[phab:T283009|T283009]]
* 17:12 hashar: Tag Quibble 1.0.0 @ {{Gerrit|a13d133f7d1}} # [[phab:T286187|T286187]] [[phab:T280506|T280506]] [[phab:T90875|T90875]] [[phab:T218534|T218534]] [[phab:T227352|T227352]]
* 11:30 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919873 {{!}} [[phab:T336218|T336218]]
* 10:55 taavi: update operations-puppet-tests-buster-docker jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/914785


== 2021-07-14 ==
== 2023-05-12 ==
* 22:47 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 15:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919347/ {{!}} [[phab:T333861|T333861]]
* 21:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in beta puppet again, should only affect deployment-webperf12.


== 2021-07-13 ==
== 2023-05-11 ==
* 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep.  Should only affect deployment-webperf12.
* 21:20 RhinosF1: beta scap is paused while MatmaRex tests some changes
* 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for [[phab:T281430|T281430]]
* 20:41 thcipriani: pausing beta-code-update-eqiad/beta-mediawiki-config-update-eqiad/beta-scap-sync-world for debugging train
* 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived [[phab:T281430|T281430]]
* 16:25 hashar: Reloading Zuul
* 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet.  Should only affect deployment-webperf12.
* 07:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918970 {{!}} [[phab:T333125|T333125]]
* 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles  # [[phab:T278990|T278990]]
* 07:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918474 {{!}} [[phab:T333125|T333125]]
* 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
* 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # [[phab:T278990|T278990]]
* 07:46 hashar: Wiping all Docker images from contint2001l


== 2021-07-12 ==
== 2023-05-10 ==
* 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, [[phab:T286506|T286506]]
* 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot ([[phab:T327424|T327424]])
* 18:43 majavah: deployment-cache-text06 varnish not starting, [[phab:T286506|T286506]], causing an outage on text traffic on deployment-prep
* 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
* 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
* 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' ([[phab:T286479|T286479]])
* 14:44 majavah: fix merge conflict on deployment-puppetmaster04
* 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
* 13:19 James_F: Zuul: Add R4356th to the CI allow list
* 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [[phab:T286207|T286207]]


== 2021-07-09 ==
== 2023-05-09 ==
* 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days ([[phab:T286375|T286375]])
* 16:36 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project analytics/datahub --change 916483,4
* 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days ([[phab:T286358|T286358]])
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/917868 {{!}} [[phab:T329514|T329514]]
* 09:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/916513/ and https://gerrit.wikimedia.org/r/c/integration/config/+/916793/
* 08:04 hashar: Reloading Zuul to enable Sonar Codehhealth on TitleBlacklist, TextExtracts, TemplateData and SyntaxHighlight_GeSHI Mediawiki extensions # [[phab:T321837|T321837]]


== 2021-07-08 ==
== 2023-05-05 ==
* 08:21 majavah: kick stuck puppet agent on deployment-logstash04
* 23:39 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/916634
* 17:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916527


== 2021-07-07 ==
== 2023-05-04 ==
* 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance [[phab:T286298|T286298]]
* 13:51 _joe_: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/915652
* 12:00 _joe_: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/914782


== 2021-07-06 ==
== 2023-05-03 ==
* 01:16 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/703219
* 09:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/914715


== 2021-07-05 ==
== 2023-05-02 ==
* 15:02 Amir1: deployed 703212 ([[phab:T286058|T286058]])
* 16:55 dancy: scap version "4.52.0" installed on 15 hosts
* 08:15 hashar: Reloaded Zuul to enable Sonar Codehealth on CategoryTree https://gerrit.wikimedia.org/r/909331 and PageImages https://gerrit.wikimedia.org/r/909330 # [[phab:T321837|T321837]]
* 08:14 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/913541 https://gerrit.wikimedia.org/r/913925 https://gerrit.wikimedia.org/r/913191


== 2021-07-02 ==
== 2023-05-01 ==
* 17:32 brennen: gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/701068 ([[phab:T274463|T274463]])
* 17:20 brennen: experimenting with ldap user sync on gitlab.devtools
* 09:49 James_F: Zuul: [pywikibot/core] Add deeptest to gate-and-submit section


== 2021-06-30 ==
== 2023-04-28 ==
* 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
* 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for [[phab:T335602|T335602]]
* 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
* 17:57 thcipriani: restart ci jenkins following upgrade
* 17:54 thcipriani: restart releases-jenkins following upgrade
* 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31


== 2021-06-29 ==
== 2023-04-27 ==
* 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # [[phab:T285811|T285811]]
* 17:10 James_F: Zuul: [wvui] Mark repo as archived for [[phab:T310244|T310244]]


== 2021-06-28 ==
== 2023-04-26 ==
* 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
* 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list
* 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for [[phab:T285125|T285125]]


== 2021-06-24 ==
== 2023-04-25 ==
* 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)
* 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False  (manually it works, but that's the same command that puppet is supposed to run !?) - [[phab:T257317|T257317]] [[phab:T326368|T326368]]


== 2021-06-23 ==
== 2023-04-24 ==
* 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick
* 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - [[phab:T327853|T327853]]


== 2021-06-22 ==
== 2023-04-23 ==
* 23:03 jeena: increasing releases-jenkins executors from 2 to 4
* 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%<nowiki>{</nowiki>facts.fqdn<nowiki>}</nowiki>' instead of performance.discovery.wmnet as the latter doesn't resolve  / would be an invalid cert for https://deployment-webperf21, ref [[phab:T291015|T291015]]
* 21:25 jeena: Updating dev-images docker-pkg files on primary contint [[phab:T273682|T273682]]
* 09:31 hashar: Removed DISPLAY=:94 being set from Zuul for most jobs https://gerrit.wikimedia.org/r/c/integration/config/+/700857/


== 2021-06-21 ==
== 2023-04-22 ==
* 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.
* 21:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/908941
* 21:47 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/909624
* 19:35 Krinkle: Create excimer_ui_server db user in Beta Cluster on deployment-db13 based on prod grants. Password stored in deployment-puppetmaster04:/var/lib/git/labs/private/ (passwords::excimer_ui_server::$excimer_db_pass). Picked db13 because recommendationapi is here (also prod m2), and because mdb (created for this purpose originally) appears broken since several OS iterations (likely forgotten due to unusual name) [[phab:T301637|T301637]], [[phab:T331956|T331956]]
* 18:05 Krinkle: Fix database hostname dns error at https://performance.wikimedia.beta.wmflabs.org/xhgui/, switch from mdb01 to mdb02, ref [[phab:T301637|T301637]]
* 18:00 Krinkle: Move deployment-xhgui03 config from to deployment-xhgui prefix, https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/1d03d4cf84a148bff7e055d9939c44f30c618d85/deployment-prep/deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud.yaml https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ab1c9d2968ba3c292b81ba76f8c0775151630f94%5E%21/, ref [[phab:T301637|T301637]]


== 2021-06-19 ==
== 2023-04-21 ==
* 13:44 majavah: remove deployment-deploy02 [[phab:T278689|T278689]]
* 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # [[phab:T335181|T335181]]
* 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host [[phab:T283013|T283013]]


== 2021-06-16 ==
== 2023-04-20 ==
* 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException [[phab:T284732|T284732]]
* 09:59 taavi: update helm-lint jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/910433/
* 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert
* 09:13 joe: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/909996


== 2021-06-14 ==
== 2023-04-19 ==
* 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
* 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for [[phab:T334998|T334998]]
* 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
* 10:56 joe: Updating docker-pkg files on contint primary for helm-linter
* 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge [[phab:T236915|T236915]]
* 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing [[phab:T236915|T236915]]
* 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
* 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
* 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli


== 2021-06-13 ==
== 2023-04-18 ==
* 18:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699567
* 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
* 18:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699541
* 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for [[phab:T334397|T334397]]


== 2021-06-11 ==
== 2023-04-17 ==
* 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
* 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for [[phab:T281500|T281500]]
* 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
* 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497  # [[phab:T73480|T73480]]
* 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
* 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # [[phab:T100837|T100837]]
* 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for [[phab:T271011|T271011]].  Comment left on gerrit patch and task,  not reverting patch in master at this stage


== 2021-06-10 ==
== 2023-04-12 ==
* 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too [[phab:T281195|T281195]]
* 15:39 jnuche: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/908248
* 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency [[phab:T281195|T281195]]
* 09:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/899523


== 2021-06-08 ==
== 2023-04-10 ==
* 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
* 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
* 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
* 18:44 thcipriani: restart zuul-merger
* 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
* 17:40 brennen: gitlab1001: running k6 data generator
* 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 [[phab:T280491|T280491]]


== 2021-06-07 ==
== 2023-04-09 ==
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # [[phab:T263617|T263617]]
* 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #[[phab:T263617|T263617]]
* 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for [[phab:T284492|T284492]]
* 11:51 hashar: zuul enqueue  --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1  # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470


== 2021-06-05 ==
== 2023-04-06 ==
* 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium
* 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list


== 2021-06-04 ==
== 2023-04-05 ==
* 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
* 18:05 James_F: Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep, for [[phab:T333661|T333661]]
* 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
* 18:05 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production, for [[phab:T331718|T331718]]
* 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 [[phab:T284345|T284345]]
* 09:44 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905664
* 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [[phab:T284345|T284345]]
* 19:20 James_F: Docker: Publishing node12 CI images [[phab:T284343|T284343]]


== 2021-06-03 ==
== 2023-04-04 ==
* 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/*  # [[phab:T284125|T284125]]
* 05:04 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905262
* 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
* 00:24 James_F: Zuul: Add Anysite to CI allowlist
* 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline [[phab:T281132|T281132]]


== 2021-06-02 ==
== 2023-04-03 ==
* 19:21 urbanecm: deployment-prep: Enlarge mwlog's /srv partition to 5 GB (was 2 GB)
* 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
* 13:49 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project search/MjoLniR --change 685569,4 #  Rerun sonar analysis against https://gerrit.wikimedia.org/r/c/search/MjoLniR/+/685569
* 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 {{!}} https://gerrit.wikimedia.org/r/905190
* 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
* 10:20 hashar: Tag Quibble 1.5.4 @ {{Gerrit|0a37ce9367672e973b49b8293f57ebffe40abdf6}} # [[phab:T332930|T332930]] [[phab:T331280|T331280]] [[phab:T333412|T333412]]
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T333586|T333586]]
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T33586|T33586]]


== 2021-05-28 ==
== 2023-03-31 ==
* 09:41 addshore: reload zuul for https://gerrit.wikimedia.org/r/697033
* 16:24 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904818
* 08:02 hashar: Successfully published image docker-registry.discovery.wmnet/releng/ci-bullseye:0.1.0 # [[phab:T283777|T283777]]
* 16:20 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904810
* 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904805/
* 14:34 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 14:26 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]


== 2021-05-27 ==
== 2023-03-29 ==
* 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
* 06:09 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/903839
* 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
* 00:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/903539
* 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
* 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
* 16:05 brennen: gitlab1001: re-running ansible and puppet per [[phab:T279545|T279545]]
* 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
* 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs


== 2021-05-26 ==
== 2023-03-23 ==
* 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` ([[phab:T279545|T279545]])
* 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.
* 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org


== 2021-05-24 ==
== 2023-03-22 ==
* 06:19 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693639
* 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # [[phab:T242031|T242031]]
* 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql {{!}} sql centralauth --write # [[phab:T242031|T242031]]


== 2021-05-21 ==
== 2023-03-20 ==
* 18:30 James_F: Zuul: Add phan to all extensions and skins [[phab:T283097|T283097]]
* 21:43 brennen: phabricator: setting sbassett to administrator per [[phab:T331928|T331928]]
* 17:59 James_F: Docker: Publishing mediawiki-phan* images where the job passes if .phan/config.php is absent [[phab:T283097|T283097]]
* 08:30 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693361 https://gerrit.wikimedia.org/r/691323


== 2021-05-20 ==
== 2023-03-19 ==
* 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
* 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in [[phab:T332507|T332507]]
* 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
* 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` [[phab:T322469|T322469]]
* 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
* 14:43 hashar: deployment-prep: databases update hourly job times out, filed [[phab:T332507|T332507]]
* 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
* 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs [[phab:T280669|T280669]]


== 2021-05-19 ==
== 2023-03-17 ==
* 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs [[phab:T280669|T280669]]
* 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 ([[phab:T332455|T332455]])
* 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps [[phab:T279275|T279275]]
* 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 ([[phab:T332455|T332455]])
* 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users [[phab:T279552|T279552]]
* 10:36 mutante: disabled phab user 'Cloudduck1233'


== 2021-05-17 ==
== 2023-03-16 ==
* 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive [[phab:T279221|T279221]]
* 10:48 hashar: Updating all jobs to switch Castor from Stretch to Bullseye based image {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/899508/ {{!}} [[phab:T278203|T278203]]
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based) [[phab:T278203|T278203]]
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based)
* 08:20 hashar: github: created https://github.com/wikimedia/mediawiki-services-machinetranslation for replication from Gerrit # [[phab:T331256|T331256]]


== 2021-05-16 ==
== 2023-03-15 ==
* 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
* 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04
* 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
* 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/898807 {{!}} [[phab:T331061|T331061]], [[phab:T331621|T331621]]
* 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034


== 2021-05-15 ==
== 2023-03-14 ==
* 21:51 James_F: Zuul: [mediawiki/tools/cli] Make mw-cli-test experimental for now [[phab:T248779|T248779]]
* 16:54 hashar: Building Docker images for Quibble 1.5.3
* 21:40 James_F: Zuul: [mediawiki/tools/cli] Add new bespoke job [[phab:T248779|T248779]]
* 16:41 hashar: Tag Quibble 1.5.3 @ {{Gerrit|df49e8a04f5}} # [[phab:T331621|T331621]] [[phab:T331061|T331061]]
* 21:09 James_F: Zuul: [mediawiki/extensions/MediaWikiAuth] Mark repo as archived [[phab:T282955|T282955]]
* 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
* 18:38 James_F: Zuul: Temporarily remove TwoColConflict from gated extensions [[phab:T234002|T234002]] [[phab:T282935|T282935]].
* 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 09:30 Majavah: create deployment-logstash04 to install elk7
* 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 09:22 Majavah: beta: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315, remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/683837 [[phab:T277990|T277990]]
* 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 06:23 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/691494/ [[phab:T281986|T281986]]
* 02:44 James_F: MediaWiki being branched to REL1_40 from {{Gerrit|fd3ce7633984d0120288d4a6fe48343cf9ba86a2}} for [[phab:T329079|T329079]]


== 2021-05-14 ==
== 2023-03-13 ==
* 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php),  [[phab:T198673|T198673]]
* 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging [[phab:T326192|T326192]]
* 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, [[phab:T198673|T198673]],  https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
* 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging [[phab:T326192|T326192]]
* 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, [[phab:T198673|T198673]], https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
* 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior
* 20:51 Krinkle: I broke beta  `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
* 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
* 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
* 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job [[phab:T174018|T174018]] [[phab:T177260|T177260]] [[phab:T280901|T280901]]
* 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI [[phab:T281538|T281538]]


== 2021-05-13 ==
== 2023-03-12 ==
* 23:14 Krinkle: root@integration-agent-qemu-1001:/home/addshore# rm qmeutest/out.img (Reclaiming space to be able to make other changes)
* 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
* 22:48 longma: restarting zuul to try getting tests to run for https://gerrit.wikimedia.org/r/c/mediawiki/skins/Modern/+/684509
* 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
* 22:57 James_F: Zuul: Add Want to the CI whitelist
* 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 [[phab:T328240|T328240]]


== 2021-05-12 ==
== 2023-03-11 ==
* 19:40 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/439774 in deployment-prep puppet.  Hasn't been touched in over a year and is preventing merges.
* 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
* 19:32 James_F: Zuul: Add PageImages, PageViewInfo, and Graph to MediaWiki gated extension set [[phab:T249674|T249674]]
* 19:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/683695 in deployment-prep puppet.  Should only affect beta logstash.
* 16:15 James_F: Docker: Publishing civicrm 0.2.1 for [[phab:T277500|T277500]]
* 16:00 James_F: Zuul: Add GrowthExperiments to the MediaWiki gated extension set [[phab:T247507|T247507]] [[phab:T249674|T249674]]
* 15:36 _joe_: updating helm-linter docker image


== 2021-05-11 ==
== 2023-03-10 ==
* 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set [[phab:T237538|T237538]] [[phab:T249674|T249674]]
* 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
* 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI
* 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
* 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]
* 14:37 TheresNoTime: (deployment-prep) clear lock
* 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`


== 2021-05-10 ==
== 2023-03-09 ==
* 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
* 20:53 brennen: scheduled tentative phabricator update window for tuesday 2023-03-14 @ 21:00 UTC
* 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default ([[phab:T271206|T271206]])
* 11:37 hashar: Rolling back selenium-daily Jenkins jobs back to NodeJS 14 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/896026/ {{!}} [[phab:T330426|T330426]]
* 14:04 CFisch_WMDE: Forward renamed config name for improved template search features ([[phab:T277028|T277028]])
* 10:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895839


== 2021-05-07 ==
== 2023-03-08 ==
* 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue [[phab:T282018|T282018]]
* 20:12 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/893074
* 10:36 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895698 {{!}} [[phab:T331505|T331505]]
* 06:26 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895142
* 06:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895329 and https://gerrit.wikimedia.org/r/c/integration/config/+/895333 {{!}} [[phab:T321837|T321837]]


== 2021-05-06 ==
== 2023-03-07 ==
* 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds [[phab:T264877|T264877]]
* 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, [[phab:T326163|T326163]]
* 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
* 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job [[phab:T271562|T271562]]
* 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893765


== 2021-05-05 ==
== 2023-03-06 ==
* 23:52 James_F: Zuul: Ensure Parsoid's tests include the Disambiguator extension (take 2) [[phab:T271863|T271863]] [[phab:T237538|T237538]]
* 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~`  [[phab:T327853|T327853]]
* 20:38 James_F: Docker: Publishing quibble-buster* 0.0.47 images
* 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small {{!}} [[phab:T327853|T327853]]
* 19:58 James_F: Docker: Publishing quibble-stretch* 0.0.47-s1 images, now with node
* 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
* 18:23 James_F: Docker: Publishing quibble-stretch* 0.0.47 images
* 12:24 hashar: Building Docker images for Quibble 1.5.2 {{!}} https://gerrit.wikimedia.org/r/894617
* 16:54 hashar: Tag Quibble 0.0.47 @ {{Gerrit|8b200cfb0}} # [[phab:T271863|T271863]] [[phab:T199403|T199403]] [[phab:T281607|T281607]]
* 12:15 hashar: Tag Quibble 1.5.2 @ {{Gerrit|932b48cab06c}} # [[phab:T331236|T331236]] [[phab:T331061|T331061]] [[phab:T323750|T323750]]
* 14:17 CFisch_WMDE: Disable ReferencePreviews beta mode on beta labs ([[phab:T271206|T271206]])
* 13:22 Majavah: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/684034 https://gerrit.wikimedia.org/r/c/operations/puppet/+/684088 - might cause beta to go read only for a bit - [[phab:T110115|T110115]]


== 2021-05-04 ==
== 2023-03-05 ==
* 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
* 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons ([[phab:T331214|T331214]]#8666432)
* 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02


== 2021-05-03 ==
== 2023-03-03 ==
* 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [[phab:T281256|T281256]]
* 19:52 zabe: deployment-db13: start replication from deployment-db11 # [[phab:T331019|T331019]]
* 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
* 19:52 zabe: deployment-db13: import dump into mariadb # [[phab:T331019|T331019]]
* 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
* 08:56 hashar: Updating Jenkins jobs with JJB 4.0.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893667/
* 13:55 CFisch_WMDE: enable new search features for the template dialog ([[phab:T271802|T271802]])
* 00:37 zabe: take deployment-prep out of read-only # [[phab:T331019|T331019]]


== 2021-05-02 ==
== 2023-03-02 ==
* 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
* 22:01 zabe: enable read-only mode and create dump of all databases # [[phab:T331019|T331019]]
* 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
* 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T331019|T331019]]
* 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref [[phab:T198673|T198673]]
* 16:35 zabe: create volume db13 and attach to deployment-db13 # [[phab:T331019|T331019]]
* 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
* 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # [[phab:T331019|T331019]]
* 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117
* 16:24 zabe: shutoff deployment-db09 # [[phab:T331019|T331019]]
* 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # [[phab:T331019|T331019]]
* 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
* 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was {{Gerrit|ed287d58a1e9a2ea39f2eb5bbba31d8fc54269c2}} ) # [[phab:T330752|T330752]]
* 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # [[phab:T330752|T330752]]
* 12:13 zabe: delete deploymentwiki # [[phab:T198673|T198673]]


== 2021-05-01 ==
== 2023-03-01 ==
* 19:19 James_F: Zuul: Add atagar to the CI allow list
* 19:30 hashar: Building Docker images for Quibble 1.5.1 ( https://gerrit.wikimedia.org/r/c/integration/config/+/893528/ )
* 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - [[phab:T278641|T278641]]
* 19:17 hashar: Tagged Quibble 1.5.1 @ {{Gerrit|9a163094a855}} # [[phab:T330750|T330750]]
* 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004
* 18:30 brennen: gitlab: enable custom issue-tracker instance-wide
* 17:37 hashar: Deploying `device-analytics-*` jobs for for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 16:32 taavi: deployment-prep: block automatic/update.php execution of UpdateForMultipleDevicesSupport maintenance script in OATHAuth by pre-inserting rows in updatelog tables, will be run manually instead. [[phab:T242031|T242031]]
* 15:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 14:07 hashar: integration: removed krenair from Puppet config `passwords::root::extra_keys`


== 2021-04-30 ==
== 2023-02-28 ==
* 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
* 15:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892548/
* 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for [[phab:T257301|T257301]]
* 14:43 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892993 {{!}} [[phab:T271418|T271418]]
* 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
* 14:08 hashar: Updating Quibble jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/891742 (node 14 shipped from the node14 images instead of being manually installed at build time)
* 14:15 Majavah: revert above as it's not working, [[phab:T206158|T206158]]
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892546 {{!}} [[phab:T311462|T311462]]
* 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 [[phab:T206158|T206158]]
* 13:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892952 {{!}} [[phab:T296348|T296348]]
* 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
* 12:11 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892919 {{!}} [[phab:T299263|T299263]]
* 02:37 James_F: Docker: Publishing node10 images based on buster [[phab:T278203|T278203]] [[phab:T240955|T240955]]
* 10:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892910/ {{!}} [[phab:T299262|T299262]]
* 09:16 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892530 {{!}} [[phab:T299158|T299158]] {{!}} [[phab:T299161|T299161]]


== 2021-04-29 ==
== 2023-02-27 ==
* 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist [[phab:T281418|T281418]]
* 15:08 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892458 {{!}} [[phab:T299283|T299283]]
* 13:02 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892435 {{!}} [[phab:T327583|T327583]]
* 12:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891968


== 2021-04-28 ==
== 2023-02-25 ==
* 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs [[phab:T279180|T279180]]
* 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # [[phab:T249268|T249268]]
* 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
* 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip <nowiki>{</nowiki><nowiki>}</nowiki> \+  # [[phab:T249268|T249268]]
* 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.


== 2021-04-27 ==
== 2023-02-24 ==
* 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for [[phab:T277742|T277742]].
* 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- [[phab:T289883|T289883]]
* 17:52 Majavah: delete deployment-sessionstore03 [[phab:T263617|T263617]] [[phab:T278641|T278641]]
* 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)
* 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for [[phab:T281283|T281283]]
* 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/680476
* 12:44 hashar: Restarted CI Jenkins for plugins upgrade
* 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2  (with ldap plugin 1.26)
* 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
* 06:40 Majavah: installing deployment-sessionstore04 [[phab:T263617|T263617]]
* 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs [[phab:T281198|T281198]]


== 2021-04-26 ==
== 2023-02-23 ==
* 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list
* 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
* 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
* 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
* 14:00 taavi: update quibble* jenkins jobs for 891303
* 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config


== 2021-04-24 ==
== 2023-02-22 ==
* 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/891281 [[phab:T328240|T328240]]
* 14:19 hashar: Updating all jobs to add `QUIBBLE_API_KEY` to the Quibble jobs {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/891278/ {{!}} [[phab:T323750|T323750]]
* 02:48 James_F: Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 for [[phab:T314470|T314470]]
* 02:45 James_F: Zuul: Migrate all MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]
* 02:43 James_F: Zuul: Migrate all non-MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]


== 2021-04-23 ==
== 2023-02-21 ==
* 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
* 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
* 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
* 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs [[phab:T329034|T329034]]
* 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts
* 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
* 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
* 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation {{!}} [[phab:T321837|T321837]]


== 2021-04-22 ==
== 2023-02-20 ==
* 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
* 00:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/890197
* 02:53 Reedy: killed a few stuck beta ci jobs
* 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
* 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 [[phab:T280874|T280874]]
* 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental [[phab:T280874|T280874]]
* 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos [[phab:T280874|T280874]]


== 2021-04-21 ==
== 2023-02-16 ==
* 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
* 10:31 jnuche: restarted Zuul to clear stale events
* 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs [[phab:T280874|T280874]]
* 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
* 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies


== 2021-04-20 ==
== 2023-02-15 ==
* 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog ([[phab:T273425|T273425]])
* 20:07 taavi: unbroke puppet on deployment-ms-*
* 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor ([[phab:T271825|T271825]])
* 18:12 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/889499
* 16:03 thcipriani: restart ci jenkins for updates
* 16:02 thcipriani: restart releases jenkins for updates


== 2021-04-19 ==
== 2023-02-14 ==
* 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental [[phab:T280491|T280491]]
* 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er [[phab:T329000|T329000]]
* 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
* 15:02 zabe: deployment-db12: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for [[phab:T280491|T280491]]
* 14:04 zabe: delete deployment-db10 and volume db10 # [[phab:T329577|T329577]]
* 11:38 zabe: deployment-db11: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 02:19 zabe: deployment-db12: import dump into mariadb # [[phab:T329577|T329577]]
* 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # [[phab:T329577|T329577]]
* 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T329577|T329577]]
* 01:50 zabe: create volume db12 and attach to deployment-db12 # [[phab:T329577|T329577]]
* 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 01:14 zabe: deployment-db11: import dump into mariadb # [[phab:T329577|T329577]]
* 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # [[phab:T329577|T329577]]
* 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # [[phab:T329577|T329577]]


== 2021-04-17 ==
== 2023-02-13 ==
* 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for [[phab:T280420|T280420]]
* 23:48 zabe: create volume db11 and attach to deployment-db11 # [[phab:T329577|T329577]]
* 23:44 zabe: shutoff deployment-db10 # [[phab:T329577|T329577]]
* 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 23:20 zabe: created backup of all databases on deployment-db09 # [[phab:T329577|T329577]]
* 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
* 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm


== 2021-04-16 ==
== 2023-02-10 ==
* 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks [[phab:T234020|T234020]] [[phab:T280167|T280167]]
* 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for [[phab:T277440|T277440]]
* 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
* 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for [[phab:T324732|T324732]]
* 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch [[phab:T278203|T278203]]
* 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs [[phab:T252432|T252432]]
* 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref [[phab:T235142|T235142]]
* 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref [[phab:T198673|T198673]]
* 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref [[phab:T198673|T198673]]
* 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
* 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
* 17:02 dancy: Updating dev-images docker-pkg files on primary contint
* 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template [[phab:T252434|T252434]]


== 2021-04-15 ==
== 2023-02-08 ==
* 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
* 23:17 James_F: Zuul: Add Nik.xyz.in e-mail to CI allow list
* 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts
* 21:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/887829 # [[phab:T329231|T329231]]
* 18:11 Krinkle: Disable spam account https://phabricator.wikimedia.org/people/manage/35991/
* 12:37 James_F: Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job
* 09:16 hashar: deployment-prep: deployment-deploy03 `systemctl stop ferm`  to allow jenkins to connect to the instance # [[phab:T329056|T329056]]


== 2021-04-14 ==
== 2023-02-07 ==
* 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov [[phab:T234020|T234020]]
* 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for [[phab:T329077|T329077]]


== 2021-04-13 ==
== 2023-02-06 ==
* 17:00 halfak: failed deploy to ORES (connection to host failed)
* 09:30 hashar: Built tox Docker images to include `bats` {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/871178/
* 16:57 halfak: deploying ores {{Gerrit|f08a3cb}}
* 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
* 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 [[phab:T280004|T280004]]
* 07:46 awight: enable syntax highlighting line numbering on all namespaces ([[phab:T267911|T267911]])


== 2021-04-12 ==
== 2023-02-03 ==
* 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; [[phab:T279853|T279853]])
* 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old
* 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # [[phab:T279853|T279853]]
* 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per [[phab:T106915|T106915]]#6279270 - [[phab:T135427|T135427]]
* 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
* 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)


== 2021-04-11 ==
== 2023-02-02 ==
* 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
* 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for [[phab:T321309|T321309]]
* 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov [[phab:T234020|T234020]]
* 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete  "' research/navigation-vectors'"`
* 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3<nowiki>{</nowiki>5,6<nowiki>}</nowiki> [[phab:T274971|T274971]]
* 00:02 James_F: Docker: Publishing quibble-buster images [[phab:T252432|T252432]]


== 2021-04-10 ==
== 2023-01-31 ==
* 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list
* 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
* 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
* 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
* 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for [[phab:T326372|T326372]]


== 2021-04-09 ==
== 2023-01-30 ==
* 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, [[phab:T233134|T233134]].
* 23:23 James_F: Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' for [[phab:T327332|T327332]]
* 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
* 18:35 James_F: Zuul: [mediawiki/extensions/PushAll] Add basic tests
* 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 [[phab:T277750|T277750]] phan for wikibase release branches
* 14:39 hashar: Building Docker image for https://gerrit.wikimedia.org/r/884904
* 09:51 Majavah: deleting deployment-jobrunner03 [[phab:T278664|T278664]]
* 10:59 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/884835/
* 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage [[phab:T234080|T234080]]


== 2021-04-08 ==
== 2023-01-27 ==
* 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
* 20:42 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/884372
* 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563
* 19:35 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/884365/
* 14:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/884143 {{!}} [[phab:T315621|T315621]]
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884309
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884283
* 08:39 hashar: devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master {{!}} [[phab:T325128|T325128]]


== 2021-04-07 ==
== 2023-01-25 ==
* 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
* 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner.  [[phab:T327949|T327949]]
* 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
* 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for [[phab:T327949|T327949]]
* 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
* 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 [[phab:T278664|T278664]]
* 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
* 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382


== 2021-04-06 ==
== 2023-01-24 ==
* 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived [[phab:T277626|T277626]]
* 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [[phab:T326372|T326372]]
* 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs [[phab:T279260|T279260]]
* 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for [[phab:T326372|T326372]]
* 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines [[phab:T279459|T279459]]
* 17:19 thcipriani: restarting ci jenkins for updates
* 17:58 James_F: Zuul: [mediawiki/services/function-<nowiki>{</nowiki>orchestr,evalu<nowiki>}</nowiki>ator] Publish images
* 17:15 thcipriani: restarted releases-jenkins for plugin updates


== 2021-04-05 ==
== 2023-01-23 ==
* 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
* 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
* 17:45 brennen: halting gitlab-test for resize
* 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for [[phab:T292818|T292818]]


== 2021-04-02 ==
== 2023-01-20 ==
* 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
* 06:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881665 {{!}} [[phab:T327301|T327301]]
* 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12


== 2021-04-01 ==
== 2023-01-19 ==
* 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
* 20:01 James_F: Zuul: Add Izno to the CI allowlist
* 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # [[phab:T299954|T299954]]
* 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
* 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298
* 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
* 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
* 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907


== 2021-03-30 ==
== 2023-01-18 ==
* 18:05 Majavah: remove <nowiki>{</nowiki>trysty,precise<nowiki>}</nowiki>-deployment-prep repos from deployment-deploy01 aptly
* 21:33 dancy: Rebooting deployment-jobrunner04 for [[phab:T327329|T327329]]
* 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
* 17:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881457 {{!}} [[phab:T327301|T327301]]
* 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those [[phab:T278689|T278689]]
* 15:42 thcipriani: deployment-jobrunner04: sudo rm -rf /srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~
* 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
* 15:31 andrewbogott: shutting down deployment-imagescaler03 as it is long-overdue for replacement.  See [[phab:T294148|T294148]] for details.
* 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
* 10:10 arturo: bump trove quotas ([[phab:T326674|T326674]])
* 13:35 Majavah: create and install deployment-deploy03 [[phab:T278689|T278689]]
* 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 07:26 Majavah: shutoff deployment-mediawiki-09 [[phab:T278664|T278664]]
* 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
* 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11


== 2021-03-29 ==
== 2023-01-17 ==
* 15:37 Majavah: hard reboot deployment-sessionstore03 [[phab:T263617|T263617]]
* 12:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880951/
* 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) [[phab:T263617|T263617]]
* 10:05 hashar: Updating sonar-scanner jobs https://gerrit.wikimedia.org/r/c/integration/config/+/880893/
* 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 ([[phab:T278664|T278664]]), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
* 09:52 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-4
* 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
* 08:32 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880503 {{!}} [[phab:T321837|T321837]]
* 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
* 00:21 James_F: Zuul: [operations/debs/cadvisor] Add debian-glue-unstable CI for [[phab:T325557|T325557]]
* 09:29 Urbanecm: Manually run puppet on mediawiki10
* 00:21 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] +AbuseFilter phan dep
* 09:28 Urbanecm: Re-enable puppet on mediawiki10
* 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
* 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - [[phab:T278664|T278664]]
* 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, [[phab:T278664|T278664]]


== 2021-03-26 ==
== 2023-01-16 ==
* 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
* 22:29 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/880541
* 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 ([[phab:T218729|T218729]])
* 00:19 Krinkle: Reloading Zuul


== 2021-03-25 ==
== 2023-01-13 ==
* 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
* 10:41 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/879754
* 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
* 01:35 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879687
* 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749


== 2021-03-24 ==
== 2023-01-12 ==
* 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm [[phab:T238707|T238707]]
* 21:49 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879649
* 10:42 hashar: Deployed docroot change https://gerrit.wikimedia.org/r/878860 "zuul: Link to report_url if available"


== 2021-03-23 ==
== 2023-01-10 ==
* 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per [[phab:T274015|T274015]]
* 21:14 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/27
* 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job [[phab:T207226|T207226]]
* 16:50 James_F: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf for [[phab:T326653|T326653]]
* 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
* 15:28 James_F: Zuul: Add experimental jobs for fundraising/REL1_39 for [[phab:T326372|T326372]]
* 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade [[phab:T252434|T252434]]
* 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template [[phab:T252434|T252434]]
* 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use


== 2021-03-22 ==
== 2023-01-09 ==
* 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
* 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
* 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
* 16:30 wm-bot2: Increased quotas by 6 cores ([[phab:T326568|T326568]]) - cookbook ran by arturo@nostromo
* 12:07 Majavah: delete deployment-restbase[01-02], [[phab:T250574|T250574]]
* 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests
* 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T276624|T276624]])
* 11:33 dcaro: Created subzone beta.wmcloud.org ([[phab:T276624|T276624]])


== 2021-03-19 ==
== 2023-01-08 ==
* 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
* 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
* 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
* 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix
* 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, [[phab:T218729|T218729]]
* 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
* 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down [[phab:T250574|T250574]]
* 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for [[phab:T250574|T250574]]


== 2021-03-18 ==
== 2023-01-07 ==
* 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw ([[phab:T277109|T277109]])
* 22:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876327
* 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
* 18:27 brennen: updating gitlab-test to 13.9.4-ce
* 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 [[phab:T277750|T277750]]
* 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
* 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # [[phab:T269354|T269354]]
* 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 [[phab:T277750|T277750]] (apitests php versions)
* 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 [[phab:T277060|T277060]]
* 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11  # [[phab:T269354|T269354]]
* 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for [[phab:T276179|T276179]]
* 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
* 08:44 Majavah: delete now unused deployment-ircd [[phab:T277081|T277081]]
* 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now


== 2021-03-17 ==
== 2023-01-06 ==
* 20:30 hashar: Reloaded Zuul for {{Gerrit|I2368478e4c4ab8752581f55a7c5ab493fafdeb41}}
* 16:59 James_F: Docker: Publishing new CI images with composer 2.5.1 for [[phab:T319257|T319257]]
* 15:37 Majavah: shutdown deployment-restbase01 for [[phab:T250574|T250574]]
* 11:43 hashar: Refreshing Jenkins jobs to update Quibble jobs with php-excimer {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/876190
* 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # [[phab:T250574|T250574]]
* 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
* 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI


== 2021-03-16 ==
== 2023-01-05 ==
* 21:37 longma: Updating dev-images docker-pkg files on primary contint
* 09:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868193 {{!}} Make CentralAuth a clone dependency for MassMessage {{!}} [[phab:T145430|T145430]]
* 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
* 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
* 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
* 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
* 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1  # [[phab:T277527|T277527]]


== 2021-03-15 ==
== 2023-01-04 ==
* 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
* 19:42 MatmaRex: Ran maintenansce script refreshGlobalimagelinks.php for [[phab:T322588|T322588]]
* 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118
* 02:47 TimStarling: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/874959


== 2021-03-13 ==
== 2023-01-03 ==
* 17:10 twentyafterfour: restart apache on gerrit1001
* 19:14 dduvall: restarting zuul
* 19:12 Krinkle: Disable publishing of rPMSM, ref [[phab:T143162|T143162]], [[phab:T326112|T326112]]
* 17:38 hashar: gerrit: reactivated account for `jmando` using `gerrit set-account 9431 --active` {{!}} [[phab:T326159|T326159]]
* 14:20 hashar: integration: restarting all instances for Linux kernel upgrade
* 07:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # [[phab:T322368|T322368]]


== 2021-03-12 ==
* 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
* 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
* 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
* 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
* 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ [[phab:T276428|T276428]]
== 2021-03-11 ==
* 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job([[phab:T277236|T277236]])
* 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 [[phab:T276419|T276419]]
* 16:49 Majavah: delete deployment-etcd-01 [[phab:T276462|T276462]]
* 13:51 Majavah: shut down deployment-db06, now unused [[phab:T277070|T277070]]
* 13:48 Majavah: set deployment-db07 as r/w [[phab:T277070|T277070]]
* 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 [[phab:T277070|T277070]]
* 13:41 Majavah: stop slave on deployment-db06 [[phab:T276968|T276968]]
* 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 [[phab:T277070|T277070]]
* 13:34 Majavah: stop and reset slave on deployment-db07 [[phab:T277070|T277070]]
* 13:32 Majavah: set deployment-db06 as read only [[phab:T277070|T277070]]
* 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/
== 2021-03-10 ==
* 19:36 Majavah: shutdown deployment-ircd [[phab:T277081|T277081]]
* 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 [[phab:T277081|T277081]]
* 18:05 Majavah: create deployment-ircd02 for [[phab:T277081|T277081]]
* 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
* 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
* 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
* 17:09 Majavah: set beta cluster mediawiki as read write on mw config ([[phab:T276968|T276968]])
* 17:03 Majavah: make deployment-db06 read-write [[phab:T276968|T276968]]
* 16:50 Majavah: `reset slave;` on new master deployment-db06 [[phab:T276968|T276968]]
* 16:49 Majavah: add deployment-db07 as a replica of db06 for [[phab:T276968|T276968]]
* 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # [[phab:T276968|T276968]]
* 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; ([[phab:T276968|T276968]])
* 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . ([[phab:T276968|T276968]])
* 15:57 Majavah: set deployment-db06 as readonly from mysql side [[phab:T276968|T276968]]
* 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` ([[phab:T276968|T276968]])
* 15:54 Urbanecm: Start mariadb on db08 ([[phab:T276968|T276968]])
* 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 ([[phab:T276968|T276968]])
* 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
* 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
* 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade ([[phab:T276968|T276968]])
* 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) ([[phab:T276968|T276968]])
== 2021-03-09 ==
* 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 ([[phab:T276968|T276968]])
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
* 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user ([[phab:T276968|T276968]])
* 20:33 Majavah: install mariadb on deployment-db08 [[phab:T276968|T276968]]
* 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db ([[phab:T276968|T276968]])
* 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica ([[phab:T276968|T276968]])
* 19:50 marxarelli: restoring database dump on deployment-db07 ([[phab:T276968|T276968]])
* 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` ([[phab:T276968|T276968]])
* 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 [[phab:T276968|T276968]]
* 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 ([[phab:T276968|T276968]])
* 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster [[phab:T276968|T276968]]
* 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump ([[phab:T276968|T276968]])
* 18:09 Majavah: set deployment-db05 to read-only to avoid issues with [[phab:T276968|T276968]]
* 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances ([[phab:T276968|T276968]])
* 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
* 17:22 marxarelli: restarting deployment-db05 via horizon
* 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
* 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
* 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
* 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
* 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
* 13:32 arturo: hard-reboot deployment-db05 because issues related to [[phab:T276922|T276922]]
* 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other
== 2021-03-08 ==
* 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
* 08:57 hashar: Nuked castor cache for labs/striker # [[phab:T276605|T276605]]
== 2021-03-07 ==
* 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
* 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
* 17:18 Majavah: shutdown deployment-memc[04-05] [[phab:T276707|T276707]]
* 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 [[phab:T276707|T276707]]
* 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 [[phab:T276707|T276707]]
* 15:44 Majavah: create deployment-memc10 on Buster [[phab:T276707|T276707]], beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
* 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, [[phab:T276707|T276707]] [[phab:T250585|T250585]]
* 13:14 Majavah: create deployment-memc09 on Buster [[phab:T276707|T276707]]
== 2021-03-06 ==
* 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
* 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 [[phab:T276654|T276654]]
* 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix [[phab:T276652|T276652]]
== 2021-03-05 ==
* 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
* 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week [[phab:T276462|T276462]]
* 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref [[phab:T276462|T276462]]
* 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref [[phab:T276462|T276462]]
* 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate [[phab:T276462|T276462]]
* 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
* 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; [[phab:T276521|T276521]] and possibly others
* 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair [[phab:T241481|T241481]], directory is different on deployment-logstash2
== 2021-03-04 ==
* 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image.  That brings in tox installed with python3 instead of python2 # [[phab:T276384|T276384]]
* 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref [[phab:T276102|T276102]] and it being decomissioned in [[phab:T242855|T242855]]
* 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for [[phab:T276419|T276419]], I believe everything has been moved to deployment-mwlog01
* 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for [[phab:T276419|T276419]]
* 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
* 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
* 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref [[phab:T276419|T276419]]
* 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
* 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # [[phab:T276422|T276422]] [[phab:T180301|T180301]]
* 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
* 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch
== 2021-03-03 ==
* 20:30 legoktm: added Majavah as projectadmin in deployment-prep (Beta Cluster)
* 19:59 James_F: Zuul: [mediawiki/services/function-schemata] Revert "Use bespoke pipeline jobs"
* 16:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Run phan with Echo
* 11:40 addshore: reload zuul for https://gerrit.wikimedia.org/r/667623
* 10:33 hashar: REplaced java-codehealth-patch job in favor of running sonar:sonar inline in all the java jobs.  Thanks gehel!  # [[phab:T264873|T264873]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/666869/
== 2021-03-02 ==
* 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
* 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).
== 2021-03-01 ==
* 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
* 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.
== 2021-02-27 ==
* 22:03 Reedy: re-armed beta keyholder... I think...
== 2021-02-26 ==
* 19:47 James_F: Zuul: [mediawiki/services/geoshapes] Add typescript service CI [[phab:T274380|T274380]]
* 01:11 legoktm: update credentials in https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ for new ci-build user ([[phab:T275559|T275559]])
== 2021-02-24 ==
* 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
* 22:16 legoktm: rebuilding Rust docker images
== 2021-02-23 ==
* 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
* 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)
== 2021-02-20 ==
* 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
* 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI
== 2021-02-19 ==
* 13:51 hashar: Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897  did not get merged
* 13:49 hashar: Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/651791/
== 2021-02-18 ==
* 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
* 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata
== 2021-02-17 ==
* 20:05 hashar: Updating all Jenkins jobs for https://gerrit.wikimedia.org/r/664897 # [[phab:T275049|T275049]]
* 17:59 hashar: Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/664680  # [[phab:T275049|T275049]]
* 03:26 James_F: Zuul: [mediawiki/core] PHP 8.0 version of composertest job to experimental
== 2021-02-16 ==
* 20:33 brennen: updating gitlab-test.wmcloud.org to 13.8.4-ce.0
* 19:58 hashar: Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # [[phab:T273822|T273822]]
* 17:02 greg-g: doing the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update dance
* 04:18 James_F: Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site
* 04:00 James_F: Zuul: Add Tim Abdullin from S&F to CI allow list
== 2021-02-15 ==
* 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # [[phab:T209953|T209953]]
== 2021-02-14 ==
* 21:31 James_F: Zuul: Add 'check php' support for library repos
* 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive [[phab:T274738|T274738]]
== 2021-02-13 ==
* 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library
== 2021-02-12 ==
* 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
* 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2021-02-11 ==
* 21:44 Krinkle: Logstash in beta is not receiving any events [[phab:T274593|T274593]]
* 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
* 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension [[phab:T274069|T274069]]
* 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
* 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
* 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339
== 2021-02-10 ==
* 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
* 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for [[phab:T274306|T274306]]
* 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
* 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension
== 2021-02-09 ==
* 22:27 brennen: once more, with feeling: attempting docker-pkg run for dev-images again
* 22:09 brennen: attempting to run docker-pkg manually for dev-images on contint2001
* 21:08 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:635361]] and [[gerrit:632173]]
* 20:10 James_F: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld
* 19:31 James_F: Docker: Building and publishing tox-buster &c. with tox 3.21.4 [[phab:T274232|T274232]]
* 04:43 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref [[phab:T273247|T273247]]
* 04:42 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify
== 2021-02-08 ==
* 21:40 brennen: enabled administrative approval for new account signups on gitlab-test
* 21:40 brennen: upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0
* 17:22 hashar: Built image docker-registry.discovery.wmnet/releng/quibble-buster-php72:0.0.45-s4
* 17:17 hashar: Building some docker images on contint.wikimedia.org
* 16:51 hashar: Now really reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 16:43 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 14:55 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658946  # [[phab:T272863|T272863]]
* 09:06 _joe_: cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 in deployment-prep
== 2021-02-06 ==
* 21:13 Reedy: unstuck beta jobs
== 2021-02-04 ==
* 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI [[phab:T273247|T273247]]
== 2021-02-03 ==
* 21:38 James_F: Zuul: Archive VirtualKeyboard extension [[phab:T273801|T273801]]
* 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
* 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
* 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL [[phab:T273295|T273295]]
== 2021-02-02 ==
* 19:43 hashar: Pruning dangling Docker images on contint2001
* 19:39 hasharDinner: Pruning dangling Docker images on contint1001
* 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [[phab:T61245|T61245]]
* 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
* 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # [[phab:T273599|T273599]]
== 2021-02-01 ==
* 16:04 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta.
* 10:55 Urbanecm: deployment-prep: Create beta votewiki ([[phab:T272608|T272608]])
* 09:15 hashar: devtools: switched gerrit-prod-1001 to local puppetmaster
== 2021-01-29 ==
* 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes
== 2021-01-28 ==
* 22:15 marxarelli: deleting unused integration-registry-1003 instance
* 17:46 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
* 17:42 marxarelli: updating jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
== 2021-01-27 ==
* 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02
== 2021-01-26 ==
* 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
* 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
* 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
* 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
* 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355
== 2021-01-25 ==
* 15:57 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
== 2021-01-22 ==
* 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout
== 2021-01-19 ==
* 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI [[phab:T271761|T271761]]
== 2021-01-18 ==
* 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker [[phab:T272320|T272320]]
== 2021-01-17 ==
* 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job [[phab:T248925|T248925]]
== 2021-01-16 ==
* 23:24 James_F: Docker: Building cascade of new php-ast image [[phab:T271428|T271428]]
== 2021-01-14 ==
* 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job
== 2021-01-13 ==
* 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents
== 2021-01-12 ==
* 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
* 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
* 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
* 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
* 19:33 James_F: Zuul: Revert Parsoid integration job injection.
* 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
* 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension [[phab:T237538|T237538]]
* 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template
== 2021-01-11 ==
* 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # [[phab:T271683|T271683]]
== 2021-01-09 ==
* 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]
* 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]
== 2021-01-08 ==
* 02:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/654964
== 2021-01-07 ==
* 18:34 James_F: Zuul: Provide experimental PHP 7.2-on-buster jobs [[phab:T252434|T252434]]
* 18:32 James_F: Zuul: [labs/tools/massmailer] Add gate-and-submit-l10n jobs [[phab:T271426|T271426]]
* 16:01 hashar: Tag Quibble 0.0.46 @ {{Gerrit|df9e75329ab}} # [[phab:T225218|T225218]] [[phab:T266441|T266441]] [[phab:T263500|T263500]]
== 2021-01-06 ==
* 19:36 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/620729 in beta puppet.
* 18:33 dpifke: Doing a test run in beta of the scap commands I'm going to run in today's backport window to roll out profiler changes.
== 2021-01-05 ==
* 23:55 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/651267 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/621095/3 in beta.  Should be a no-op (removes hopefully unused code).
== 2021-01-04 ==
* 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job [[phab:T269719|T269719]]
* 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled  for now due to [[phab:T271122|T271122]]
* 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot  # [[phab:T271122|T271122]]
* 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts
== 2021-01-02 ==
* 19:13 James_F: Zuul: Add CI for the Mirage skin [[phab:T270979|T270979]]
== 2021-01-01 ==
* 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master [[phab:T270976|T270976]]
* 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting [[phab:T251967|T251967]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Latest revision as of 15:12, 26 May 2023

2023-05-26

2023-05-25

  • 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # T334521
  • 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` # T334521
  • 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait # T334521
  • 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd

2023-05-24

  • 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, T319477
  • 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # T336362

2023-05-23

  • 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
  • 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` T337172

2023-05-21

2023-05-17

  • 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # T336840
  • 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)

2023-05-16

2023-05-12

2023-05-11

2023-05-10

  • 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot (T327424)

2023-05-09

2023-05-05

2023-05-04

2023-05-03

2023-05-02

2023-05-01

  • 17:20 brennen: experimenting with ldap user sync on gitlab.devtools

2023-04-28

  • 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for T335602

2023-04-27

  • 17:10 James_F: Zuul: [wvui] Mark repo as archived for T310244

2023-04-26

  • 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list

2023-04-25

  • 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (manually it works, but that's the same command that puppet is supposed to run !?) - T257317 T326368

2023-04-24

  • 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - T327853

2023-04-23

  • 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%{facts.fqdn}' instead of performance.discovery.wmnet as the latter doesn't resolve / would be an invalid cert for https://deployment-webperf21, ref T291015

2023-04-22

2023-04-21

  • 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # T335181

2023-04-20

2023-04-19

  • 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for T334998
  • 10:56 joe: Updating docker-pkg files on contint primary for helm-linter

2023-04-18

  • 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
  • 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for T334397

2023-04-17

  • 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for T281500

2023-04-12

2023-04-10

  • 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
  • 18:44 thcipriani: restart zuul-merger

2023-04-09

  • 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2

2023-04-06

  • 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list

2023-04-05

2023-04-04

2023-04-03

  • 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
  • 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 | https://gerrit.wikimedia.org/r/905190
  • 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
  • 10:20 hashar: Tag Quibble 1.5.4 @ 0a37ce9 # T332930 T331280 T333412
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T333586
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T33586

2023-03-31

2023-03-29

2023-03-23

  • 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.

2023-03-22

  • 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # T242031
  • 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql | sql centralauth --write # T242031

2023-03-20

  • 21:43 brennen: phabricator: setting sbassett to administrator per T331928

2023-03-19

  • 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in T332507
  • 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` T322469
  • 14:43 hashar: deployment-prep: databases update hourly job times out, filed T332507

2023-03-17

  • 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 (T332455)
  • 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 (T332455)
  • 10:36 mutante: disabled phab user 'Cloudduck1233'

2023-03-16

2023-03-15

  • 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T327708
  • 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
  • 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 | https://gerrit.wikimedia.org/r/c/integration/config/+/898807 | T331061, T331621
  • 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034

2023-03-14

  • 16:54 hashar: Building Docker images for Quibble 1.5.3
  • 16:41 hashar: Tag Quibble 1.5.3 @ df49e8a04f5 # T331621 T331061
  • 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
  • 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for T326163
  • 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for T326163
  • 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 02:44 James_F: MediaWiki being branched to REL1_40 from fd3ce76 for T329079

2023-03-13

  • 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging T326192
  • 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging T326192
  • 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior

2023-03-12

  • 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
  • 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
  • 22:57 James_F: Zuul: Add Want to the CI whitelist
  • 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 T328240

2023-03-11

  • 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-10

  • 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
  • 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`
  • 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233
  • 14:37 TheresNoTime: (deployment-prep) clear lock
  • 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-09

2023-03-08

2023-03-07

  • 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, T326163
  • 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/893765

2023-03-06

  • 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~` T327853
  • 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small | T327853
  • 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
  • 12:24 hashar: Building Docker images for Quibble 1.5.2 | https://gerrit.wikimedia.org/r/894617
  • 12:15 hashar: Tag Quibble 1.5.2 @ 932b48cab06c # T331236 T331061 T323750

2023-03-05

  • 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons (T331214#8666432)

2023-03-03

2023-03-02

  • 22:01 zabe: enable read-only mode and create dump of all databases # T331019
  • 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T331019
  • 16:35 zabe: create volume db13 and attach to deployment-db13 # T331019
  • 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # T331019
  • 16:24 zabe: shutoff deployment-db09 # T331019
  • 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # T331019
  • 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
  • 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was ed287d5 ) # T330752
  • 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # T330752
  • 12:13 zabe: delete deploymentwiki # T198673

2023-03-01

2023-02-28

2023-02-27

2023-02-25

  • 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
  • 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)

2023-02-24

  • 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- T289883
  • 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)

2023-02-23

  • 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
  • 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
  • 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
  • 14:00 taavi: update quibble* jenkins jobs for 891303
  • 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config

2023-02-22

2023-02-21

  • 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
  • 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs T329034
  • 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
  • 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
  • 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation | T321837

2023-02-20

2023-02-16

  • 10:31 jnuche: restarted Zuul to clear stale events

2023-02-15

2023-02-14

  • 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er T329000
  • 15:02 zabe: deployment-db12: start replicating from deployment-db09 # T329577
  • 14:04 zabe: delete deployment-db10 and volume db10 # T329577
  • 11:38 zabe: deployment-db11: start replicating from deployment-db09 # T329577
  • 02:19 zabe: deployment-db12: import dump into mariadb # T329577
  • 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # T329577
  • 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T329577
  • 01:50 zabe: create volume db12 and attach to deployment-db12 # T329577
  • 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # T329577
  • 01:14 zabe: deployment-db11: import dump into mariadb # T329577
  • 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # T329577
  • 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # T329577

2023-02-13

  • 23:48 zabe: create volume db11 and attach to deployment-db11 # T329577
  • 23:44 zabe: shutoff deployment-db10 # T329577
  • 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # T329577
  • 23:20 zabe: created backup of all databases on deployment-db09 # T329577
  • 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
  • 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm

2023-02-10

  • 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for T277440
  • 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for T324732

2023-02-08

2023-02-07

  • 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for T329077

2023-02-06

2023-02-03

  • 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old

2023-02-02

  • 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for T321309
  • 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete "' research/navigation-vectors'"`

2023-01-31

  • 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
  • 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
  • 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
  • 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for T326372

2023-01-30

2023-01-27

2023-01-25

  • 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner. T327949
  • 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for T327949

2023-01-24

  • 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now T326372
  • 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for T326372
  • 17:19 thcipriani: restarting ci jenkins for updates
  • 17:15 thcipriani: restarted releases-jenkins for plugin updates

2023-01-23

  • 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
  • 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for T292818

2023-01-20

2023-01-19

  • 20:01 James_F: Zuul: Add Izno to the CI allowlist
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # T299954
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
  • 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
  • 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
  • 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907

2023-01-18

2023-01-17

2023-01-16

2023-01-13

2023-01-12

2023-01-10

2023-01-09

  • 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
  • 16:30 wm-bot2: Increased quotas by 6 cores (T326568) - cookbook ran by arturo@nostromo
  • 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests

2023-01-08

  • 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
  • 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix

2023-01-07

2023-01-06

2023-01-05

2023-01-04

2023-01-03


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=601311"