You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Release Engineering/SAL
Jump to navigation
Jump to search
2021-04-14
- 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov T234020
2021-04-13
- 17:00 halfak: failed deploy to ORES (connection to host failed)
- 16:57 halfak: deploying ores f08a3cb
- 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
- 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 T280004
- 07:46 awight: enable syntax highlighting line numbering on all namespaces (T267911)
2021-04-12
- 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; T279853)
- 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # T279853
- 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per T106915#6279270 - T135427
- 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
- 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)
2021-04-11
- 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
- 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov T234020
- 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3{5,6} T274971
- 00:02 James_F: Docker: Publishing quibble-buster images T252432
2021-04-10
- 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list
2021-04-09
- 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, T233134.
- 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
- 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 T277750 phan for wikibase release branches
- 09:51 Majavah: deleting deployment-jobrunner03 T278664
- 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage T234080
2021-04-08
- 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
- 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563
2021-04-07
- 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
- 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
- 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
- 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 T278664
- 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
- 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382
2021-04-06
- 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived T277626
- 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs T279260
- 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines T279459
- 17:58 James_F: Zuul: [mediawiki/services/function-{orchestr,evalu}ator] Publish images
2021-04-05
- 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
- 17:45 brennen: halting gitlab-test for resize
2021-04-02
- 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
- 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12
2021-04-01
- 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
- 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
- 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
- 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298
2021-03-30
- 18:05 Majavah: remove {trysty,precise}-deployment-prep repos from deployment-deploy01 aptly
- 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
- 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those T278689
- 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
- 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
- 13:35 Majavah: create and install deployment-deploy03 T278689
- 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
- 07:26 Majavah: shutoff deployment-mediawiki-09 T278664
- 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
- 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11
2021-03-29
- 15:37 Majavah: hard reboot deployment-sessionstore03 T263617
- 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) T263617
- 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 (T278664), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
- 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
- 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
- 09:29 Urbanecm: Manually run puppet on mediawiki10
- 09:28 Urbanecm: Re-enable puppet on mediawiki10
- 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
- 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - T278664
- 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, T278664
2021-03-26
- 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
- 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 (T218729)
2021-03-25
- 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
- 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
- 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749
2021-03-24
- 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm T238707
2021-03-23
- 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per T274015
- 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job T207226
- 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
- 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade T252434
- 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template T252434
- 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch T252434
- 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch T252434
- 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use
2021-03-22
- 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
- 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
- 12:07 Majavah: delete deployment-restbase[01-02], T250574
- 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. (T276624)
- 11:33 dcaro: Created subzone beta.wmcloud.org (T276624)
2021-03-19
- 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
- 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
- 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, T218729
- 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
- 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down T250574
- 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for T250574
2021-03-18
- 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw (T277109)
- 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
- 18:27 brennen: updating gitlab-test to 13.9.4-ce
- 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 T277750
- 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
- 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # T269354
- 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 T277750 (apitests php versions)
- 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 T277060
- 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11 # T269354
- 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for T276179
- 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
- 08:44 Majavah: delete now unused deployment-ircd T277081
- 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now
2021-03-17
- 20:30 hashar: Reloaded Zuul for I236847
- 15:37 Majavah: shutdown deployment-restbase01 for T250574
- 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # T250574
- 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
- 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI
2021-03-16
- 21:37 longma: Updating dev-images docker-pkg files on primary contint
- 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
- 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
- 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
- 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
- 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1 # T277527
2021-03-15
- 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
- 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118
2021-03-13
- 17:10 twentyafterfour: restart apache on gerrit1001
2021-03-12
- 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
- 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
- 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
- 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
- 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ T276428
2021-03-11
- 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job(T277236)
- 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236)
- 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236)
- 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 T276419
- 16:49 Majavah: delete deployment-etcd-01 T276462
- 13:51 Majavah: shut down deployment-db06, now unused T277070
- 13:48 Majavah: set deployment-db07 as r/w T277070
- 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 T277070
- 13:41 Majavah: stop slave on deployment-db06 T276968
- 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 T277070
- 13:34 Majavah: stop and reset slave on deployment-db07 T277070
- 13:32 Majavah: set deployment-db06 as read only T277070
- 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/
2021-03-10
- 19:36 Majavah: shutdown deployment-ircd T277081
- 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 T277081
- 18:05 Majavah: create deployment-ircd02 for T277081
- 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
- 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
- 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
- 17:09 Majavah: set beta cluster mediawiki as read write on mw config (T276968)
- 17:03 Majavah: make deployment-db06 read-write T276968
- 16:50 Majavah: `reset slave;` on new master deployment-db06 T276968
- 16:49 Majavah: add deployment-db07 as a replica of db06 for T276968
- 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # T276968
- 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; (T276968)
- 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . (T276968)
- 15:57 Majavah: set deployment-db06 as readonly from mysql side T276968
- 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` (T276968)
- 15:54 Urbanecm: Start mariadb on db08 (T276968)
- 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 (T276968)
- 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
- 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
- 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade (T276968)
- 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) (T276968)
2021-03-09
- 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround (T276968)
- 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 (T276968)
- 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
- 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user (T276968)
- 20:33 Majavah: install mariadb on deployment-db08 T276968
- 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db (T276968)
- 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica (T276968)
- 19:50 marxarelli: restoring database dump on deployment-db07 (T276968)
- 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968)
- 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 T276968
- 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968)
- 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968
- 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump (T276968)
- 18:09 Majavah: set deployment-db05 to read-only to avoid issues with T276968
- 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968)
- 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
- 17:22 marxarelli: restarting deployment-db05 via horizon
- 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
- 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
- 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
- 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
- 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
- 13:32 arturo: hard-reboot deployment-db05 because issues related to T276922
- 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other
2021-03-08
- 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
- 08:57 hashar: Nuked castor cache for labs/striker # T276605
2021-03-07
- 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
- 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
- 17:18 Majavah: shutdown deployment-memc[04-05] T276707
- 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707
- 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707
- 15:44 Majavah: create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
- 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585
- 13:14 Majavah: create deployment-memc09 on Buster T276707
2021-03-06
- 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
- 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 T276654
- 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix T276652
2021-03-05
- 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
- 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
- 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
- 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462
- 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462
- 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462
- 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate T276462
- 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
- 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others
- 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2
2021-03-04
- 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384
- 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref T276102 and it being decomissioned in T242855
- 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for T276419, I believe everything has been moved to deployment-mwlog01
- 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for T276419
- 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
- 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
- 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref T276419
- 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
- 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # T276422 T180301
- 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
- 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch
2021-03-03
- 20:30 legoktm: added Majavah as projectadmin in deployment-prep (Beta Cluster)
- 19:59 James_F: Zuul: [mediawiki/services/function-schemata] Revert "Use bespoke pipeline jobs"
- 16:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Run phan with Echo
- 11:40 addshore: reload zuul for https://gerrit.wikimedia.org/r/667623
- 10:33 hashar: REplaced java-codehealth-patch job in favor of running sonar:sonar inline in all the java jobs. Thanks gehel! # T264873 | https://gerrit.wikimedia.org/r/c/integration/config/+/666869/
2021-03-02
- 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
- 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).
2021-03-01
- 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
- 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.
2021-02-27
- 22:03 Reedy: re-armed beta keyholder... I think...
2021-02-26
- 19:47 James_F: Zuul: [mediawiki/services/geoshapes] Add typescript service CI T274380
- 01:11 legoktm: update credentials in https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ for new ci-build user (T275559)
2021-02-24
- 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
- 22:16 legoktm: rebuilding Rust docker images
2021-02-23
- 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
- 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)
2021-02-20
- 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
- 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI
2021-02-19
- 13:51 hashar: Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897 did not get merged
- 13:49 hashar: Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." | https://gerrit.wikimedia.org/r/c/integration/config/+/651791/
2021-02-18
- 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
- 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata
2021-02-17
- 20:05 hashar: Updating all Jenkins jobs for https://gerrit.wikimedia.org/r/664897 # T275049
- 17:59 hashar: Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/664680 # T275049
- 03:26 James_F: Zuul: [mediawiki/core] PHP 8.0 version of composertest job to experimental
2021-02-16
- 20:33 brennen: updating gitlab-test.wmcloud.org to 13.8.4-ce.0
- 19:58 hashar: Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # T273822
- 17:02 greg-g: doing the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update dance
- 04:18 James_F: Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site
- 04:00 James_F: Zuul: Add Tim Abdullin from S&F to CI allow list
2021-02-15
- 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # T209953
2021-02-14
- 21:31 James_F: Zuul: Add 'check php' support for library repos
- 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive T274738
2021-02-13
- 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library
2021-02-12
- 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
- 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
2021-02-11
- 21:44 Krinkle: Logstash in beta is not receiving any events T274593
- 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
- 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension T274069
- 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
- 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
- 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339
2021-02-10
- 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
- 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for T274306
- 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
- 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension
2021-02-09
- 22:27 brennen: once more, with feeling: attempting docker-pkg run for dev-images again
- 22:09 brennen: attempting to run docker-pkg manually for dev-images on contint2001
- 21:08 brennen: Updating dev-images docker-pkg files on primary contint for gerrit:635361 and gerrit:632173
- 20:10 James_F: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld
- 19:31 James_F: Docker: Building and publishing tox-buster &c. with tox 3.21.4 T274232
- 04:43 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref T273247
- 04:42 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify
2021-02-08
- 21:40 brennen: enabled administrative approval for new account signups on gitlab-test
- 21:40 brennen: upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0
- 17:22 hashar: Built image docker-registry.discovery.wmnet/releng/quibble-buster-php72:0.0.45-s4
- 17:17 hashar: Building some docker images on contint.wikimedia.org
- 16:51 hashar: Now really reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
- 16:43 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
- 14:55 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658946 # T272863
- 09:06 _joe_: cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 in deployment-prep
2021-02-06
- 21:13 Reedy: unstuck beta jobs
2021-02-04
- 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
- 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI T273247
2021-02-03
- 21:38 James_F: Zuul: Archive VirtualKeyboard extension T273801
- 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
- 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
- 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL T273295
2021-02-02
- 19:43 hashar: Pruning dangling Docker images on contint2001
- 19:39 hasharDinner: Pruning dangling Docker images on contint1001
- 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move T61245
- 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
- 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # T273599
2021-02-01
- 16:04 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta.
- 10:55 Urbanecm: deployment-prep: Create beta votewiki (T272608)
- 09:15 hashar: devtools: switched gerrit-prod-1001 to local puppetmaster
2021-01-29
- 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes
2021-01-28
- 22:15 marxarelli: deleting unused integration-registry-1003 instance
- 17:46 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
- 17:42 marxarelli: updating jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
2021-01-27
- 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02
2021-01-26
- 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
- 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
- 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
- 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
- 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355
2021-01-25
- 15:57 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
2021-01-22
- 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout
2021-01-19
- 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI T271761
2021-01-18
- 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker T272320
2021-01-17
- 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job T248925
2021-01-16
- 23:24 James_F: Docker: Building cascade of new php-ast image T271428
2021-01-14
- 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job
2021-01-13
- 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents
2021-01-12
- 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
- 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
- 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
- 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
- 19:33 James_F: Zuul: Revert Parsoid integration job injection.
- 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
- 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension T237538
- 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template
2021-01-11
- 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # T271683
2021-01-09
- 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing T271575
- 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing T271575
2021-01-08
- 02:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/654964
2021-01-07
- 18:34 James_F: Zuul: Provide experimental PHP 7.2-on-buster jobs T252434
- 18:32 James_F: Zuul: [labs/tools/massmailer] Add gate-and-submit-l10n jobs T271426
- 16:01 hashar: Tag Quibble 0.0.46 @ df9e75329ab # T225218 T266441 T263500
2021-01-06
- 19:36 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/620729 in beta puppet.
- 18:33 dpifke: Doing a test run in beta of the scap commands I'm going to run in today's backport window to roll out profiler changes.
2021-01-05
- 23:55 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/651267 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/621095/3 in beta. Should be a no-op (removes hopefully unused code).
2021-01-04
- 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job T269719
- 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled for now due to T271122
- 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot # T271122
- 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts
2021-01-02
- 19:13 James_F: Zuul: Add CI for the Mirage skin T270979
2021-01-01
- 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master T270976
- 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting T251967