You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Release Engineering/SAL"

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/560128)
imported>Stashbot
(James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change T292910)
(473 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== 2019-12-21 ==
== 2021-10-22 ==
* 21:59 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/560128
* 21:19 James_F: Docker: Publishing quibble-buster-php73-coverage:1.1.1-s3 with quibble 1.1.1 and composer 2.1.8 for a change [[phab:T292910|T292910]]
* 18:06 dancy: Updating deb packages on deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud
* 00:31 James_F: Zuul: [DiscussionTools] Add EventLogging as dependency for Phan jobs


== 2019-12-20 ==
== 2021-10-21 ==
* 23:03 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --sleep 2 --batch-size=10 ([[phab:T241210|T241210]])
* 18:07 dduvall: replacing runner-1001 with new instance ([[phab:T293835|T293835]])
* 22:00 marxarelli: created trigger-helloworldoid-pipeline-test and helloworldoid-pipeline-test jobs from jjb
* 09:49 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test [[phab:T292728|T292728]]
* 11:43 hashar: integration/zuul.git : pushed upstream tags
* 11:41 hashar: integration/zuul push upstream 'master' branch to our 'upstream' branch


== 2019-12-19 ==
== 2021-10-19 ==
* 13:20 James_F: [BETA CLUSTER] mwscript maintenance/cleanupUsersWithNoId.php --wiki=testwiki --prefix=imported [[phab:T235971|T235971]]
* 22:24 brennen: gitlab-test: deleted unused gitlab-runner-test
* 11:59 James_F: Docker: Publishing java8-sonar-scanner:0.5.1
* 22:20 brennen: adding Dzahn and Aokoth to gitlab-test and gitlab-runners projects, cleaning up contractor access
* 09:41 hashar: Deploying Zuul change "Codehealth pipeline: Don't report back to gerrit"  https://gerrit.wikimedia.org/r/559419 for [[phab:T217008|T217008]]
* 21:20 brennen: Updating dev-images docker-pkg files on primary contint for php 8 images ([[phab:T283426|T283426]])
* 18:00 James_F: Zuul: [mediawiki/services/function-schemata] Switch from pipeline to direct CI
* 11:32 hashar: Archiving integration/gear.git , no more used after Python gear 0.16.0 has been released # [[phab:T289512|T289512]]


== 2019-12-18 ==
== 2021-10-18 ==
* 19:13 halfak: deploying ores {{Gerrit|80b1e62}}
* 16:26 Lucas_WMDE: killed one beta-scap-sync-world job that seemed stuck for a long time
* 14:34 James_F: Zuul: Add ol ui li to i18n security allow list
* 14:43 James_F: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [[phab:T290274|T290274]]
* 14:30 James_F: Docker: Push java8-sonar-scanner:0.5.0
* 09:41 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # [[phab:T248676|T248676]]


== 2019-12-17 ==
== 2021-10-16 ==
* 18:04 Krinkle: Create Jenkins Console section for "setup diffConfig", example at https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-diffConfig-docker/181/console
* 16:47 Lucas_WMDE: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T293070|T293070]], based on [[phab:T271808|T271808]]#6739578
* 17:39 James_F: Zuul: Revert Kartographer as dependency for WBMI per [[phab:T240971|T240971]]
* 01:17 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/730949
* 16:06 James_F: Zuul: Add Kartographer as dependency for WBMI
* 13:49 James_F: Zuul: [labs/tools/sonarqubebot] Add CI to run composer-test [[phab:T240937|T240937]]
* 11:54 James_F: Starting the train cut for 1.35.0-wmf.11


== 2019-12-16 ==
== 2021-10-15 ==
* 09:35 hashar: doc1001: sudo -u doc-uploader rm -fR /srv/docroot/org/wikimedia/doc/DOCKER-mediawiki-core
* 16:14 brennen: gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for [[phab:T292372|T292372]]


== 2019-12-15 ==
== 2021-10-14 ==
* 17:42 James_F: Zuul: [mediawiki/extensions/CreateAPage] Add WikiEditor as a dependency
* 16:13 addshore: reloaded zuul for zuul: mw-ext-MailGun will need PHP 7.3+ [[phab:T236396|T236396]]
* 17:13 James_F: Zuul: [mediawiki/extensions/CreateAPage] Enable CI tests via quibble
* 14:13 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/730207


== 2019-12-14 ==
== 2021-10-13 ==
* 19:49 James_F: Zuul: Deploying https://gerrit.wikimedia.org/r/557275
* 23:12 bd808: Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors.
* 19:49 James_F: Zuul: Deploying https://gerrit.wikimedia.org/r/557138
* 16:48 brennen: Updating dev-images docker-pkg files on primary contint for [[phab:T293277|T293277]]
* 11:56 hashar: integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes'


== 2019-12-13 ==
== 2021-10-12 ==
* 20:19 hashar: Regenerating MediaWiki core doxygen (and jsduck doc) # [[phab:T218233|T218233]]
* 19:11 hashar: Building Docker images for https://gerrit.wikimedia.org/r/730279
* 20:18 hashar: Reloaded Zuul for "Use Zuul mutex for mediawiki-core-doxygen-docker-publish" https://gerrit.wikimedia.org/r/#/c/integration/config/+/557102/
* 17:28 James_F: Zuul: [mediawiki/extensions/THICC] Mark as archived for [[phab:T292948|T292948]]
* 18:04 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/doxygen:0.6.1
* 16:39 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/557001
* 16:28 twentyafterfour: deployed codehealth jenkins jobs
* 12:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/556998/
* 11:08 hashar: deployment-mediawiki07 : removing faulty entry mwdeploy:x:497:498::/var/lib/mwdeploy:/bin/bash in /etc/passwd # [[phab:T73480|T73480]]
* 02:57 Krinkle: Restarting deployment-mediawiki-07. - [[phab:T180761|T180761]]
* 02:32 Krinkle: It appears puppet-agent has been locally disabled on deployment-mediawiki-07 for at least three days with "no reason given". Re-enabling  now to unbreak https://gerrit.wikimedia.org/r/556854  for [[phab:T180761|T180761]]
* 02:09 Krinkle: Create 'mongo' security group and apply to deployment-xhgui01 (ingress tcp/27017).  [[phab:T180761|T180761]]
* 01:26 Krinkle: Set `profile::webperf::site::xhgui_host: deployment-xhgui01.deployment-prep.eqiad.wmflabs` in Hiera for deployment-webperf11.  [[phab:T180761|T180761]]
* 00:22 Krinkle: Apply Puppet role class "xhgui::app" to deployment-xhgui01. [[phab:T238788|T238788]], [[phab:T180761|T180761]]


== 2019-12-12 ==
== 2021-10-11 ==
* 14:56 hashar: Removed jsduck from integration-slave-jessie-1002 and integration-slave-jessie-1004
* 15:35 hashar: gerrit: marked integration/composer read-only. We are no more using it and the few use cases left are tracked at https://phabricator.wikimedia.org/T249949
* 14:09 hashar: Removed saucelabs-01 and saucelabs-02 , the last jobs using them got removed. # [[phab:T240575|T240575]]
* 02:08 Krinkle: Browsers fail to connect with https://upload.wikimedia.beta.wmflabs.org/ (Certificate expired two days ago)
* 13:33 hashar: Remove label contintLabsSlave from integration-slave-jessie-1002 and integration-slave-jessie-1004 # [[phab:T225031|T225031]]
* 11:16 hashar: Manually rebuilding last build of https://integration.wikimedia.org/ci/job/scap-beta-deb/ to generate a scap package without relying on zuul-cloner {{!}}  https://gerrit.wikimedia.org/r/556643 {{!}} [[phab:T240551|T240551]]


== 2019-12-11 ==
== 2021-10-08 ==
* 20:26 James_F: Docker: Pushing node10-test:0.6.0 and friends
* 21:17 hashar: Purging Docker images on all CI agents
* 15:20 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/556359 [[phab:T240445|T240445]] (also pulled in 2 more changes?)
* 20:00 hashar: Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620
* 19:23 hashar: Building CI Docker images for Quibble 1.1.1
* 19:08 hashar: Tag quibble 1.1.1 @ {{Gerrit|b54af2aa60}}
* 19:07 brennen: gitlab1001: restarting to check correct session_duration value of 604800
* 11:01 hashar: Rebuilding CI Docker images based on Bullseye to include apt security updates


== 2019-12-10 ==
== 2021-10-07 ==
* 21:27 James_F: Zuul: Add Wugapodes to CI whitelist.
* 22:12 James_F: Zuul: [design/codex] Provide initial CI, node14-browser suite
* 13:20 hashar: Removing m4executor labels from Jenkins agents # [[phab:T225031|T225031]]
* 17:07 James_F: Zuul: [mediawiki/extensions/ParsoidBatchAPI] Mark as archived [[phab:T252744|T252744]]
* 13:16 hashar: Update Jenkins jobs to drop the m4executor label https://gerrit.wikimedia.org/r/#/c/integration/config/+/556039/ # [[phab:T225031|T225031]]
* 13:29 hashar: restarting CI Jenkins for git plugin update
* 13:11 hashar: integration: rearmed keyholder on integration-cumin
* 13:14 hashar: Upgraded CI Jenkins on contint2001
* 13:10 hashar: Upgraded releases Jenkins from 2.289.2 to 2.303.2
* 13:04 hashar: Updating Jenkins jobs for Quibble 1.1.0 # [[phab:T291549|T291549]]
* 04:46 Krinkle: Temporarily change `mediawiki-fresnel-patch-docker` job change back to quibble-fresnel:1.0.1 ref [[phab:T292684|T292684]]
* 04:41 Krinkle: Reconfigure `mediawiki-fresnel-patch-docker` job to revert quibble change back to quibble-fresnel:1.0.1
* 01:06 Reedy: unstick beta ci jobs


== 2019-12-09 ==
== 2021-10-06 ==
* 19:58 James_F: Adding wikibase termbox docs publishing post-merge job
* 11:19 hashar: Tag Quibble 1.1.0 @ {{Gerrit|3dabcab}}
* 19:49 hashar: Removed Jenkins label "hasSlaveScripts" from contint1001 # [[phab:T225031|T225031]]
* 18:08 Amir1: soft rebooting mwmaint01
* 17:51 Amir1: ladsgroup@deployment-mwmaint01:/srv/mediawiki-staging$ sudo apt-get install php-memcached ([[phab:T239894|T239894]])
* 17:41 James_F: Zuul: Add JJMC89 and TechneSiyam to CI whitelist
* 10:22 hashar: castor: nuked cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node10-docker/ # [[phab:T240174|T240174]]


== 2019-12-06 ==
== 2021-10-05 ==
* 10:34 hashar: Purged pbuilder / cowbuilder etc from integration-slave-jessie-1002 and integration-slave-jessie-1004 # [[phab:T224943|T224943]]
* 21:47 brennen: updating dev-images docker-pkg files on primary contint for https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/7
* 12:03 majavah: root@deployment-cache-text06:~# systemctl restart sssd # [[phab:T286502|T286502]]
* 11:11 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/726576


== 2019-12-05 ==
== 2021-10-04 ==
* 20:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/#/c/554944/
* 21:09 brennen: finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 ([[phab:T292256|T292256]])
* 19:22 Reedy: killed stuck beta CI jobs, stuff is flowing again
* 21:02 brennen: gitlab-runners: upgrading each shared runner to 14.3.2
* 19:15 James_F: Zuul: [labs/toollabs] Drop trusty, add jessie to gate as well as test [[phab:T224943|T224943]]
* 18:12 James_F: Docker: Publish composer 2.x-based images for [[phab:T279857|T279857]]
* 18:36 James_F: Zuul: Add Majavah to CI whitelist
* 16:46 James_F: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
* 10:14 hashar: Deleted integration-slave-jessie-1001 , created integration-agent-pkgbuilder-1002 # [[phab:T224943|T224943]]
* 15:54 James_F: Zuul: Add Lindsay Wardell personal e-mail to CI allow list
* 10:14 hashar: Deleted integration-slave-jessie-1001 , created integration-agent-pkgbuilder-1002 # TT224943
* 14:31 brennen: gitlab1001: upgrading to 14.3.2 ([[phab:T292256|T292256]])


== 2019-12-04 ==
== 2021-09-30 ==
* 22:35 hashar: integration: added Stretch instance integration-agent-pkgbuilder-1001 for Debian glue jobs # [[phab:T224943|T224943]]
* 21:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/683031
* 21:48 hashar: Updated debian glue non voting jobs to use git instead of zuul-cloner {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/554588/ {{!}} [[phab:T224943|T224943]]
* 19:22 James_F: Zuul: Add CI for wikibase/vuejs-components
* 17:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php dewiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat  # make myself bureaucrat to confirm users
* 16:56 Amir1: MariaDB [wikidatawiki]> delete * from wb_changes_dispatch; ([[phab:T237551|T237551]])
* 16:49 James_F: Zuul: Add CI for operations/software/netbox-extra [[phab:T233183|T233183]]
* 15:57 James_F: Zuul: Add CI for EventStreamConfig MW extension [[phab:T233634|T233634]]
* 15:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/554530


== 2019-12-03 ==
== 2021-09-29 ==
* 21:24 James_F: Enabling PHP74 tests as voting for MW skins [[phab:T233012|T233012]]
* 15:52 hashar: Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # [[phab:T277500|T277500]]
* 17:33 kevinbazira: deploying ores {{Gerrit|6dd1fef87fcd5b76eb52e4a64087fbe8a982c54f}}
* 15:03 brennen: gitlab: pausing shared runners while working through access-level implications
* 16:33 hashar: Restarted Jenkins
* 16:10 hashar: CI Jenkins: updating collapsing console sections plugin with upstream fixes (PR #16 and #18) # [[phab:T236222|T236222]]
* 15:56 James_F: Zuul: Add IAmNetx to CI whitelist
* 10:45 hashar: Building releng/operations-puppet to upgrade tox and ship python3.4 3.6 and 3.7 # [[phab:T239334|T239334]]
* 01:43 James_F: Docker: Pushing operations-dnslist:0.0.6


== 2019-12-02 ==
== 2021-09-27 ==
* 20:18 Krenair: deactivated and cleaned cert for deployment-eventgate-2 instance that appears to have been deleted at some point
* 20:52 James_F: Zuul: [integration/visualdiff] Add basic node12 CI job
* 20:15 brennen: Updating dev-images docker-pkg files on contint1001 for [[phab:T238605|T238605]]
* 20:15 brennen: gitlab1001: login and user rename as "brennen" checks out; applying user mappings for remaining ~80 users
* 17:46 James_F: Zuul: Parsoid now has PHP74 as voting.
* 19:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/724103/3
* 17:26 James_F: Zuul: Adding experimental PHP74 jobs to anything that moves [[phab:T236132|T236132]]
* 19:24 hashar: Updating Jenkins jobs using a Stretch related image to latest version of the image. https://gerrit.wikimedia.org/r/c/integration/config/+/724156/ # [[phab:T291425|T291425]]
* 17:13 James_F: Docker: Pushing initial php74 images forT236132
* 17:23 hashar: Building java8-scala-spark image
* 09:51 hashar: jenkins: removing Command Agent Launcher plugin https://plugins.jenkins.io/command-launcher
* 07:11 hashar: Switching python based jobs to docker-registry.wikimedia.org/releng/tox-buster:0.5.0 # [[phab:T291292|T291292]]


== 2019-11-30 ==
== 2021-09-24 ==
* 13:27 Krenair: found new instance deployment-xhgui01 with broken puppet (not trusting our puppetmaster CA), ran the usual commands to get it working
* 15:41 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/coal/+/722948 and latest https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep.  Should only affect deployment-webperf11.


== 2019-11-29 ==
== 2021-09-23 ==
* 15:14 hashar: deployment-prep: rearming keyholder on deployment-deploy01
* 19:55 brennen: gitlab-ansible-test: sudo gitlab-ctl cleanse to drop test data and reset
* 18:41 James_F: Zuul: Create 'extension-apitests' template for simplicity
* 15:54 brennen: gitlab1001: brief downtime to apply [[gerrit:714382{{!}}gitlab cas: uid instead of CN; add nickname_key]] for [[phab:T288392|T288392]]
* 15:13 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/723217
* 14:46 hashar: Building releng/tox-buster:0.5.0 image
* 11:44 hashar: Reloaded zuul for "zuul: promote shellcheck job for integration/config" https://gerrit.wikimedia.org/r/c/integration/config/+/723160/1
* 10:57 hashar: Building tox images for improved shellcheck package https://gerrit.wikimedia.org/r/721895


== 2019-11-28 ==
== 2021-09-22 ==
* 13:59 hashar: gerrit: changed labs.git HEAD to point to refs/meta/config , master did not exist and that makes it a PERMISSIONS repository
* 15:47 Amir1: set row format for text table in simplewiki to compressed
* 13:57 hashar: gerrit: changed jsdoc.git HEAD to point to refs/meta/config , master did not exist and that makes it a PERMISSIONS repository
* 15:22 Amir1: restart mysql on db07 to add performance schema
* 11:40 hashar: deployment-prep: rebased puppet. Notably to fix wmf-beta-update-databases ( https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/553200/ ) # [[phab:T239399|T239399]]


== 2019-11-27 ==
== 2021-09-21 ==
* 18:30 hashar: Fixed up the Jenkins job for beta cluster
* 16:26 dduvall: deleted wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to a branch cut job gone awry. re-running job to cut wmf/1.38.0-wmf.1 ([[phab:T281165|T281165]])
* 16:04 dduvall: deleting wmf/1.37.0-wmf.24 and wmf/1.38.0-wmf.1 branches due to the branch cut job gone awry
* 15:49 James_F: Re-running the branch cut job for [[phab:T281165|T281165]]
* 15:48 James_F: Updated the releases-jenkins automatic branch cut job to make 1.38.0 branches (missed ahead of [[phab:T281165|T281165]])


== 2019-11-26 ==
== 2021-09-17 ==
* 20:00 Krinkle: Click "Disable Publishing" for extension-CollaborationKit, [[phab:T143162|T143162]], [[phab:T219604|T219604]]
* 19:05 hashar: Building Docker images for [tox-buster] Install shellcheck and cascade [integration/config] - https://gerrit.wikimedia.org/r/721881
* 15:50 James_F: Docker: Pushing composer-php72:0.2.2 and composer-php73:0.1.7 for [[phab:T235221|T235221]]
* 18:08 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 10:42 hashar: Building docker-registry.discovery.wmnet/releng/operations-puppet:0.5.5 for jbond42
* 18:07 Krinkle: Re-recreating qemu-1002 as integration-agent-qemu-1003 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T28477|T28477]]
* 01:02 Krenair: Shut down deployment-logstash2 [[phab:T238707|T238707]]
* 16:19 dpifke: Enabled TLS on Jumbo Kafka instances in deployment-prep.


== 2019-11-25 ==
== 2021-09-16 ==
* 19:12 James_F: Docker: node10-test-browser-php72-composer:0.1.2
* 22:17 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/721567 in deployment-prep, should only affect deployment-webperf11.
* 18:20 James_F: Following the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update runbook for [[phab:T239145|T239145]]
* 16:51 addshore: reload zuul for Promote branchdeploy for query-builder to test https://gerrit.wikimedia.org/r/721581 (https://phabricator.wikimedia.org/T278706)
* 13:27 Reedy: Actually reloading Zuul to deploy  https://gerrit.wikimedia.org/r/683589
* 13:01 addshore: reload zuul for https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) 1 experimental branchdeploy job


== 2019-11-24 ==
== 2021-09-15 ==
* 03:25 Krinkle: Create deployment-xhgui01 as Beta version of xhgui1001/xhgui2001 (Debian 9 Stretch, m1.small). – [[phab:T238788|T238788]], [[phab:T238098|T238098]], [[phab:T180761|T180761]]
* 22:59 dpifke: Un-cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep for now; more work needed.
* 03:14 Krinkle: Delete deployment-webperf01 instance (to be re-created as xhgui01).
* 22:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/721044 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep; should only affect deployment-webperf11.
* 08:58 Amir1: update wb_changes_dispatch set chd_disabled = 1 where chd_site = 'enwiki'; ([[phab:T290985|T290985]])


== 2019-11-22 ==
== 2021-09-14 ==
* 19:21 James_F: Zuul: Fix name of BlueSpiceUEModulePDFSubpages repo in config; add Flow to phan dependencies for CollaborationKit
* 17:17 Amir1: delete from wb_changes_dispatch where chd_db = 'enwiki'; ([[phab:T290985|T290985]])
* 19:19 James_F: Zuul: Drop the eventgate-ci repo
* 19:19 James_F: Zuul: Add Nullzero and Lens0021 to the whitelist


== 2019-11-20 ==
== 2021-09-13 ==
* 23:33 paladox: ssh -p 29418 paladox@gerrit.wikimedia.org replication start mediawiki/extensions/LDAPGroups --wait (per hauskater)
* 09:33 hashar: Castor cache: nuked files that were last changed more than six months ago to free up disk space
* 21:28 Krinkle: Create deployment-xhgui01 in Beta Cluster as m1.small/Debian 10 Buster. – [[phab:T238098|T238098]], [[phab:T180761|T180761]]
* 18:47 hauskater: Manual replication cmd issued for new repo.
* 18:23 hauskater: gerrit: Created `operations/software/netbox-extras.git`; request on mediawiki
* 17:35 James_F: JJB: Pushed wikimedia-production-pipeline-publish and trigger-wikimedia-production-pipeline-publish
* 16:37 James_F: Zuul: Add CI for wikimedia/production
* 16:37 James_F: Zuul: Add CI for mediawiki/services/eventstreams [[phab:T238658|T238658]]
* 00:53 Krenair: [[phab:T238707|T238707]] changed dateFormat (under management -> advanced settings) from 'MMMM Do YYYY, HH:mm:ss.SSS' to 'YYYY-MM-DDTHH:mm:ss', and dateFormat:tz from Browser to UTC to match old instance
* 00:51 Krenair: [[phab:T238707|T238707]] created old-logstash-beta proxy to point at old instance, created default Index Pattern on new logstash-beta
* 00:47 Krenair: [[phab:T238707|T238707]] moved kibana4/logstash-beta proxies to deployment-logstash03, copied /etc/logstash/htpasswd file


== 2019-11-18 ==
== 2021-09-10 ==
* 14:04 James_F: Zuul: Add BlueSpiceUEModulePDFRecursive / drop BlueSpiceRecursive, and fix BlueSpiceUEModuleSubpages naming
* 21:52 James_F: Created experimental integration-agent-docker-1021 for [[phab:T252071|T252071]]
* 21:48 James_F: Deleting CI agent integration-agent-docker-1001 for [[phab:T252071|T252071]]
* 21:44 James_F: Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for [[phab:T252071|T252071]]
* 21:23 James_F: Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job
* 17:41 James_F: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI
* 02:18 James_F: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12
* 02:13 James_F: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job
* 02:10 James_F: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12
* 01:58 James_F: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental
* 01:48 James_F: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices


== 2019-11-17 ==
== 2021-09-09 ==
* 15:23 James_F: Zuul: Add dependency to BlueSpiceUEModuleBookPDF
* 22:20 brennen: gitlab-ansible-test: resetting instance data
* 15:23 James_F: Zuul: Archive the OAuth2 extension [[phab:T238429|T238429]]
* 19:42 James_F: Docker: Building node<nowiki>{</nowiki>10,12<nowiki>}</nowiki>-test-browser-php80-composer for [[phab:T290651|T290651]]
* 12:24 hauskater: Deleted GitHub mirror for extension-OAuth2 refs. [[phab:T238429|T238429]]
* 10:56 hashar: Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17
* 00:57 James_F: Zuul: Enable seccheck on Wikisource; add BlueSpiceUEModuleBookPDF, BlueSpiceRecursive, BlueSpiceSubpages
* 10:37 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496
* 00:27 James_F: Zuul: Add Wikibase as dependency for Wikisource


== 2019-11-15 ==
== 2021-09-08 ==
* 21:56 James_F: Docker: Adding new images refreshed for PHP 7.2.24 [[phab:T237275|T237275]]
* 20:16 thcipriani: self +2 on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719500 to unbreak beta
* 16:45 James_F: Gearman merge queue clear thanks to hashar, all fixed.
* 18:47 brennen: contint1001 / contint2001: /srv/dev-images: git remote set-url origin 'https://gitlab.wikimedia.org/releng/dev-images.git'
* 16:19 James_F: Gearman having fun. Stuck with 1233 merge changes.
* 18:28 dduvall: Running ./fab deploy_docker to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719350
* 17:05 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355
* 16:37 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719493
* 14:55 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/714328/
* 14:05 brennen: runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners ([[phab:T289802|T289802]])


== 2019-11-13 ==
== 2021-09-07 ==
* 15:07 James_F: Zuul: Add BlueSpiceWikiExplorer
* 20:49 James_F: Marked https://gerrit.wikimedia.org/g/mediawiki/tools/cli as read-only and pointed users to GitLab.
* 20:46 brennen: migrating dev-images to https://gitlab.wikimedia.org/releng/dev-images and deactivating on gerrit
* 18:16 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 18:11 dduvall: creating 2 new jenkins jobs for deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/719311
* 16:25 James_F: Docker: Publishing node12-test-browser-php<nowiki>{</nowiki>72,80<nowiki>}</nowiki>-composer images
* 16:19 James_F: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add 4 dependencies


== 2019-11-12 ==
== 2021-09-03 ==
* 15:30 James_F: Zuul: Enable phan for Wikisource extension [[phab:T237680|T237680]]
* 23:02 Krinkle: Creating integration-agent-qemu-1002 (Debian 11 Bullseye, g3.cores8.ram24.disk20.ephemeral40.4xiops), ref [[phab:T284774|T284774]]
* 15:16 James_F: Zuul: Add BlueSpicePageFormsConnector and BlueSpiceArticlePreviewCapture
* 17:42 dduvall: deploying blubberoid:2021-09-03-160524-production to eqiad/codfw (https://gerrit.wikimedia.org/r/c/blubber/+/716519) ([[phab:T289367|T289367]])
* 11:35 hashar: Reloading Zuul for  https://gerrit.wikimedia.org/r/550048 " Archive TimezoneSelector extension" [[phab:T237894|T237894]]
* 17:36 dduvall: staging blubberoid to deploy https://gerrit.wikimedia.org/r/c/blubber/+/716519
* 11:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/550308  Add Jeblad in CI whitelist [


== 2019-11-11 ==
== 2021-09-02 ==
* 18:28 MaxSem: Nuked HHVM and php7 tags on all beta wikis [[phab:T75181|T75181]]
* 15:17 brennen: gitlab-test: testing upgrade path to 14.x
* 04:41 Reedy: apt-get clean/autoclean and purge some 3.16 kernels from deployment-sca01


== 2019-11-08 ==
== 2021-09-01 ==
* 17:46 MaxSem: Upgraded php7.2 on deployment-mwmaint01, was too old for MW
* 21:22 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/716041 in beta.
* 15:50 James_F: Zuul: Add BlueSpiceBookshelf and BlueSpiceBookshelfUI
* 16:55 urbanecm: deployment-prep: Unlock scap
* 15:50 urbanecm: deployment-prep: Lock scap again
* 15:40 urbanecm: deployment-prep: Lock scap to be able to test something
* 14:08 urbanecm: deployment-prep: Create foundationwiki ([[phab:T290164|T290164]])
* 14:07 urbanecm: urbanecm@deployment-mediawiki11:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]
* 13:58 urbanecm: urbanecm@deployment-cache-text06:~$ sudo run-puppet-agent # [[phab:T290164|T290164]]


== 2019-11-07 ==
== 2021-08-31 ==
* 22:38 James_F: Zuul: Update archived repo failure message to point to new policy page
* 21:38 dduvall: deploying new blubberoid to eqiad/codfw following successful testing in staging
* 19:33 James_F: Zuul: Mega re-ordering
* 21:35 dduvall: staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276
* 18:22 James_F: Add node10-docker to ProveIt
* 14:29 hashar: Restarting CI Jenkins for plugins upgrade
* 00:16 Reedy: rm /srv/mediawiki-staging.git/gc.log && git gc --aggressive


== 2019-11-06 ==
== 2021-08-30 ==
* 22:46 Krinkle: Fix closing pattern of "setup quibble" Console Section in Jenkins to not break on composertest jobs.
* 19:53 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta
* 20:31 brennen: Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/548941
* 19:27 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 17:43 mdholloway: deployment-prep dropped and rebuilt MachineVision extension tables
* 19:16 brennen: gitlab-test: powering off gitlab (former main test instance)
* 15:40 James_F: Upgraded mediawiki-quibble-composer-mysql-php72-docker as a test flight for [[phab:T236907|T236907]] [[phab:T237049|T237049]]
* 19:11 James_F: Zuul: [mediawiki/services/apple-search] Add postmerge publish
* 11:31 hauskater: Deleted https://github.com/wikimedia/labs-tools-discordwiki {{!}} [[phab:T237491|T237491]]
* 19:10 brennen: gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test
* 09:04 hashar: Restarted CI Jenkins
* 18:23 brennen: gitlab-test: associating floating IP to primary test box
* 09:02 hashar: Jenkins: upgrade Monitoring plugin (javamelody) to 1.80.0 + misc updates
* 16:36 James_F: Zuul: [mediawiki/services/apple-search] Remove composer-package
* 15:16 James_F: Zuul: [mediawiki/services/apple-search] Add pipeline CI for [[phab:T289224|T289224]]
* 12:20 Amir1: foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5


== 2019-11-05 ==
== 2021-08-27 ==
* 18:40 twentyafterfour: branching 1.35.0-wmf.5
* 19:24 James_F: Docker: Publish initial node14 CI images for [[phab:T267888|T267888]]
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/548827
* 16:16 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php deploymentwiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=steward  # make myself steward to remove an erroneous global lock
* 15:14 James_F: Zuul: Adding InterwikiExtracts extension
* 12:16 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php dewiki 'Lucas Werkmeister (WMDE)' --force --sysop # make myself admin for [[phab:T235152|T235152]]


== 2019-11-04 ==
== 2021-08-25 ==
* 16:52 James_F: Zuul: Make debian-glue voting for prometheus-ipsec-exporter
* 17:48 twentyafterfour: updating zuul config after deploying https://gerrit.wikimedia.org/r/714538
* 16:49 James_F: Zuul: Add coverage reporting to Cite


== 2019-11-02 ==
== 2021-08-24 ==
* 19:07 James_F: Zuul: Adding phan to ULS and GettingStarted
* 23:15 James_F: Zuul: Configure the REL1_37 test and gate pipelines [[phab:T289587|T289587]]
* 17:55 Urbanecm: deployment-prep:  mwscript initSiteStats.php --wiki=cswiki --update ([[phab:T236823|T236823]])
* 22:18 thcipriani: phab1001:sudo /srv/phab/phabricator/bin/bulk make-silent --id 2822 (releng-logspam -> unstewarded production error)
* 17:52 Urbanecm: deployment-prep: mwscript rebuildrecentchanges.php --wiki=cswiki ([[phab:T236823|T236823]])
* 20:02 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/712990
* 17:52 Urbanecm: deployment-prep: mwscript importDump.php --wiki=cswiki /tmp/Wikipedie-20191031004003.xml ([[phab:T236823|T236823]])
* 16:44 Urbanecm: deployment-prep: mwscript createAndPromote.php --wiki=cswiki --force --bureaucrat --sysop 'Martin Urbanec'
* 16:26 Urbanecm: mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=cswiki ([[phab:T237077|T237077]])
* 16:16 Amir1: Created cswiki ([[phab:T237077|T237077]])
* 00:15 paladox: ssh -p 29418 paladox@gerrit.wikimedia.org replication start --url gerrit2001.wikimedia.org --all


== 2019-11-01 ==
== 2021-08-23 ==
* 21:06 paladox: ssh -p 29418 paladox@gerrit.wikimedia.org replication start --url gerrit2001.wikimedia.org --all
* 20:37 James_F: Docker: Publish php-ast with 1.0.14 ([[phab:T289429|T289429]]) and no longer support PHP 7.0 or 7.1 (last trace!)
* 16:40 Amir1: ladsgroup@deployment-deploy01:~$ mwscript sql.php --wiki=wikidatawiki /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/sql/increase_wbxl_language.sql ([[phab:T237102|T237102]])
* 16:39 James_F: Zuul: Reloading with pipelines with _s not .s


== 2019-10-31 ==
== 2021-08-21 ==
* 20:29 tgr: importing a bunch of pages from production cswiki via importDump.php for [[phab:T236823|T236823]] (for reals now)
* 15:03 majavah: fixing deployment-prep puppet merge conflicts re: swift
* 18:38 James_F: Docker: Pushing updates for [[phab:T236907|T236907]]
* 18:06 James_F: Docker: Pushing php71:0.1.6 php72:0.2.3 php73:0.2.4
* 16:25 James_F: Zuul: [puppet] Complete the un-submodule cleanup [[phab:T230206|T230206]]
* 15:15 James_F: Zuul: Archive operations/puppet/nginx repo in CI
* 15:15 James_F: JJB: puppet-nginx-rake-docker deleted
* 14:46 hashar: Building CI containers for php7.2 php-xdebug upgrade #[[phab:T234418|T234418]]
* 13:15 hauskater: Diffusion Set default branch for Diffusion <nowiki>{</nowiki>rAPIOS<nowiki>}</nowiki> GitHub mirror from 'master' to 'develop' ('master' is not updated c. 2015; thus serving very outdated data).
* 01:14 tgr: importing a bunch of pages from production cswiki via importDump.php for [[phab:T236823|T236823]]


== 2019-10-30 ==
== 2021-08-20 ==
* 22:22 Krenair: added jdl to deployment-prep https://www.mediawiki.org/w/index.php?title=Wikimedia_Product/Analytics_Infrastructure&diff=3487111&oldid=3487110&diffmode=source
* 21:11 urbanecm: urbanecm@deployment-deploy01:/srv/mediawiki-staging/private$ rm mwblocker.log # remove weird blank log file
* 21:52 hashar: Rolled back all CI WMCS Docker to 18.06.2
* 18:56 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/713439
* 21:51 James_F: Zuul: Add Echo as phan dependency for CentralAuth
* 18:55 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 21:48 hashar: Triggering build of mwcore-phpunit-coverage-master , should use Docker 18.06.2 and be faster # [[phab:T232706|T232706]]
* 18:54 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do echo "test $i" {{!}} mwscript edit.php --wiki=<nowiki>{</nowiki>cswiki,enwiki<nowiki>}</nowiki> --user="Martin Urbanec (test $i)" --summary="test" Sandbox; done
* 21:17 hashar: Rolling back docker on CI Stretch instances  to 18.06.2~ce~3-0~debian (matching jessie)
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=enwiki
* 20:44 hashar: integration-agent-docker-1005 downgrade docker to docker-ce 18.06.2~ce~3-0~debian , the one we had on Jessie # [[phab:T236675|T236675]]
* 18:49 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki
* 19:11 hauskater: Disabled explicit mirroring of rCICF to GitHub via Diffusion URI; let Gerrit replication take over
* 18:46 urbanecm: urbanecm@deployment-mwmaint01:/srv/mediawiki/php-master$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki "Martin Urbanec (test $i)"; done
* 18:51 James_F: Zuul: Add CI for new XSL extension
* 18:40 urbanecm: urbanecm@deployment-mwmaint01:~$ for i in <nowiki>{</nowiki>1..20<nowiki>}</nowiki>; do mwscript createAndPromote.php --wiki=cswiki "Martin Urbanec (test $i)" "$password"; done # to test a feature that needs a lot of different accounts
* 16:58 James_F: Zuul: Add Echo as a phan and real dependency for MachineVision
* 16:30 majavah: restart sssd on deployment-cache-text06, [[phab:T286502|T286502]]?
* 15:56 hauskater: gerrit : created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/XLS # [[phab:T236914|T236914]]
* 16:24 majavah: deployment-prep: configure wikifunctions.beta.wmflabs.org dns zones and add to acme-chief [[phab:T284162|T284162]]
* 11:17 hashar: Bring back online a couple instances I have put offline. That was preventing some jobs to run :/
* 09:45 elukey: mediawiki mcrouter's on deployment-prep are now using deployment-memc-08 (Buster node with memcached) - to report any issue [[phab:T213089|T213089]]
* 09:44 elukey: manual rebase fix on deployment-puppetmaster03 to unblock git-sync - [[phab:T236876|T236876]]


== 2019-10-29 ==
== 2021-08-19 ==
* 19:06 paladox: updaed ownership of https://gerrit.wikimedia.org/r/#/admin/projects/openstack/horizon/deploy,access to wmcs-trusted
* 22:58 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/713612/
* 18:55 paladox: granted ldap/ops ownership of https://gerrit.wikimedia.org/r/#/admin/projects/openstack/horizon/deploy,access
* 17:26 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/546987
* 16:31 hashar: integration-agent-docker-1005 change sudo and ldap client (to sssd) # [[phab:T236675|T236675]]
* 14:34 hashar: integration-slave-jessie-* : rm /var/cache/pbuilder/base-buster-amd64.cow/var/lib/apt/lists/mirrors.wikimedia.org_debian_dists_buster_InRelease  && /usr/sbin/cowbuilder --update --basepath "/srv/pbuilder/base-buster-amd64.cow" # [[phab:T236796|T236796]]
* 14:24 hashar: integration-slave-jessie-1004 : updating buster cow image to verify the cron updater actually works: /usr/sbin/cowbuilder --update --basepath "/srv/pbuilder/base-buster-amd64.cow"
* 12:14 Urbanecm: deployment-prep: delete from user_properties where up_user=11942 and up_property like "%growth%" and up_property like "%suggested%";
* 09:45 hashar: Purging Docker images on CI instances


== 2019-10-28 ==
== 2021-08-17 ==
* 19:04 James_F: Zuul: Add CI for operations/software/httpbb [[phab:T236699|T236699]]
* 21:23 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:709831{{!}}add buster-apache2, log errors to stdio]] ([[phab:T283416|T283416]])
* 16:11 James_F: Zuul: Add BlueSpiceUserInfo
* 14:57 hashar: Updating mediawiki-quibble-api-testing-vendor-mysql-php72-docker mediawiki-quibble-api-testing-vendor-sqlite-php72-docker # [[phab:T236333|T236333]]
* 14:49 hashar: contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/546505/  sudo docker tag docker-registry.discovery.wmnet/releng/quibble-stretch-php72:0.0.40  docker-registry.discovery.wmnet/releng/quibble-stretch-php72:0.0.39-s2
* 14:47 hashar: contint1001: docker image prune
* 14:45 James_F: Zuul: Drop PHP56 jobs for SmashPig [[phab:T224906|T224906]]
* 14:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/


== 2019-10-27 ==
== 2021-08-16 ==
* 16:47 James_F: Zuul: Add seccheck and phan to GitGadgets extension
* 23:08 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/711731
* 00:18 Krinkle: krinkle@contint Replay various mediawiki/core git tags to zuul 'publish' pipeline for [[phab:T232055|T232055]]
* 00:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/546339


== 2019-10-26 ==
== 2021-08-15 ==
* 21:25 Krinkle: krinkle@contint `zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref 'refs/tags/1.30.1' --newrev 'a0b31d95610c8f415a2cc29349cdbf14c23c7d4b' --oldrev '0000000000000000000000000000000000000000'` for [[phab:T232055|T232055]]
* 17:44 James_F: Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI
* 21:20 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/546343
* 20:54 Krinkle: krinkle@contint Replay oojs/core git tags to zuul 'publish' pipeline for [[phab:T232055|T232055]]
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/546342
* 20:47 Krinkle: krinkle@contint1001$ `zuul enqueue-ref --trigger gerrit --pipeline publish --project unicodejs --ref 'refs/tags/v12.0.0' --newrev '63ab6302019d3b6b1b762b5c0094370e3030e94c' --oldrev '0000000000000000000000000000000000000000'` for [[phab:T232055|T232055]]
* 20:35 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/546340


== 2019-10-25 ==
== 2021-08-13 ==
* 20:59 paladox: created cloud/instance-puppet-dev gerrit repo per andrewbogott
* 20:09 urbanecm: Manually start `beta-update-databases-eqiad` CI job
* 18:30 brennen: Updating docker-pkg files on contint1001 for [[phab:T236333|T236333]]
* 20:06 urbanecm: deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py
* 09:49 hashar: Restarting CI Jenkins for PAM plugin upgrade
* 20:03 urbanecm: Kill beta-scap-sync-world job for the usual reason
* 09:20 hashar: Updated integration/jenkins-job-builder.git to 2.9.1 # [[phab:T236036|T236036]]
* 13:13 majavah: `mwscript extensions/CentralAuth/maintenance/importMissingLocalNames.php --wiki metawiki` on the beta cluster
* 09:19 hashar: Reconfiguring all Jenkins jobs defined by JJB due to upgrade to jjb 2.9.1 # [[phab:T236036|T236036]]


== 2019-10-24 ==
== 2021-08-11 ==
* 14:17 James_F: Zuul: Drop BlueSpiceBlog from CI, archived. [[phab:T231853|T231853]]
* 00:52 James_F: Zuul: Add Aca to the CI allow list
* 14:05 James_F: Zuul: Drop BlueSpiceUserPreferences from CI, archived.
* 00:52 James_F: Zuul: [mediawiki/extensions/SimpleCalendar] Add basic quibble CI
* 10:04 hashar: Updated integration/jenkins-job-builder.git to 2.0.3 # [[phab:T188398|T188398]]
* 10:00 hashar: Reconfiguring most Jenkins jobs due to PostBuildScript plugin change  https://gerrit.wikimedia.org/r/#/c/integration/config/+/544907/


== 2019-10-23 ==
== 2021-08-10 ==
* 18:09 paladox: create wikimedia/fundraising/analytics per [[phab:T236306|T236306]]
* 16:57 James_F: Zuul: Update e-mail address for Zabe in the allow list
* 17:00 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/545580 Standalone Selenium jobs for Wikibase # [[phab:T232759|T232759]]
* 16:06 James_F: Ran `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Wikibase` on doc1001 for [[phab:T288396|T288396]]
* 09:12 hashar: Pooling back integration-agent-docker-1006 , went offline for unknown reason?
* 09:09 hashar: CI Jenkins: downgrading AnsiColor plugin from 0.6.2 to 0.5.3  # [[phab:T236222|T236222]]
* 08:33 hashar: contint1001: backing up Zuul logs to /var/log/zuul/backup-[[phab:T236114|T236114]] so we get gerrit activities traces for [[phab:T236114|T236114]]


== 2019-10-22 ==
== 2021-08-09 ==
* 13:01 James_F: Limited pywikibot to one concurrent job in Wikimedia TravisCI to avoid resource exhaustion
* 21:02 urbanecm: Remove hanging beta-scap-sync-world job in CI to unblock beta auto-updates


== 2019-10-21 ==
== 2021-08-07 ==
* 19:00 Krinkle: Pruning old image from contint1001: 0.0.38                        {{Gerrit|073a49221d74}}
* 00:48 James_F: Docker: Publish quibble-buster-php73-coverage 1.1.1 for [[phab:T287918|T287918]].
* 19:00 Krinkle: Pruning old image from contint1001: 0.0.38-s1                    {{Gerrit|2b39d2f00de3}}
* 00:29 James_F: Zuul: Add skin-coverage jobs to all Wikimedia production skins [[phab:T287918|T287918]]
* 18:42 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/544992
* 00:27 James_F: Zuul: Provide a skin-coverage template [[phab:T287918|T287918]]
* 18:38 hashar: Jenkins: upgraded logstash plugin to 2.3.1 , overriding a custom build snapshot of 1.2.1.  # [[phab:T78705|T78705]]
* 12:06 hashar: Draining merger:merge request. On contint2001  : sudo chown -R root:root /srv/zuul/git/integration/config
* 12:04 hashar: Draining merger:merge request. On contint1001  : sudo chown -R root:root /srv/zuul/git/integration/config
* 10:23 hashar: Restored integration/config on zuul-mergers : sudo chown -R zuul:zuul /srv/zuul/git/integration/config
* 10:15 hashar: contint1001 / contint2001 : marking integration/config zuul merger repo readonly:  sudo chown -R root:root /srv/zuul/git/integration/config
* 09:48 hashar: jenkins-job-builder : remove cherry-pick which is no more needed (to support multiple macro in conditional-step builder https://review.openstack.org/#/c/471030/1 ) == rollback from {{Gerrit|a06d173e98c}} to tag 2.0.2
* 09:12 hashar: update Quibble jobs to container 0.0.39-s1
* 09:04 hashar: Restarting CI Jenkins for plugin upgrades


== 2019-10-20 ==
== 2021-08-06 ==
* 21:28 Krinkle: Click "Disable Publishing" for extensions/OAuth and extensions/RevisionSlider in Phabricator, [[phab:T143162|T143162]] ,[[phab:T219604|T219604]]
* 23:53 James_F: Docker: Publishing quibble-buster-php73-coverage 1.1.0 for [[phab:T287918|T287918]]
* 09:38 James_F: Zuul: Run phan job for Translate [[phab:T224756|T224756]]
* 23:47 James_F: Zuul: [mediawiki/skins/Mirage] Not a production skin; move to right section
* 09:37 James_F: Zuul: Add BlueSpiceReminder as dependency for BlueSpiceExpiry


== 2019-10-19 ==
== 2021-08-05 ==
* 20:48 Krinkle: Permissions are broken again on doc1001 for /srv/docroot (various files wrongly chmoded as 660 or wrongly owned by nobody, or hashar). Unable to git-pull.
* 22:26 brennen: gitlab: setting CI access on all repos to "Project members only", per https://www.mediawiki.org/wiki/GitLab/Policy#Permissions - may need revisited depending on effects
* 22:04 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/710108
* 15:10 hashar: integration: sudo cumin --force 'name:docker' 'docker container prune -f && docker image prune -f'
* 15:09 hashar: integration-agent-docker-1020: docker container prune && docker image prune
* 15:02 hashar: Building Docker image helm-linter for https://gerrit.wikimedia.org/r/c/integration/config/+/710276


== 2019-10-18 ==
== 2021-08-04 ==
* 21:26 paladox: create cloud/instance-puppet repo per andrewbogott request
* 22:53 bd808: Updated composer-github-oauthtoken in Jenkins config to us a newer personal access token from GitHub per notices about https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
* 20:34 hashar: integration: sudo cumin --force 'name:docker' 'docker image prune -f'
* 17:49 brennen: gitlab-test: testing upgrade to 13.12.9
* 20:34 hashar: integration: sudo cumin --force 'name:docker' 'docker container prune -f'
* 16:41 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/710066 # [[phab:T288111|T288111]]
* 18:42 James_F: Docker: Rebuilding the world onto composer-php70:0.1.1
* 15:13 thcipriani: puppet fixed on deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki>
* 18:32 James_F: Zuul clean sync as the last patch was seemingly already deployed?
* 15:08 thcipriani: rebase deployment-puppetmaster04:labs/private causing deployment-deploy<nowiki>{</nowiki>01,03<nowiki>}</nowiki> failure for...¯\_(ツ)_/¯
* 18:19 James_F: Docker: Building lintr:0.2.1 and php70-compile:0.1.4
* 17:09 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/544219
* 13:51 hashar: Building Quibble 0.0.39 CI containers
* 13:44 hashar: Tag Quibble 0.0.39 @ {{Gerrit|a1e0b85}} # [[phab:T235023|T235023]] [[phab:T235564|T235564]]
* 12:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/544186


== 2019-10-17 ==
== 2021-08-03 ==
* 20:48 hauskater: Arm eventlogging key via keyholder for beta cluster following the instructions on Wikitech for [[phab:T235674|T235674]]
* 21:12 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:698880{{!}}add buster php images]]
* 17:10 James_F: Zuul: OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments
* 16:33 James_F: Zuul: Moving OOUI from node 6 to node 10 job [[phab:T235570|T235570]]
* 16:11 James_F: Docker: Pushing node10-test-browser-php72-composer:0.1.1


== 2019-10-16 ==
== 2021-08-02 ==
* 22:42 thcipriani: deployment-deploy02:sudo vipw to remove mwdeploy user
* 23:27 Reedy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/709563
* 22:10 James_F: https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php72-docker/8502/console got stuck in quibble's composer install step for half an hour; manually aborted. :-(
* 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709562
* 18:27 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move to PHP72+
* 23:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709558
* 18:26 James_F: Zuul: Activate tests for new repo BlueSpiceDistributionConnector
* 22:54 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709555 https://gerrit.wikimedia.org/r/709556
* 17:44 James_F: Marking deployment-deplog01 offline temporarily for [[phab:T235674|T235674]]
* 21:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/709543
* 08:57 elukey: created deployment-memc08 in deployment-prep as memcached test host for Buster - [[phab:T213089|T213089]]


== 2019-10-15 ==
== 2021-07-30 ==
* 21:55 James_F: Docker: Provide node10-test-browser-php72-composer [[phab:T235570|T235570]]
* 21:27 dduvall: "Total reclaimed space: 141.4GB" on releases1002 following docker prune
* 19:40 James_F: Created the wmf.N milestones for 1.35.0 out until wmf.15 for ReleaseTaggerBot: https://phabricator.wikimedia.org/project/subprojects/4306/
* 21:24 dduvall: running `docker system prune -af` on releases1002
* 19:19 James_F: Zuul: Enable Phan for FlaggedRevs [[phab:T224769|T224769]]
* 18:55 hashar: Jenkins: deleted no more used jobs operations-mw-config-hhvm-composer-lint-docker and  operations-mw-config-hhvm-composer-test-docker
* 18:01 James_F: Zuul: Enable Phan for OpenStackManager [[phab:T224765|T224765]]
* 17:33 James_F: Zuul: Enable Phan for ReadingLists [[phab:T224762|T224762]]
* 16:49 James_F: Zuul: Phan dependencies for ReadingLists and OpenStackManager.
* 15:36 James_F: Docker: Rebuild images for PHP 7.2.22 [[phab:T232165|T232165]]
* 13:04 hashar: Building docker-registry.wikimedia.org/releng/tox-homer:0.1.0 for XioNoX / operations/software/homer
* 12:08 paladox: create group “bluespice-admin” in gerrit per [[phab:T234224|T234224]]
* 09:31 hashar: CI Jenkins: uninstalling BlueOcean meta plugin to let us uninstall plugins we do not care about # [[phab:T233695|T233695]]


== 2019-10-14 ==
== 2021-07-29 ==
* 11:40 hashar: contint1001: deleting old Quibble containers
* 22:12 jeena: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/708596
* 09:34 hashar: Upgraded CI jobs to Quibble 0.0.38
* 14:06 brennen: gerrit: added ldap/ops to ownership of operations/gitlab-ansible
* 09:32 hashar: Deleting old release-quibble* jobs from Jenkins. They are no more used.
* 13:21 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/708764 "Add michaelcochez to whitelist users"


== 2019-10-12 ==
== 2021-07-28 ==
* 16:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/542690
* 17:53 andrewbogott: rebooting deployment-logstash03 as it's in an inconsistent config state
* 14:44 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/708536


== 2019-10-11 ==
== 2021-07-27 ==
* 17:58 Krinkle: Click "Disable Publishing" on extension-Elastica Phab mirror, [[phab:T235233|T235233]], [[phab:T143162|T143162]]
* 16:55 dduvall: creating new gitlab runner instance runner-1002 for testing
* 16:45 James_F: Zuul: Add dependency "BlueSpiceExtendedSearch" to BlueSpiceStatistics
* 16:32 hashar: cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches
* 16:03 paladox: ssh -p 29418 gerrit.wikimedia.org gerrit create-project cloud.git --permissions-only  - [[phab:T235279|T235279]]
* 15:59 paladox: ssh -p 29418 gerrit.wikimedia.org gerrit create-project cloud.git --permissions-only --description="Container for all other cloud/* projects for permissions, etc." - [[phab:T235279|T235279]]
* 13:32 hashar: Build docker-registry.discovery.wmnet/releng/tox-pyspark:0.5.1 for ebernhardson
* 01:37 Reedy: Killed the job that was waiting for executor for beta-scap-eqiad, it's now running after 4 hours not


== 2019-10-10 ==
== 2021-07-26 ==
* 21:08 Krenair: Ran apt upgrades on deployment-mediawiki-0[79] to bring beta cluster back online, MW started requiring a version higher than we were running
* 20:45 brennen: runner-1001: installed docker & gitlab-runner, registered runner-1001 to the gitlab instance for pipeline experimentation ([[phab:T287279|T287279]])
* 20:52 James_F: Zuul: Make mediawiki-quibble-api-testing* voting [[phab:T230340|T230340]]
* 18:10 dduvall: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 16:22 James_F: Zuul: Also enable phan and phan-seccheck on DiscussionTools [[phab:T234481|T234481]]
* 18:00 dduvall: creating 2 new jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/707879
* 16:15 James_F: Zuul: Add CI configuration for new DiscussionTools repo [[phab:T234481|T234481]]
* 08:24 hashar: jjb: update jobs to Quibble 1.0.1  # https://gerrit.wikimedia.org/r/c/integration/config/+/708038
* 10:35 hashar: gerrit: added onimisionipe and gehel to wikidata-query-gui groups # [[phab:T235159|T235159]]


== 2019-10-09 ==
== 2021-07-23 ==
* 20:01 hashar: gerrit set-account owl --active  # [[phab:T234328|T234328]]
* 19:26 brennen: gitlab-runners: launched runner-1001, g3.cores8.ram36.disk20 to install baseline experimental runner ([[phab:T287279|T287279]])
* 19:55 urandom: Upgrading deployment-sessionstore01.deployment-prep.eqiad.wmflabs to Cassandra 3.11.4 -- [[phab:T200803|T200803]]
* 14:08 hashar: Building Docker images for quibble 1.0.1
* 19:49 urandom: Upgrading deployment-restbase02.deployment-prep.eqiad.wmflabs to Cassandra 3.11.4 -- [[phab:T200803|T200803]]
* 13:45 hashar: Tag Quibble 1.0.1 @ {{Gerrit|5a2548699a}} # [[phab:T287001|T287001]]
* 19:46 urandom: Upgrading deployment-restbase01.deployment-prep.eqiad.wmflabs to Cassandra 3.11.4 -- [[phab:T200803|T200803]]
* 17:52 hashar: Build Quibble 0.0.38 CI Docker images
* 17:52 hashar: Tagged Quibble 0.0.38 @ {{Gerrit|d22e0f55b}} earlier today to fix $wgServer # [[phab:T235023|T235023]]
* 15:55 James_F: Docker: [mediawiki-phan] Avoid unbound variable -> 0.5.3
* 07:37 hashar: Build Quibble 0.0.37 docker containers
* 07:28 hashar: Tag Quibble 0.0.37 @ {{Gerrit|387d33c13}}


== 2019-10-08 ==
== 2021-07-21 ==
* 22:13 James_F: Zuul: Actually restrict mwcore-codehealth-master-non-voting and mwext-codehealth-master-non-voting to master
* 21:06 brennen: gitlab1001: running ansible for logging typo fix ([[phab:T274462|T274462]])
* 20:19 hashar: Tag Quibble 0.0.36 @ {{Gerrit|4b46156}} # [[phab:T181942|T181942]] [[phab:T190829|T190829]] [[phab:T199939|T199939]] [[phab:T230340|T230340]] [[phab:T233140|T233140]] [[phab:T233143|T233143]]
* 20:36 dancy: Newest scap deployed to beta cluster
* 16:27 James_F: Docker: [mediawiki-phan] Fix phan version validation
* 19:06 brennen: gitlab1001: running ansible to deploy nginx logging and status changes ([[phab:T274462|T274462]], [[phab:T275170|T275170]])
* 16:46 dancy: restarting Gerrit to fix plugins
* 16:07 dancy: Updating plugins on releases-jenkins
* 15:00 urbanecm: deployment-prep: Change password for `Martin Urbanec` at votewiki


== 2019-10-07 ==
== 2021-07-20 ==
* 20:59 hashar: deployment-prep: on a few mediawiki deployment targets:  sudo rm -fR /srv/mediawiki/.~tmp~/
* 23:17 brennen: removed erroneous listing of myself as a train deployer for this week from deployment schedule, added hashar ([[phab:T281156|T281156]])
* 20:32 brennen: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/540701 is merged and parsoid dev images are being published
* 18:39 hashar: Rolling back Jenkins jobs from Quibble 1.0.0 to 0.0.47   # [[phab:T287001|T287001]]
* 20:22 thcipriani: remove shadow mwdeploy user from a bunch of other deployment-prep hosts to (hopefully) fix beta-scap-eqiad
* 19:29 thcipriani: deployment-deploy01:sudo rm -rf  /srv/mediawiki/.~tmp~
* 19:29 James_F: Docker: [mediawiki-phan] Fix syntax error in shell scripts
* 18:50 thcipriani: deployment-deploy01 vipw && vipw -s to remove local mwdeploy user (use user 603 from ldap instead)
* 17:39 James_F: Zuul: Move mediawiki/tools/release to php72+
* 17:35 James_F: Zuul: Add phan dependencies for FlaggedRevs [[phab:T224769|T224769]]
* 17:25 James_F: Zuul: [mediawiki/tools/phan,PerfCheckPlugin] Move to PHP72+
* 16:51 James_F: Zuul: Add phan dependencies for Translate [[phab:T224756|T224756]]
* 15:34 James_F: Docker: [mediawiki-phan] Allow requiring phan like normal in composer.json [[phab:T220589|T220589]]
* 15:13 James_F: Docker: [mediawiki-phan] Drop back-compatibility hacks for old phan
* 13:37 hashar: integration/quibble : grants kostajh +2 (in addition to awight and hashar)
* 08:33 hashar: gerrit:  `gerrit index start changes --force`  # [[phab:T233989|T233989]]
* 08:32 hashar: gerrit:  `gerrit index start changes`   # [[phab:T233989|T233989]]


== 2019-10-04 ==
== 2021-07-19 ==
* 21:13 hauskater: gerrit: Ran replication start mediawiki/extensions/WikiSEO --wait # New repository
* 18:42 brennen: gerrit1001: ran puppet; noted that quotes were added to jvm configuration values
* 19:34 marxarelli: re-running `mediawiki/tools/release/make-release/branch.py --bundle '*' ...` due to https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/540953 cc: [[phab:T232024|T232024]]
* 19:17 Krinkle: Fix Jenkins Console section "setup quibble prepare" pattern to account for recent changes on "mwext-php72-phan-docker" which was eating the pattern incorrectly
* 18:55 marxarelli: branching REL1_34 for remaining extensions/skins and core, `mediawiki/tools/release/make-release/branch.py REL1_34 --bundle '*' --core --branchpoint master --core-version 1.34.0` (note: core branchpoint is overridden in settings.yaml at base of wmf/1.34.0-wmf.25) cc: [[phab:T232024|T232024]]
* 18:34 marxarelli: branching REL1_34 for all wmf-deployed extensions, `mediawiki/tools/release/make-release/branch.py REL1_34 --bundle wmf --branchpoint wmf/1.34.0-wmf.25 --core-version 1.34.0` cc: [[phab:T232024|T232024]]
* 17:45 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/540699 for parsoid publication pipeline
* 07:44 hashar: Adding labels to Jenkins agent "Docker blubber m4executor" on integration-agent-jessie-docker-1001  {{!}} Jessie based VM to potentially compare speed of jobs with the Stretch based ones


== 2019-10-03 ==
== 2021-07-18 ==
* 21:03 James_F: Docker: Bump things over to use php70 which adds intl
* 08:48 majavah: set shared_acme_certificates: <nowiki>{</nowiki><nowiki>}</nowiki> on deployment-prep shared hiera, [[phab:T276653|T276653]]
* 20:57 James_F: Docker: Rename php to php70 and add intl
* 19:12 James_F: [wikimedia/fundraising/crm] AAdd experimental php70 job [[phab:T230446|T230446]]
* 18:25 James_F: Zuul: [mediawiki/tools/codesniffer] Move to PHP7.2+
* 11:09 hashar: integration-castor03: pruning HHVM jobs caches: rm -fR /srv/jenkins-workspace/caches/*/*/*hhvm* # [[phab:T234384|T234384]]
* 11:07 hashar: jenkins: removing global environment variable: HHVM_REPO_CENTRAL_PATH=$WORKSPACE/central.hhbc {{!}} https://integration.wikimedia.org/ci/configure {{!}} [[phab:T234384|T234384]]


== 2019-10-02 ==
== 2021-07-16 ==
* 21:06 James_F: Zuul: Misc HHVM test clean-ups.
* 21:01 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/698774
* 20:37 James_F: Zuul: Migrate fundraising tests from HHVM to PHP70 [[phab:T234384|T234384]]
* 17:44 brennen: Try #2: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 19:16 James_F: Zuul: Drop all HHVM testing except for Fundraising [[phab:T234384|T234384]]
* 17:12 hashar: Tag Quibble 1.0.0 @ {{Gerrit|a13d133f7d1}} # [[phab:T286187|T286187]] [[phab:T280506|T280506]] [[phab:T90875|T90875]] [[phab:T218534|T218534]] [[phab:T227352|T227352]]
* 19:01 James_F: Zuul: Drop -nohhvm jobs, no longer needed
* 18:55 James_F: Zuul: Drop HHVM testing from all quibble jobs [[phab:T234384|T234384]]
* 17:57 hashar: Fixed puppet ssl issue on integration-agent-docker-1004 and integration-agent-docker-1008 ( rm -fR /var/lib/puppet/ssl )
* 03:07 Krinkle: Fix "setup quibble mw-install" Jenkins Console section to not eat all output on new quibble-selenium jobs ([[phab:T232759|T232759]])


== 2019-10-01 ==
== 2021-07-14 ==
* 15:17 hashar: Deleted integration-agent-docker-1015 (was too slow)
* 22:47 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:704162{{!}}Add a Swift language dev image for CI testing]] ([[phab:T284195|T284195]])
* 15:17 hashar: Pooled unused Jessie agent https://integration.wikimedia.org/ci/computer/integration-agent-jessie-docker-1001/
* 21:14 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in beta puppet again, should only affect deployment-webperf12.
* 13:55 hashar: Gerrit: disabled bot account "owl" , see whether that helps on the JVM gc issue [[phab:T234328|T234328]]
* 13:43 hashar: Gerrit: disabling bot account "owl" , see whether that helps on the JVM gc issue
* 11:13 hashar: shutting down integration-castor03 # [[phab:T232646|T232646]]


== 2019-09-30 ==
== 2021-07-13 ==
* 20:47 James_F: Zuul: Add phan job for WikibaseQualityConstraints [[phab:T224773|T224773]] and some phan dependencies for WikimediaMaintenance [[phab:T224770|T224770]].
* 23:22 dpifke: Re-cherry-picking newer https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 patch in deployment-prep.  Should only affect deployment-webperf12.
* 19:03 hashar: Update jobs wmf-quibble-core-vendor-mysql-hhvm-docker wmf-quibble-core-vendor-mysql-php72-docker wmf-quibble-vendor-mysql-hhvm-docker wmf-quibble-vendor-mysql-php72-docker {{!}} to stop running selenium tests {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/539630/
* 19:29 James_F: Manually deleted Jade extension coverage from doc1001 for [[phab:T281430|T281430]]
* 19:01 hashar: Reloading Zuul for  https://gerrit.wikimedia.org/r/539630
* 19:24 James_F: Zuul: [mediawiki/extensions/Jade] Mark repo as archived [[phab:T281430|T281430]]
* 18:53 hashar: Create jobs wmf-quibble-selenium-hhvm-docker wmf-quibble-selenium-php72-docker {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/539630/
* 16:33 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/703912 in deployment-prep puppet.  Should only affect deployment-webperf12.
* 15:23 James_F: Zuul: Added TheSandDoctor to the CI whitelist.
* 10:22 hashar: gerrit: pushed upstream tags for plugins/gitiles  # [[phab:T278990|T278990]]
* 13:10 hashar: integration-castor03 : sudo mkdir -p /usr/local/lib/python2.7/dist-packages
* 09:13 James_F: Zuul: [mediawiki/extensions/Report] Add basic quibble CI job
* 12:49 hashar: Fixed cumin on integration project # [[phab:T234203|T234203]]
* 08:24 hashar: Updated operations/software/gerrit branches to 3.2.11 # [[phab:T278990|T278990]]
* 11:08 hashar: Restarting Docker on CI agents to clear out some docker/iptables oddity # [[phab:T234197|T234197]]
* 07:46 hashar: Wiping all Docker images from contint2001l
* 10:59 hashar: Restarted Docker on integration-agent-docker-1001 [[phab:T234197|T234197]]


== 2019-09-28 ==
== 2021-07-12 ==
* 18:35 hauskatze: gerrit: Ran gerrit gc labs/tools/stewardbots --show-progress --aggressive
* 18:45 majavah: upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, [[phab:T286506|T286506]]
* 18:43 majavah: deployment-cache-text06 varnish not starting, [[phab:T286506|T286506]], causing an outage on text traffic on deployment-prep
* 18:23 majavah: hard reboot deployment-cache-text06 once I got in using a root ssh key
* 16:15 majavah: hard reboot deployment-cache-text06, refusing to let me log in and console full of errors
* 14:48 Amir1: ran $ ./jjb-update 'wikidata-query-gui-build' ([[phab:T286479|T286479]])
* 14:44 majavah: fix merge conflict on deployment-puppetmaster04
* 13:19 James_F: Zuul: Add Voidwalker to the CI allow list
* 13:19 James_F: Zuul: Add R4356th to the CI allow list
* 13:05 James_F_: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [[phab:T286207|T286207]]


== 2019-09-27 ==
== 2021-07-09 ==
* 21:01 hashar: [[phab:T233989|T233989]] I have failed to run (must be run offline): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --list
* 14:47 bd808: Slienced puppet failure alert for deployment-parsoid12 for 7 days ([[phab:T286375|T286375]])
* 20:01 hashar: Marking integration-agent-docker-1015 offline due to cloudvirt1004 being wayyyyy too slow [[phab:T223971|T223971]]
* 00:18 bd808: Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days ([[phab:T286358|T286358]])
* 18:23 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/539447


== 2019-09-26 ==
== 2021-07-08 ==
* 23:42 brennen: Updating dev-images docker-pkg files on contint1001 for stretch-php72-fpm-apache2 publication
* 08:21 majavah: kick stuck puppet agent on deployment-logstash04
* 22:47 James_F: Zuul: Re-enable trigger-mediawiki-pipeline-dev now that we're ready [[phab:T218360|T218360]]
* 12:43 hashar: Pooling integration-agent-docker-1015  and integration-agent-docker-1016
* 11:48 hashar: Pooling integration-agent-docker-1014
* 11:45 hashar: Pooling integration-agent-docker-1013


== 2019-09-25 ==
== 2021-07-07 ==
* 23:01 andrewbogott: moving deployment-mwmaint01 and deployment-ircd to cloudvirt1021
* 18:10 majavah: create shellbox.svc.deployment-prep.eqiad1.wikimedia.cloud. record as a CNAME to deployment-shellbox instance [[phab:T286298|T286298]]
* 20:14 hashar: Pooled integration-agent-docker-1007
* 19:36 hashar: Deleting integration-agent-docker-1007 it is too slow ( [[phab:T223971|T223971]] )
* 15:15 andrewbogott: moving deployment-snapshot01 to cloudvirt1021
* 15:02 andrewbogott: moving deployment-dumps-puppetmaster02 to cloudvirt1021
* 11:01 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php enwiki --force --sysop 'Lucas Werkmeister (WMDE)' # making myself admin for Wikidata Bridge testing
* 07:14 hashar: Restarted memcached on deployment-memc04, deployment-memc05 , deployment-memc06, deployment-memc07, deployment-imagescaler01 # [[phab:T232796|T232796]]


== 2019-09-24 ==
== 2021-07-06 ==
* 15:27 hashar: deployment-logstash2: removing class profile::elastalert , does not exist and breaks puppet # [[phab:T233134|T233134]]
* 01:16 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/703219


== 2019-09-23 ==
== 2021-07-05 ==
* 20:42 hashar: Added integration-agent-docker-1012 # [[phab:T226233|T226233]]
* 15:02 Amir1: deployed 703212 ([[phab:T286058|T286058]])
* 20:38 hashar: Added integration-agent-docker-1011 # [[phab:T226233|T226233]]
* 20:37 hashar: Added integration-agent-docker-1010 # [[phab:T226233|T226233]]
* 20:23 hashar: Added integration-agent-docker-1009 # [[phab:T226233|T226233]]
* 19:15 hashar: Buidling Docker image releng/gerrit:0.1.0-s2 (bazel upgrade)
* 19:11 hashar: Disconnected and reconnected integration-agent-docker-1006 to have jenkins-deploy to be a meember of docker group
* 19:09 hashar: Added integration-agent-docker-1008 # [[phab:T226233|T226233]]
* 19:06 hashar: Added integration-agent-docker-1007 # [[phab:T226233|T226233]]
* 19:03 hashar: Buidling Docker image releng/bazel:0.3.0
* 18:43 hashar: Added integration-agent-docker-1006 # [[phab:T226233|T226233]]


== 2019-09-21 ==
== 2021-07-02 ==
* 19:36 James_F: Zuul: [WikibaseMediaInfo] Add phan dependency [[phab:T224772|T224772]]
* 17:32 brennen: gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/701068 ([[phab:T274463|T274463]])
* 09:49 James_F: Zuul: [pywikibot/core] Add deeptest to gate-and-submit section


== 2019-09-20 ==
== 2021-06-30 ==
* 21:47 James_F: jjb: Deleting mediawiki-tarball-builder; follow-up can happen in {{Gerrit|I52d40ae453}}
* 22:42 brennen: gitlab: published https://gitlab.wikimedia.org/releng/gitlab-settings
* 21:45 James_F: jjb: All executors moved from DebianJessieDocker to Docker
* 22:39 brennen: gitlab: creating people, people/wmf, and people/wmf/release-engineering groups; mandating 2fa for people/wmf
* 21:35 James_F: jjb: Dropping differential-docker-test
* 17:57 thcipriani: restart ci jenkins following upgrade
* 21:34 James_F: jjb: Dropping wmf-sonar-*
* 17:54 thcipriani: restart releases-jenkins following upgrade
* 21:33 James_F: jjb: Dropping wikimedia-fundraising-crm-composer-php56-docker
* 12:36 James_F: Zuul: [mediawiki/core] Drop PHP70/71 testing for REL1_31
* 21:32 James_F: jjb: Dropping puppet-zookeeper-rake-docker
* 21:31 James_F: jjb: Dropping mwgate-npm-node-6-docker
* 21:31 James_F: jjb: Dropping php-compile-hhvm-jessie-docker
* 21:30 James_F: jjb: Dropping Schedule a Build
* 19:41 James_F: Zuul: Drop ortiz-rlang (merged but not deployed)
* 19:41 James_F: Zuul: [JsonConfig] Add phan dependency and phan
* 15:16 hashar: Pooled integration-agent-docker-1005 based on Stretch # [[phab:T226233|T226233]]
* 15:14 hashar: Pooled integration-agent-docker-1004 based on Stretch # [[phab:T226233|T226233]]
* 14:47 hashar: Pooled integration-agent-docker-1002 integration-agent-docker-1003 based on Stretch # [[phab:T226233|T226233]]
* 13:23 hashar: Deleting Jenkins agent integration-r-lang-01 (unused) # [[phab:T153856|T153856]]
* 13:14 hashar: Pooled integration-agent-docker-1001 based on Stretch # [[phab:T226233|T226233]]
* 01:12 James_F: Zuul: [PropertySuggester] Add phan
* 01:00 James_F: Zuul: [PropertySuggester] Add phan dependency [[phab:T224774|T224774]]


== 2019-09-19 ==
== 2021-06-29 ==
* 16:52 hashar: Building releng/helm-linter:0.2.0 [[phab:T233291|T233291]]
* 21:45 urbanecm: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/updateMenteeData.php --wiki=cswiki # [[phab:T285811|T285811]]
* 09:31 hashar: Upgrading Jenkins plugins on CI and release hosts


== 2019-09-18 ==
== 2021-06-28 ==
* 19:22 hashar: Keeping https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/5393/ forever for [[phab:T233143|T233143]]
* 12:06 majavah: revert manual scap downgrade on deployment-mediawiki11
* 18:54 James_F: Zuul: Drop extension-seccheck-non-voting and skin-seccheck-non-voting, unused
* 11:59 majavah: downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for [[phab:T285125|T285125]]
* 18:47 James_F: [[phab:T231928|T231928]] terminating endlessly-running attempts to build kartotherian and tilerator
* 17:19 James_F: Zuul: [[phab:T231928|T231928]] Migrate kartotherian to custom pipeline jobs
* 07:43 hashar: Raised integration-castor03 egress trafic shaping from 50mbps to 100mbps # [[phab:T213223|T213223]]
* 07:36 hashar: Raised integration-castor03 egress trafic shaping from 50mbps to 100mbps # [[phab:T232644|T232644]]


== 2019-09-17 ==
== 2021-06-24 ==
* 20:52 brennen: Updating dev-images docker-pkg files on contint1001 for new restbase image
* 01:11 Krinkle: deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)
* 18:32 James_F: jjb: Updated the rest of the quibble jobs to 0.0.35.
* 18:14 James_F: jjb: Upgrading wmf-quibble-vendor-mysql-php72-docker
* 17:56 James_F: jjb: Upgrading mediawiki-quibble-vendor-mysql-php72-docker
* 17:51 James_F: jjb: Updating parsoidsvc-parsertests-docker
* 17:49 James_F: jjb: Updating mwext-phpunit-coverage*
* 17:40 James_F: jjb: Updated wikibase-client-docker, wikibase-repo-docker
* 17:36 James_F: jjb: Updating mediawiki-fresnel-patch-docker
* 17:30 James_F: jjv: Updated mwext-php72-phan-docker, mwext-php72-phan-seccheck-docker, mwskin-php72-phan-docker, mwskin-php72-phan-seccheck-docker
* 17:29 James_F: jjb: Updated mwext-codehealth-patch and mwext-codehealth-master-non-voting
* 17:16 James_F: jjv: Updating mwcore-phpunit-coverage-master,mwcore-phpunit-coverage-patch
* 16:58 James_F: jjb: Updating mwcore-codehealth-patch experimentally
* 16:51 James_F: jjb: Experimentally updated mwext-codehealth-master-non-voting
* 16:46 James_F: jjb: Experimentally updated mwext-php72-phan-seccheck-docker-non-voting
* 16:04 hashar: Quibble 0.0.35 containers for CI have been build.
* 15:40 hashar: Building docker images for Quibble 0.0.35
* 15:32 hashar: Tagging quibble 0.0.35 {{Gerrit|48f6b1d8db9eed8572f255fe86f920753149153c}} # [[phab:T225730|T225730]] [[phab:T231862|T231862]] [[phab:T219694|T219694]]
* 14:31 hashar: Raise egress filter trafic shaping on integration-castor03 from 240Mbit to 400Mbit ( [[phab:T232644|T232644]] / [[phab:T188375|T188375]] )
* 12:11 hashar: Remove Jenkins CI webproxy configuration , that breaks jobs using the git plugin!
* 11:26 hashar: Configured CI jenkins to use a webproxy for the update center due to mysterious "connection reset" when downloading plugins
* 09:24 hashar: Manually triggered https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9920/ (switch to php7.2)
* 09:16 hashar: Rebuilding releng/doxygen container  to use php7.2
* 09:15 hashar: On doxygen jobs, change container to use php7.2 instead of php7.0
* 08:19 hashar: Building docker container releng/quibble-stretch-bundle:0.0.34-2  (switch to php7.2)
* 00:29 James_F: Zuul: Dropping php70 and php71 from quibble jobs for master and REL1_34 [[phab:T228342|T228342]]


== 2019-09-16 ==
== 2021-06-23 ==
* 23:54 James_F: Zuul: Initial CI for WebToolsManager
* 16:32 Reedy: beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick
* 18:52 hashar: Gerrit: tweaked members of the Analytics group # [[phab:T233026|T233026]]
* 18:51 hashar: Gerrit: tweaked members of the Analytics group # [[phab:T233026|T233026]]
* 16:00 James_F: Zuul: Restore maps/kartotherian/package to unarchived CI-less midden


== 2019-09-14 ==
== 2021-06-22 ==
* 20:45 Krenair: shifted deployment-cache-upload05 to using unified cert from acme-chief
* 23:03 jeena: increasing releases-jenkins executors from 2 to 4
* 21:25 jeena: Updating dev-images docker-pkg files on primary contint [[phab:T273682|T273682]]
* 09:31 hashar: Removed DISPLAY=:94 being set from Zuul for most jobs https://gerrit.wikimedia.org/r/c/integration/config/+/700857/


== 2019-09-13 ==
== 2021-06-21 ==
* 23:40 Krinkle: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
* 23:27 Krinkle: Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching.
* 16:19 James_F: Zuul: Add initial jobs for mediawiki/tools/api-testing [[phab:T230340|T230340]]
* 12:39 hashar: integration-slave-docker-1043 : killed stall container # [[phab:T232706|T232706]]


== 2019-09-12 ==
== 2021-06-19 ==
* 23:37 James_F: Zuul: Define pipelines for REL1_34 [[phab:T232025|T232025]]
* 13:44 majavah: remove deployment-deploy02 [[phab:T278689|T278689]]
* 22:19 hashar: integration-slave-jessie-1002: rm /srv/pbuilder/build/* /srv/pbuilder/aptcache/*/*
* 08:05 majavah: creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host [[phab:T283013|T283013]]
* 22:16 hashar: Pooling back integration-slave-jessie-1002 in Jenkins ( /srv was 95% full )
* 22:15 hashar: integration-slave-jessie-1002: apt-get clean
* 21:45 James_F: Temporarily taking deployment-deploy01 offline as CI is stuck
* 20:54 James_F: Zuul: Drop operations-mw-config-hhvm-composer-test-docker, the PHP72 one suffices
* 20:31 James_F: Zuul: Add seccheck to PageNotice
* 16:36 halfak: deploying ores {{Gerrit|7d45b80}}
* 15:53 hauskatze: Phabricator: Switched articlequality, drafttopic, draftquality and editquality Diffusion (ORES project) to mirror to Gerrit via SSH/K19 refs. [[phab:T232661|T232661]]
* 08:14 hashar: Marking integration-slave-docker-1043 offline , triggering mwcore-phpunit-coverage-master again # [[phab:T232706|T232706]]


== 2019-09-11 ==
== 2021-06-16 ==
* 22:31 Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - [[phab:T143162|T143162]]
* 21:47 James_F: Zuul: Install CI for mediawiki/libs/NormalizedException [[phab:T284732|T284732]]
* 21:06 James_F: Zuul: Reload to add operations-mw-config-php72-composer-test-docker and operations-mw-config-hhvm-composer-lint-docker
* 05:44 majavah: restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert
* 14:27 hashar: puppet cert clean integration-agent-docker-1001.integration.eqiad.wmflabs # [[phab:T232619|T232619]] {{!}} might unbreak puppet


== 2019-09-10 ==
== 2021-06-14 ==
* 11:57 hashar: Gerrit: deleting mediawiki/services/service-template-node and mediawiki/services/service-runner (unused) # [[phab:T231953|T231953]]
* 22:06 brennen: gitlab-test: repointing floating IP to ansible test box, running ansible to test issue & wiki default config
* 08:03 hashar: Updated Phan jobs for [[phab:T232413|T232413]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/535507 phan: stop installing MediaWiki
* 21:41 brennen: gitlab-test: repointing floating IP to main test instance; gitlab-ctl reconfigure to test some feature flags
* 16:57 James_F: uul: [mediawiki/tools/api-testing] Publish docs on postmerge [[phab:T236915|T236915]]
* 16:31 James_F: Zuul: [mediawiki/tools/api-testing] Add npm run doc and publishing [[phab:T236915|T236915]]
* 16:28 James_F: Zuul: Add Yashvarshney02 to Trusted users
* 16:25 James_F: Zuul: Add Jay (CIS-A2K) to Trusted users
* 16:22 James_F: Zuul: Add initial CI for cloud/toolforge/jobs-framework-cli


== 2019-09-09 ==
== 2021-06-13 ==
* 15:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535198  Shibboleth disable selenium tests
* 18:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699567
* 12:26 hashar: Reloading Zuul to add multiple extension dependencies
* 18:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/699541
* 12:15 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535161 " FundraiserLandingPage requires EventLogging"


== 2019-09-06 ==
== 2021-06-11 ==
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/534872
* 20:49 brennen: gitlab1001: resetting application data, re-running ansible playbook
* 19:51 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534862/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534863/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534864/
* 15:50 James_F: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 19:37 hashar: Reloading zuul for {{Gerrit|Id07a109accc88b138193463a8e7335a60e501910}} Add UniversalLanguageSelector to ExternalGuidance's dependencies
* 15:25 James_F: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [[phab:T284345|T284345]]
* 14:47 majavah: generate and add my (taavi) own root key to deployment-prep
* 14:14 hashar: deployment-imagescaler03: delete local mwdeploy user with uid 497 # [[phab:T73480|T73480]]
* 12:31 hashar: deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine.
* 10:30 hashar: deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # [[phab:T100837|T100837]]
* 02:03 Reedy: beta-update-databases-eqiad seemingly broken by CategoryTree fix for [[phab:T271011|T271011]].  Comment left on gerrit patch and task,  not reverting patch in master at this stage


== 2019-09-05 ==
== 2021-06-10 ==
* 19:36 James_F: Zuul: Drop debian-glue-non-voting from cassandra-table-properties
* 21:12 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as phan dep too [[phab:T281195|T281195]]
* 21:05 James_F: Zuul: [mediawiki/extensions/ProofreadPage] Add Scribunto as a dependency [[phab:T281195|T281195]]


== 2019-09-04 ==
== 2021-06-08 ==
* 18:48 James_F: Zuul: Move quibble and mwgate basic test jobs from php70 to php72
* 20:32 brennen: gitlab1001: k6_gitlab: running test data creation
* 16:49 James_F: Zuul: Replacing basic quibble HHVM test with PHP72 https://gerrit.wikimedia.org/r/533973
* 19:59 brennen: gitlab1001: gitlab-ansible run to reset configuration
* 16:25 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534481/
* 19:40 brennen: gitlab1001: resetting all application data for a second attempt at test data creation
* 12:20 hashar: Dropped Phan jobs for extensions from release branches  [[phab:T226945|T226945]]
* 17:40 brennen: gitlab1001: running k6 data generator
* 17:25 James_F: Zuul: [mediawiki/extensions/Wikibase] Switch legacy Ruby jobs to 2.5 [[phab:T280491|T280491]]


== 2019-09-03 ==
== 2021-06-07 ==
* 21:59 James_F: Zuul: Move ORES and CirrusSearch to codehealth [[phab:T224990|T224990]]
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # [[phab:T263617|T263617]]
* 20:47 James_F: Zuul: Suspend trigger-mediawiki-pipeline-dev for now [[phab:T218360|T218360]]
* 22:02 urbanecm: urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #[[phab:T263617|T263617]]
* 15:57 James_F: Zuul: Enable the publish pipeline for mediawiki/services/kartotherian  [[phab:T223275|T223275]]
* 21:40 James_F: Docker: Pushing node12-test ano node12-test-browser 0.0.2 for [[phab:T284492|T284492]]
* 15:36 James_F: Zuul: Add phan dependencies for MachineVision
* 11:51 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/tegola --change 698470,1  # request by mbsantos for https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/698470
* 14:34 James_F: Zuul: And wmf/ branches for VisualEditor core.
* 14:21 James_F: Zuul: Drop EducationProgram from dependencies
* 09:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/533916 "Apply REL based pipelines"
* 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533905/ "Remove wmf pipelines from non mediawiki repos"
* 08:32 hashar: err
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533909/ "zuul: add pipelines for MediaWiki releases"


== 2019-09-02 ==
== 2021-06-05 ==
* 17:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533885/ zuul: introduce test-wmf pipeline
* 20:34 James_F: Zuul: [mediawiki/extensions/TitleIcon] Switch to non-composer, with-selenium
* 17:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533880/ "zuul: rename gate-and-submit-swat to use -wmf"
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533536/ "zuul: drop skip-if for fundraising/REL branch"
* 08:22 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533532/ "zuul: pipelines for fundraising branches"


== 2019-08-31 ==
== 2021-06-04 ==
* 16:20 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/533628/
* 23:57 Krinkle: integration-agent-qemu-1001 back up, Thanks andrewbogott
* 12:28 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]
* 23:47 Krinkle: Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001
* 12:22 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} T 231696
* 20:13 James_F: Zuul: Switch almost all node10 jobs to node12 [[phab:T284345|T284345]]
* 12:18 hauskatze: github: created https://github.com/wikimedia/operations-debs-prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]
* 20:11 James_F: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [[phab:T284345|T284345]]
* 19:20 James_F: Docker: Publishing node12 CI images [[phab:T284343|T284343]]


== 2019-08-30 ==
== 2021-06-03 ==
* 18:28 James_F: Zuul: Move Popups and MobileFrontend to codehealth
* 19:06 hashar: contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/*  # [[phab:T284125|T284125]]
* 15:40 Krinkle: Updated "Prod Error" task form to have Title at the top and Trace below Description.
* 00:39 James_F: Zuul: Add EventLogging to dependencies of PropertySuggester
* 10:53 hashar: deployment-mediawiki-07: nuke /mnt/mediawiki  , last touched in April 2018 and using space on /
* 00:24 James_F: Zuul: Add Anysite to CI allowlist
* 00:19 James_F: Zuul: [mediawiki/services/image-suggestion-api] Use bespoke pipeline [[phab:T281132|T281132]]


== 2019-08-29 ==
== 2021-06-02 ==
* 23:23 James_F: Zuul: Move all archived repos to their own section at the end (no-op)
* 19:21 urbanecm: deployment-prep: Enlarge mwlog's /srv partition to 5 GB (was 2 GB)
* 18:38 James_F: Zuul: Adding Wikibase back into dependencies for MachineVision
* 13:49 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project search/MjoLniR --change 685569,4  #  Rerun sonar analysis against https://gerrit.wikimedia.org/r/c/search/MjoLniR/+/685569
* 15:47 James_F: Zuul: Drop Wikibase dependency from MachineVision
* 02:32 Krinkle: Delete erroneous branch 'REL1_34-5' ({{Gerrit|108cb95c57a6e02b8}}) from mediawiki/extensions/SoftRedirector. ref https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/


== 2019-08-28 ==
== 2021-05-28 ==
* 23:00 James_F: Zuul: add tests for the labs/tools/maintain-kubeusers repo [[phab:T228499|T228499]]
* 09:41 addshore: reload zuul for https://gerrit.wikimedia.org/r/697033
* 23:00 James_F: Zuul: [TimedMediaHandler] Run phan job [[phab:T224766|T224766]] (merged but not deployed?)
* 08:02 hashar: Successfully published image docker-registry.discovery.wmnet/releng/ci-bullseye:0.1.0 # [[phab:T283777|T283777]]
* 21:08 thcipriani: restarting zuul server since it's out-of-sync with reality
* 16:11 hashar: Upgraded Jenkins for security update (2.176.3)
* 16:08 hashar: integration:  docker container prune -f ; docker image prune -f
* 00:07 James_F: JJB: Delete mwext-npm-doc-publish, no longer used
* 00:07 James_F: Zuul: Migrate MobileFrontend to extension-javascript-documentation [[phab:T230831|T230831]]


== 2019-08-27 ==
== 2021-05-27 ==
* 20:25 James_F: Zuul: TwnMainPage: switch to extension-quibble-composer-nohhvm [[phab:T231289|T231289]]
* 23:14 brennen: gitlab1001: gitlab-ctl stop nginx - pausing httpd for the weekend
* 20:36 brennen: gitlab1001: temporarily disabling backup cron jobs
* 17:46 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/696565 https://gerrit.wikimedia.org/r/696566
* 16:52 brennen: gitlab1001: ran gitlab-ctl start; logins now working; will add banner to effect that this is all provisional state
* 16:05 brennen: gitlab1001: re-running ansible and puppet per [[phab:T279545|T279545]]
* 00:14 James_F: Zuul: [wikimedia/irc/ircservserv-config] Fix bad copy-paste
* 00:06 James_F: Zuul: [wikimedia/irc/ircservserv-config] Add bespoke pipeline jobs


== 2019-08-26 ==
== 2021-05-26 ==
* 22:31 James_F: Zuul: Move Popups to extension-javascript-documentation
* 17:37 brennen: gitlab1001: reset admin password and ran `gitlab-ctl stop` ([[phab:T279545|T279545]])
* 22:12 James_F: Zuul: Move MinervaNeue to extension-javascript-documentation
* 16:24 brennen: running gitlab-ansible's install-gitlab-server.sh against gitlab1001.wikimedia.org
* 18:23 James_F: Zuul: [MinervaNeue, MobileFrontend, Popups] Add extension-javascript-documentation template
* 15:35 hashar: deployment-mwmaint01: /var/cache/hhvm/cli.hhbc.sq3 (1.3GB) - [[phab:T161598|T161598]]
* 13:51 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/532379


== 2019-08-25 ==
== 2021-05-24 ==
* 21:27 Reedy: centralauth.globalblocks `ALTER TABLE /*_*/globalblocks MODIFY gb_reason varbinary(767) NOT NULL;
* 06:19 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693639
* 16:43 Urbanecm: Revert 'Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]'
* 16:33 Urbanecm: Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]


== 2019-08-23 ==
== 2021-05-21 ==
* 11:12 Urbanecm: deployment-prep: run scap sync to ensure everything is deployed ([[phab:T231058|T231058]])
* 18:30 James_F: Zuul: Add phan to all extensions and skins [[phab:T283097|T283097]]
* 07:12 rxy: Applied SQL queries per [[phab:T231058#5433197]]
* 17:59 James_F: Docker: Publishing mediawiki-phan* images where the job passes if .phan/config.php is absent [[phab:T283097|T283097]]
* 08:30 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/693361 https://gerrit.wikimedia.org/r/691323


== 2019-08-21 ==
== 2021-05-20 ==
* 15:49 James_F: Zuul: Add phan for LiquidThreads
* 21:36 Krinkle: Fix broken Jenkins config for console sections of selenium jobs to accomodate for updates to wdio
* 08:41 hashar: Build docker-registry.discovery.wmnet/releng/quibble-coverage:0.0.34-4  # [[phab:T230423|T230423]]
* 01:43 James_F: Zuul: Add Southparkfan to CI allowlist
* 01:33 James_F: Zuul: [labs/codesearch] Add "test" deployment pipeline job too
* 00:18 James_F: Zuul: [labs/codesearch] Install deployment pipeline
* 00:14 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s2 with no memory limit for coverage jobs [[phab:T280669|T280669]]


== 2019-08-20 ==
== 2021-05-19 ==
* 15:55 James_F: Zuul: Update Shreyas Minocha's whitelist e-mail
* 23:44 James_F: Publishing quibble-buster-php73-coverage:0.0.47-s1 with a 4GiB memory limit for coverage jobs [[phab:T280669|T280669]]
* 15:45 urandom: creating Parsoid/PHP storage schema in deployment-prep -- [[phab:T230792|T230792]]
* 16:42 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add Scribunto & EventLogging deps [[phab:T279275|T279275]]
* 09:39 James_F: Zuul: [kartotherian] Disable pipeline on gate as it doesn't pass [[phab:T223275|T223275]]
* 16:39 James_F: Zuul: Add H.krishna123 to the list of trusted users [[phab:T279552|T279552]]


== 2019-08-19 ==
== 2021-05-17 ==
* 18:43 thcipriani: git -C /srv/mediawiki-staging/php-master/extensions/ConfirmEdit checkout maintenance/GenerateFancyCaptchas.php
* 18:01 James_F: Zuul: [mediawiki/extensions/RelatedLinks] Archive [[phab:T279221|T279221]]


== 2019-08-15 ==
== 2021-05-16 ==
* 12:13 James_F: Zuul: Make GrowthExperiments depend on Echo for non-Phan too.
* 19:58 Krinkle: deployment-mediawiki11$ apt-get install memkeys
* 10:50 James_F: Zuul: Make GrowthExperiments depend on Echo
* 09:29 Majavah: fix labs/private merge conflicts on deployment-puppetmaster04
* 09:05 James_F: Zuul: Kartotherian pipeline enablement [[phab:T223275|T223275]]


== 2019-08-14 ==
== 2021-05-15 ==
* 18:40 hauskatze: github: Created https://github.com/wikimedia/operations-debs-prometheus-ipsec-exporter && forced replication to mirror changes # [[phab:T230443|T230443]]
* 21:51 James_F: Zuul: [mediawiki/tools/cli] Make mw-cli-test experimental for now [[phab:T248779|T248779]]
* 18:29 hauskatze: Created new gerrit project: `operations/debs/prometheus-ipsec-exporter.git` # [[phab:T230443|T230443]]
* 21:40 James_F: Zuul: [mediawiki/tools/cli] Add new bespoke job [[phab:T248779|T248779]]
* 14:42 James_F: Move all BlueSpice extensions to not run HHVM [[phab:T230482|T230482]]
* 21:09 James_F: Zuul: [mediawiki/extensions/MediaWikiAuth] Mark repo as archived [[phab:T282955|T282955]]
* 18:38 James_F: Zuul: Temporarily remove TwoColConflict from gated extensions [[phab:T234002|T234002]] [[phab:T282935|T282935]].
* 09:30 Majavah: create deployment-logstash04 to install elk7
* 09:22 Majavah: beta: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315, remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/683837 [[phab:T277990|T277990]]
* 06:23 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/691494/ [[phab:T281986|T281986]]


== 2019-08-13 ==
== 2021-05-14 ==
* 22:56 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/530012
* 21:31 Krinkle: Delete now-unreadable unread echo notifications from deploymentwiki and clear cache badge count cache (echo_unread_wikis: 9892 rows affected, Echo/maintenance/recomputeNotifCounts.php),  [[phab:T198673|T198673]]
* 21:10 Krinkle: Delete beta cluster commonswiki.globalusage data for deploymentwiki, [[phab:T198673|T198673]],  https://wikitech.wikimedia.org/wiki/Delete_a_wiki (86 rows affected)
* 21:09 Krinkle: Delete beta cluster centralauth rows relating to deploymentwiki, [[phab:T198673|T198673]], https://wikitech.wikimedia.org/wiki/Delete_a_wiki (12600 rows affected)
* 20:51 Krinkle: I broke beta  `InvalidArgumentException: mcrouter-with-onhost-tier not present in $wgObjectCaches` - working on it
* 13:08 addshore: Github, Allowed Wikimedia Helper Bot for GitHub to read `github/workflows/dependabot-gerrit.yml`
* 10:26 addshore: reload zuul for WMDE: Add Marta to trusted emails [integration/config] - https://gerrit.wikimedia.org/r/691117
* 02:16 James_F: Zuul: [mediawiki/skins/MinervaNeue] Drop Ruby-based selenium job [[phab:T174018|T174018]] [[phab:T177260|T177260]] [[phab:T280901|T280901]]
* 02:09 James_F: Zuul: [operations/container/miscweb] Install bespoke pipeline CI [[phab:T281538|T281538]]


== 2019-08-08 ==
== 2021-05-13 ==
* 17:14 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/529108
* 23:14 Krinkle: root@integration-agent-qemu-1001:/home/addshore# rm qmeutest/out.img (Reclaiming space to be able to make other changes)
* 03:18 thcipriani: cancelled stuck beta-scap-eqiad job to unblock postmerge
* 22:48 longma: restarting zuul to try getting tests to run for https://gerrit.wikimedia.org/r/c/mediawiki/skins/Modern/+/684509


== 2019-08-07 ==
== 2021-05-12 ==
* 16:00 thcipriani: restarting jenkins for updates
* 19:40 dpifke: Un-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/439774 in deployment-prep puppet.  Hasn't been touched in over a year and is preventing merges.
* 19:32 James_F: Zuul: Add PageImages, PageViewInfo, and Graph to MediaWiki gated extension set [[phab:T249674|T249674]]
* 19:23 dpifke: Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/683695 in deployment-prep puppet.  Should only affect beta logstash.
* 16:15 James_F: Docker: Publishing civicrm 0.2.1 for [[phab:T277500|T277500]]
* 16:00 James_F: Zuul: Add GrowthExperiments to the MediaWiki gated extension set [[phab:T247507|T247507]] [[phab:T249674|T249674]]
* 15:36 _joe_: updating helm-linter docker image


== 2019-08-06 ==
== 2021-05-11 ==
* 21:24 James_F: Zuul: [TheWikipediaLibrary] Enable plan
* 23:32 James_F: Zuul: Add Disambiguator to the MediaWiki gated extension set [[phab:T237538|T237538]] [[phab:T249674|T249674]]
* 21:05 James_F: Zuul: [TheWikipediaLibrary] Enable phan-seccheck
* 23:25 James_F: Zuul: [mediawiki/extensions/NCBITaxonomyLookup] Enable basic quibble CI
* 21:01 James_F: Zuul: [TheWikipediaLibrary] Add phan dependencies
* 19:26 paladox: ssh -p 29418 gerrit.wikimedia.org replication start --url github mediawiki/core --wait --now
* 17:09 accraze: deploying ores {{Gerrit|d08fa628aacb82529dbb4be357b68dd55c15fdee}}
* 11:42 paladox: ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --wait


== 2019-08-05 ==
== 2021-05-10 ==
* 22:24 accraze: deploying ores {{Gerrit|4270244d4a54c520f581dd33b312aa52f9a4c736}}
* 14:38 James_F: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency
* 19:22 James_F: Zuul: [mediawiki/services/chromium-render] Enable pipeline testing and publishing [[phab:T217114|T217114]]
* 14:04 CFisch_WMDE: Improve comment around ReferencePreviews beta cluster default ([[phab:T271206|T271206]])
* 14:04 CFisch_WMDE: Forward renamed config name for improved template search features ([[phab:T277028|T277028]])


== 2019-08-04 ==
== 2021-05-07 ==
* 12:11 hauskatze: gerrit: ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/OATHAuth --wait # [[phab:T229756|T229756]]
* 16:37 James_F: Zuul: [operations/software/mailman-templates] Add CI of debian-glue [[phab:T282018|T282018]]


== 2019-08-02 ==
== 2021-05-06 ==
* 17:30 James_F: Zuul: [operations/mediawiki-config] Stop running php70 jobs"
* 02:52 James_F: jjb: Enable Sonar analysis for mjolnir builds [[phab:T264877|T264877]]
* 02:14 James_F: Zuul: [mediawiki/extensions/UploadWizard] Drop tox job, not useful
* 00:16 James_F: Zuul: [mediawiki/services/parsoid] Drop parsoidsvc-parsertests-docker job [[phab:T271562|T271562]]


== 2019-08-01 ==
== 2021-05-05 ==
* 20:19 Krinkle: Resolve Jenkins admin warning about legacy token by revoking it from the only user id "<unknown>" that seems to have one
* 23:52 James_F: Zuul: Ensure Parsoid's tests include the Disambiguator extension (take 2) [[phab:T271863|T271863]] [[phab:T237538|T237538]]
* 19:00 James_F: Zuul: [BlueSpiceExtendedSearch] Disable HHVM tests, this doesn't work there
* 20:38 James_F: Docker: Publishing quibble-buster* 0.0.47 images
* 19:58 James_F: Docker: Publishing quibble-stretch* 0.0.47-s1 images, now with node
* 18:23 James_F: Docker: Publishing quibble-stretch* 0.0.47 images
* 16:54 hashar: Tag Quibble 0.0.47 @ {{Gerrit|8b200cfb0}} # [[phab:T271863|T271863]] [[phab:T199403|T199403]] [[phab:T281607|T281607]]
* 14:17 CFisch_WMDE: Disable ReferencePreviews beta mode on beta labs ([[phab:T271206|T271206]])
* 13:22 Majavah: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/684034 https://gerrit.wikimedia.org/r/c/operations/puppet/+/684088 - might cause beta to go read only for a bit - [[phab:T110115|T110115]]


== 2019-07-31 ==
== 2021-05-04 ==
* 19:20 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$  mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki ([[phab:T225056|T225056]])
* 23:34 James_F: Zuul: Add Adam Hammad to CI allow list
* 18:27 James_F: Zuul: Move three repos to generic tox-docker job
* 17:02 Amir1: stop exim4 and upgrade it in deployment-mx02
* 17:50 James_F: Zuul: Move cassandra-table-properties to use cassandra-table-properties-tox-docker
* 16:27 James_F: Zuul: Archive SemanticTitle [[phab:T229422|T229422]]
* 15:55 James_F: Docker: Pushing operations-puppet:0.5.4
* 15:46 James_F: Docker: Pushing quibble-coverage:0.0.34-3
* 14:21 andrewbogott: moving deployment-sca02 to cloudvirt1030
* 12:59 andrewbogott: moving deployment-elastic05, deployment-kafka-main-2, deployment-mx02, deployment-webperf11 to new cloudvirts


== 2019-07-30 ==
== 2021-05-03 ==
* 21:17 James_F: Docker: Push operations-puppet:0.5.3
* 19:42 James_F: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [[phab:T281256|T281256]]
* 16:17 James_F: Docker: Pushing quibble-coverage:0.0.34-2
* 17:05 James_F: Docker: Publishing quibble-buster images with python3-distutils so quibble can build
* 15:39 cdanis: manually set dbctl etcd key to dummy value: cdanis@deployment-etcd-01:~$ etcdctl -u root -C http://127.0.0.1:2378 set /conftool/v1/mediawiki-config/eqiad/dbconfig
* 16:07 James_F: Zuul: Add Luca Mauri to the CI allow list
* 14:44 James_F: Docker: Pushing quibble-coverage:0.0.34-1
* 13:55 CFisch_WMDE: enable new search features for the template dialog ([[phab:T271802|T271802]])


== 2019-07-29 ==
== 2021-05-02 ==
* 18:23 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Make PHP72 job voting, drop HHVM one
* 18:58 Majavah: add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address)
* 18:13 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Provide experimental PHP72 job to replace HHVM one
* 18:50 Majavah: adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org
* 18:42 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - <https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117>, ref [[phab:T198673|T198673]]
* 18:41 Krinkle: Run `puppet agent -tv` on deployment-cache-text06 and deployment-mediawiki11
* 18:37 Krinkle: Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117


== 2019-07-27 ==
== 2021-05-01 ==
* 20:38 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-SemanticHighcharts refs. [[phab:T199421|T199421]]
* 19:19 James_F: Zuul: Add atagar to the CI allow list
* 10:37 Majavah: installing deployment-urldownloader03 to replace 02 - [[phab:T278641|T278641]]
* 04:05 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/684004


== 2019-07-26 ==
== 2021-04-30 ==
* 21:10 legoktm: deploying https://gerrit.wikimedia.org/r/525871
* 20:13 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/683987
* 18:38 James_F: Zuul: operations/software/cassandra-table-properties initial jobs
* 19:21 James_F: Docker: Publishing mediawiki-phan-taint-check-demo:0.1.1 for [[phab:T257301|T257301]]
* 16:02 hauskatze: Deleted https://github.com/wikimedia/mediawiki-extensions-Copyvio {{!}} refs. [[phab:T228087|T228087]]
* 14:21 Majavah: add profile::pki::client to all deployment-prep instances to trust deployment-prep cfssl certificates, already deployed on production
* 16:00 James_F: Zuul: [Copyvio] Mark as archived for CI [[phab:T228087|T228087]]
* 14:15 Majavah: revert above as it's not working, [[phab:T206158|T206158]]
* 15:49 Krinkle: Fix Jenkins Console output section "Selenium" to account for Quibble 0.34 changes (multiple Selenium sections, one implicitly closing the other)
* 14:13 Majavah: deployment-cache-text: trying out useusing HTTPS for backend traffic to deployment-mediawiki11 [[phab:T206158|T206158]]
* 15:45 James_F: doc1001: sudo -u doc-uploader rm -rf /srv/docroot/org/wikimedia/doc/cover-extensions/Copyvio/ # [[phab:T228087|T228087]]
* 12:37 Majavah: force reboot deployment-cache-text06, not letting me to log in, this will disrupt beta cluster availability
* 04:43 James_F: Zuul: [mediawiki/libs/LangConv] Make this PHP7.2+ only
* 02:37 James_F: Docker: Publishing node10 images based on buster [[phab:T278203|T278203]] [[phab:T240955|T240955]]
* 04:36 James_F: Zuul: Add testing for mediawiki/libs/LangConv


== 2019-07-25 ==
== 2021-04-29 ==
* 23:12 hashar: Purging all docker images on Jenkins instances: sudo cumin --trace --force 'name:docker' 'docker images{{!}} awk "{ print \$3}"{{!}}xargs docker rmi'
* 12:19 Majavah: dropping jade_diff_judgement, jade_diff_label, jade_revision_judgement, jade_revision_label tables on all-labs.dblist [[phab:T281418|T281418]]
* 22:52 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for  https://gerrit.wikimedia.org/r/525673
* 22:30 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525673
* 21:28 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 (second attempt) for [[phab:T229020|T229020]]  using https://gerrit.wikimedia.org/r/#/c/integration/config/+/525570/
* 21:00 James_F: Zuul: Add phan to EventBus
* 18:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525608 - [[phab:T227004|T227004]])
* 17:38 greg-g: Added Jforrester to the Release Engineering group in Gerrit
* 17:17 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for [[phab:T229020|T229020]]
* 16:56 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525591
* 16:54 hashar: Tagged Quibble 0.0.34 {{Gerrit|ff8d86ffca97d17521f96d534f1e43780e6a22d5}} # [[phab:T229020|T229020]]
* 16:41 James_F: Zuul: [analytics-wikistats2] Drop node 6, switch to node10-browser-docker [[phab:T211784|T211784]]
* 16:20 hashar: Rolling back Quibble based jobs to 0.0.32 :-(
* 15:38 hashar: Upgrading Jenkins jobs to Quibble 0.0.33
* 15:09 hashar: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/525559
* 14:52 hashar: Tagged Quibble 0.0.33 {{Gerrit|b2f9e36}}  {{!}} [[phab:T193824|T193824]] [[phab:T87781|T87781]] [[phab:T199116|T199116]]


== 2019-07-24 ==
== 2021-04-28 ==
* 20:00 James_F: Zuul: Mark VIKI, VikiSemanticTitle, and VikiTitleIcon as broken.
* 15:27 James_F: Zuul: [mediawiki/libs/metrics-platform] Add pipeline-based CI jobs [[phab:T279180|T279180]]
* 10:55 hauskatze: Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete # [[phab:T249268|T249268]]
* 10:54 hauskatze: deployment-prep Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 07:26 hashar: contint2001: sudo -u jenkins find *quibble* -path '*/archive/log/rawSeleniumVideoGrabs/*' -delete
* 07:19 hashar: contint2001: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip <nowiki>{</nowiki><nowiki>}</nowiki> \+  # [[phab:T249268|T249268]]
* 01:19 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/performance/navtiming/+/682886 in beta, should only affect deployment-webperf11.


== 2019-07-23 ==
== 2021-04-27 ==
* 17:01 James_F: Zuul: [trigger-mediawiki-pipeline-dev] Only trigger on master
* 19:16 James_F: Docker: Rebuilding all Sury-php derivatives for [[phab:T277742|T277742]].
* 16:00 hashar: gerrit: adding Jakob_WMDE to wmf-deployment group ( https://gerrit.wikimedia.org/r/#/admin/groups/21,members ) # [[phab:T227193|T227193]]
* 17:52 Majavah: delete deployment-sessionstore03 [[phab:T263617|T263617]] [[phab:T278641|T278641]]
* 08:00 hashar: Restore puppet class contint::packages::javascript on permanent slave ( reverting https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524221/ ) # [[phab:T228639|T228639]]
* 16:35 James_F: Docker: Publishing composer-scratch 1.10.22 and its cascade for [[phab:T281283|T281283]]
* 14:18 hashar: Updating most jenkins jobs to change cleanup commands from stretch to buster {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/680476
* 12:44 hashar: Restarted CI Jenkins for plugins upgrade
* 12:24 hashar: Upgraded releases Jenkins from 2.263.3 to 2.277.2  (with ldap plugin 1.26)
* 12:11 hashar: Upgrading Jenkins plugins on the releases jenkins
* 06:40 Majavah: installing deployment-sessionstore04 [[phab:T263617|T263617]]
* 05:29 Majavah: restart cassandra on deployment-sessionstore03 refs [[phab:T281198|T281198]]


== 2019-07-22 ==
== 2021-04-26 ==
* 22:43 James_F: Zuul: [wikimedia/iegreview] Adding mediawiki-i18n-check-docker template
* 16:53 James_F: Zuul: Add AnjaliKumari to the CI allow list
* 10:49 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master/extensions$ sudo rm -rf Wikidata


== 2019-07-19 ==
== 2021-04-24 ==
* 19:11 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/524565
* 17:47 James_F: Zuul: [mediawiki/extensions/MultimediaViewer] Drop Ruby selenium test job


== 2019-07-18 ==
== 2021-04-23 ==
* 22:18 James_F: Zuul: Archive maps/kartotherian and maps/tilerator
* 22:14 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/682029
* 21:17 mutante: rebooting gerrit server
* 16:30 Majavah: remove deployment-prep hiera settings for phabricator, given there is no phabricator instance on that project
* 19:18 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.1
* 09:12 Majavah: signing puppet certs for deployment-eventlog08 and running puppet for the first time to stop annoying email alerts
* 19:03 James_F: Zuul: [analytics-mediawiki-storage] Drop node 6, switch to node10-browser-docker
* 18:25 mutante: contint2001 - puppet run reinstalled python-dateutil and ruby-safe-yeaml, stopped zuul service but no more failures now
* 18:25 mutante: contint2001 /etc/init.d/zuul-merger stop; apt install zuul (this time it worked to configure zuul after also stopping zuul-merge) ; apt-get autoremove (removed unused python packages); puppet agent -tv
* 18:22 mutante: contint2001 /usr/sbin/dpkg-reconfigure: zuul is broken or not fully installed
* 18:22 mutante: contint2001 systemctl stop zuul ; apt install zuul ... still fails
* 18:10 James_F: Zuul: [wikimedia/services/kartotherian] Switch to custom job [[phab:T228170|T228170]]
* 18:06 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.0
* 13:51 hashar: integration: in hiera project config, remove  classes: [ contint::packages::apt ]  {{!}} [[phab:T225735|T225735]]
* 13:48 hashar: integration: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523150/  for apt unattended_upgrades {{!}} [[phab:T225735|T225735]]
* 13:37 hashar: integration: rm /etc/apt/sources.list.d/sury-php.list  and remove php packages {{!}} https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523148/ {{!}} [[phab:T225735|T225735]]


== 2019-07-17 ==
== 2021-04-22 ==
* 16:49 James_F: Zuul: [PageForms] Add phan dependencies on AdminLinks, Cargo, PageSchemas [[phab:T228155|T228155]]
* 06:06 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/680697
* 16:48 James_F: Zuul: Register initial tests for eventgate-wikimedia [[phab:T226668|T226668]]
* 02:53 Reedy: killed a few stuck beta ci jobs
* 16:08 hashar: Jenkins: updating wikidata-query-gui-build to get rid of slave-scripts/bin/npm-setup.sh which requires an obsolete/non existing tmpfs  # [[phab:T227818|T227818]]
* 02:51 Krinkle: The 'beta-mediawiki-config-update-eqiad' jobs have been stuck for ~ 8 hours
* 09:48 hashar: Reloaded Zuul [Convert2Wiki] Archive extension
* 02:19 James_F: Zuul: Switch bundle-yard-publish jobs to Ruby 2.5 [[phab:T280874|T280874]]
* 00:42 James_F: Docker: Deploy ci-buster and lintr:0.2.0
* 01:49 James_F: Zuul: [mediawiki/vagrant] Add mediawiki-vagrant-ruby2.5-rake-docker as experimental [[phab:T280874|T280874]]
* 00:04 James_F: Docker: Complete rebuild finished and published; only took 2.5 hours. But now [[phab:T228196|T228196]] is fixed elsewise anyway, oh well.
* 01:44 James_F: Docker: Publishing rake-vagrant-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 00:45 James_F: Zuul: Add experimental Ruby 2.5 jobs for two repos [[phab:T280874|T280874]]


== 2019-07-16 ==
== 2021-04-21 ==
* 21:58 hauskatze: GitHub: deleted `wikimedia/mediawiki-extensions-Convert2Wiki` refs. [[phab:T228198|T228198]]
* 23:43 James_F: Zuul: [operations/puppet-lint/wmf_styleguide-check] Switch to Ruby 2.5
* 21:29 James_F: Docker: Publishing a whole new set of RelEng images for [[phab:T228196|T228196]]
* 23:25 James_F: Zuul: Provide experimental Ruby 2.5 rake jobs [[phab:T280874|T280874]]
* 20:25 James_F: Docker: Running a general rebuild for all missing RelEng images [[phab:T228196|T228196]]
* 22:56 James_F: Zuul: Add mwgate-ruby2.5-rake-docker experimentally to mwgate-rake
* 16:49 James_F: Zuul: [ContentTranslation] Adding CentralAuth to phan dependencies
* 22:36 James_F: Docker: Publishing rake-ruby2.5:0.1.0 for [[phab:T280874|T280874]]
* 16:48 James_F: Zuul: Adding first tasks for mediawiki/services/kartotherian [[phab:T228170|T228170]]
* 18:47 James_F: Add ImageMap to the list of Parsoid's ext dependencies
* 15:19 James_F: Ran manual scap pull on deployment-deploy01
* 15:10 James_F: beta-scap-eqiad failing because mw-config repo is out of date?
* 14:57 James_F: Taking deployment-deploy01 offline
* 14:50 James_F: Jenkins deadlock? Beta jenkins jobs haven't successfully run for ~20 hours.


== 2019-07-15 ==
== 2021-04-20 ==
* 22:40 James_F: Zuul: [CentralAuth] Add phan [[phab:T189371|T189371]]
* 07:19 CFisch_WMDE: enable changes to the descriptions in the VE transclusion dialog ([[phab:T273425|T273425]])
* 18:05 James_F: Zuul: [MinervaNeue] Make phan voting [[phab:T133664|T133664]]
* 07:17 CFisch_WMDE: enable suggested values paramter in TemplateData and VisualEditor ([[phab:T271825|T271825]])
* 17:00 James_F: Zuul: [GrowthExperiments] Add MobileFrontend to phan dependencies
* 16:49 James_F: Zuul: [MinervaNeue] Add phan as non-voting [[phab:T133664|T133664]]
* 16:04 Urbanecm: Run urbanecm@deployment-deploy01:~$ mwscript migrateUserGroup.php --wiki=commonswiki Image-reviewer image-reviewer ([[phab:T216406|T216406]])
* 15:59 James_F: Zuul: Adding Echo and MobileFrontend dependencies to skins/MinervaNeue [[phab:T133664|T133664]]
* 15:38 James_F: Zuul: [SecurePoll] Make seccheck voting [[phab:T205972|T205972]]
* 15:37 James_F: Zuul: [CentralAuth] Add seccheck [[phab:T189227|T189227]]
* 13:30 hashar: Deleting compiler1003.puppet-diffs.eqiad.wmflabs .. insanely slow for some reason (was on cloudvirt1002) # [[phab:T221969|T221969]] )
* 12:55 hashar: Creating compiler1003.puppet-diffs.eqiad.wmflabs [172.16.2.46] m1.large / 4vCPUs  # [[phab:T221969|T221969]]


== 2019-07-14 ==
== 2021-04-19 ==
* 00:55 Krinkle: krinkle@doc1001: Remove https://doc.wikimedia.org/cover-extensions/$DOC_BASENAME/  - a failed build for GrowthExperiments
* 23:04 James_F: Zuul: Add legacy-quibble-rubyselenium-docker as experimental [[phab:T280491|T280491]]
* 17:58 Majavah: apply hack (https://phabricator.wikimedia.org/T277206#7015609) to deployment-puppetmaster04 to unbreak maintenance scripts until we have conftool
* 15:24 James_F: Re-pushing mwselenium-quibble-docker back to master for [[phab:T280491|T280491]]


== 2019-07-12 ==
== 2021-04-17 ==
* 23:07 James_F: Zuul: Instead of hacking around for WebAuthn, define bespoke quibble php72+ template
* 07:23 Majavah: restart uwsgi-ores on deployment-ores01 for [[phab:T280420|T280420]]
* 22:10 James_F: Zuul: Make seccheck voting on WebAuthn [[phab:T226737|T226737]]
* 19:28 James_F: Zuul: Add MobileFrontend and MinervaNeue to Wikibase's dependencies
* 16:56 James_F: Zuul: Add Flow as phan dependency to GrowthExperiments
* 00:32 James_F: Zuul: Migrate from php70-phan-seccheck to php72-phan-seccheck [[phab:T226420|T226420]]


== 2019-07-11 ==
== 2021-04-16 ==
* 23:33 James_F: Pushing docker mediawiki-phan-seccheck:0.4.0
* 23:20 James_F: Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks [[phab:T234020|T234020]] [[phab:T280167|T280167]]
* 21:34 James_F: Pushing new php-based docker images with php-bcmath extension for [[phab:T227043|T227043]]
* 22:39 James_F: Docker: Publishing quibble-buster-php72-bundle
* 20:30 mutante: add project member cdanis
* 22:00 James_F: Docker: Publishing quibble-fresnel based on buster not stretch [[phab:T278203|T278203]]
* 18:57 Krinkle: Fix "setup quibble prepare" Console Section pattern in Jenkins to account for changes in 'mediawiki-quibble-composertest-php72-docker' job
* 19:43 James_F: Zuul: Drop the now duplicate PHP72 'buster' quibble jobs [[phab:T252432|T252432]]
* 18:22 James_F: Zuul: Archive ZeroBanner and ZeroPortal; drop from dependencies
* 19:11 Krinkle: Remove `profile::mediawiki::install_hhvm: false` Hiera config in Horizon for deployment-prep. This variable is no longer used. ref [[phab:T235142|T235142]]
* 17:50 James_F: jforrester@doc1001:/srv/docroot/org/wikimedia/doc/cover-extensions$ rm -rf WikibaseQuality/ # [[phab:T205065|T205065]]
* 19:06 Krinkle: Change profile::mail::mx::verp_post_connect_server in Horizon for deployment-prep from `deployment.wikimedia.beta.wmflabs.org` to `meta.wikimedia.beta.wmflabs.org`, ref [[phab:T198673|T198673]]
* 16:15 James_F: Zuul: Archiving WikibaseQuality
* 19:05 Krinkle: Change profile::mail::mx::verp_bounce_post_url in Horizon for deployment-prep from `http://deployment.wikimedia.beta.wmflabs.org/w/api.php` to `http://meta.wikimedia.beta.wmflabs.org/w/api.php`, ref [[phab:T198673|T198673]]
* 09:48 hashar: jenkins: add more log details to hudson.model.Queue (FINER > FINEST) https://integration.wikimedia.org/ci/log/Jenkins%20Queue/configure  # [[phab:T72597|T72597]]
* 18:47 Krinkle: Delete forceupdate.beta.wmflabs.org from DNS for deployment-prep (created 2020-03-18, comment "I'm going to delete this in a moment")
* 09:06 hashar: beta cluster jobs are dead locked. Taking a thread dump in case it helps figure out what is going on. [[phab:T72597|T72597]]
* 17:44 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/680392
* 17:02 dancy: Updating dev-images docker-pkg files on primary contint
* 16:46 James_F: Zuul: Make php72_buster jobs voting for extension-quibble template [[phab:T252434|T252434]]


== 2019-07-10 ==
== 2021-04-15 ==
* 21:33 James_F: Zuul: Fix gerrit-docker pointer
* 18:26 paladox: gerrit: created openstack/horizon/trove-dashboard per andrewbogott (with parent set as openstack/horizon/horizon)
* 18:27 James_F: Zuul: Adding phan to StructuredDiscussions [[phab:T224759|T224759]]
* 16:47 Majavah: manually rebase deployment-puppetmaster04 due to local hacks having conflicts
* 17:55 James_F: Zuul: Disabling mwgate-rake for mediawiki/extensions/CentralNotice
* 13:30 hashar: on zuul-merger: rm /srv/zuul/git/operations/software/gerrit  due to [[phab:T157818|T157818]]  # [[phab:T189549|T189549]]


== 2019-07-09 ==
== 2021-04-14 ==
* 18:47 James_F: Zuul: Move mediawiki-quibble-composertest from php70 to php72
* 16:19 James_F: Docker: Publish quibble-buster-php73-coverage fixing loading of pcov [[phab:T234020|T234020]]
* 18:18 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move back to php72 as primary
* 18:08 James_F: Push docker composer-package-php72:0.2.2-s2
* 17:21 James_F: Push docker mediawiki-phan-seccheck:0.3.6-2
* 16:18 James_F: Push docker image +mediawiki-phan-seccheck:0.3.6-1
* 15:53 James_F: Pushing docker images php-ast:1.0.1-2 and mediawiki-phan-seccheck:0.3.6
* 15:29 thcipriani: contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to [[phab:T157818|T157818]]
* 15:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/520437
* 14:08 James_F: Pushing mediawiki-phan-seccheck:0.3.6 docker image.
* 09:48 hashar: contint1001: removing /var/lib/docker , no more needed since we now use /mnt/docker # [[phab:T207707|T207707]]
* 00:04 James_F: Pushing php-ast:1.0.1-1 and related composer-package* docker image updates


== 2019-07-08 ==
== 2021-04-13 ==
* 23:54 James_F: Zuul: Move wikimania-scholarships from php56 to php72
* 17:00 halfak: failed deploy to ORES (connection to host failed)
* 23:35 James_F: Dropping mwext-MobileFrontend-npm-run-lint-modules-docker from Zuul [[phab:T224997|T224997]]
* 16:57 halfak: deploying ores {{Gerrit|f08a3cb}}
* 21:39 thcipriani: reenable jenkins jobs service-pipeline-test and service-pipeline-test-and-publish following docker mount move
* 16:41 marxarelli: deleting errant wmf/1.36.0-wmf.39 branches in mediawiki/core and submodule repos
* 21:27 thcipriani: docker-pkg failed building docker-registry.discovery.wmnet/releng/composer-php56:0.1.5
* 12:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678829 https://gerrit.wikimedia.org/r/678830 [[phab:T280004|T280004]]
* 21:18 thcipriani: fab deploy_docker to pull down ci docker-pkg images (I hope)
* 07:46 awight: enable syntax highlighting line numbering on all namespaces ([[phab:T267911|T267911]])
* 21:15 thcipriani: restart docker on contint1001 to pickup config change for https://gerrit.wikimedia.org/r/520738/
* 21:14 thcipriani: disabled jenkins jobs service-pipeline-test and service-pipeline-test-and-publish pre-contint1001 docker restart
* 13:38 hashar: Build docker-registry.wikimedia.org/releng/bazel:0.1.0  # [[phab:T189549|T189549]]


== 2019-07-06 ==
== 2021-04-12 ==
* 11:20 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --wiki=deploymentwiki {{!}} [[phab:T209565|T209565]]
* 15:46 Urbanecm: deployment-prep: Run `mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php` on all beta wikis with GrowthExperiments installed (wikis that are both in all-labs and growthexperiments, plus enwiki; [[phab:T279853|T279853]])
* 11:09 hauskatze: beta cluster: maurelio@deployment-deploy01 foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --dry-run {{!}} [[phab:T209565|T209565]]
* 15:40 Urbanecm: deployment-prep: urbanecm@deployment-deploy01:~$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=cswiki # [[phab:T279853|T279853]]
* 14:39 Majavah: remove https://gerrit.wikimedia.org/r/c/operations/puppet/+/263024 cherry pick from beta cluster per [[phab:T106915|T106915]]#6279270 - [[phab:T135427|T135427]]
* 14:38 Majavah: fix parsoid CI ferm rule local hack puppet patch on deployment-puppetmaster04 after it broke due to operations/puppet changes
* 11:30 Urbanecm: deployment-prep: Beta is down due to my change, fix on its way (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/678578)


== 2019-07-05 ==
== 2021-04-11 ==
* 21:17 James_F: Adding node10-docker for react.ooui
* 14:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/678378
* 19:08 James_F: Pushing java8-sonar-scanner:0.4.11
* 00:52 James_F: dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov [[phab:T234020|T234020]]
* 00:08 James_F: Zuul: [mediawiki/core] Enforce PHP 8.0 composer test for REL1_3<nowiki>{</nowiki>5,6<nowiki>}</nowiki> [[phab:T274971|T274971]]
* 00:02 James_F: Docker: Publishing quibble-buster images [[phab:T252432|T252432]]


== 2019-07-03 ==
== 2021-04-10 ==
* 21:01 accraze: deploying ores {{Gerrit|676f7ba43c23e7b6dfa20b3b3c03924cb49acb90}}
* 17:36 James_F: Zuul: Add Meno25 to &email_allowlist list
* 13:12 Urbanecm: deployment-prep Manually ran sudo -u jenkins-deploy scap sync-file wmf-config/CommonSettings-labs.php 'Not synced yet?' to sync file that's mysteriously not synced, but present on deployment host


== 2019-07-02 ==
== 2021-04-09 ==
* 20:34 James_F: Zuul: Adding quibble to the PopupAlerts, UnblockMe and Scribe extensions
* 18:41 Krinkle: Logstash in beta shows no messages from mediawiki for 24 hours, [[phab:T233134|T233134]].
* 18:39 Krinkle: Logstash in beta has no messages for 24 hours.
* 12:59 addshore: reload zuul for https://gerrit.wikimedia.org/r/673503 and https://gerrit.wikimedia.org/r/678242 [[phab:T277750|T277750]] phan for wikibase release branches
* 09:51 Majavah: deleting deployment-jobrunner03 [[phab:T278664|T278664]]
* 03:32 James_F: Docker: Publishing new quibble images with pcov instead of xdebug for test coverage [[phab:T234080|T234080]]


== 2019-07-01 ==
== 2021-04-08 ==
* 23:24 James_F: Pushing quibble-coverage:0.0.32-1 image
* 15:57 James_F: Zuul: [mediawiki/extensions/WikiToLDAP] Add quibble and phan job
* 22:30 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 06:52 Majavah: deployment-docker-cpjobqueue01 edit configuration to pick up https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/676563
* 20:40 James_F: Rebuilt codehealth jjbs using node10-test with https://gerrit.wikimedia.org/r/519652
* 18:49 James_F: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental
* 18:43 James_F: Reloading Zuul for mediawiki/tools/phan/SecurityCheckPlugin php72 tests


== 2019-06-30 ==
== 2021-04-07 ==
* 17:03 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/519793
* 16:03 James_F: Zuul: Add Bharatkhatri in the CI allow list
* 16:03 James_F: Zuul: [wikidata/query-builder] Add gate-and-submit-l10n template
* 15:12 Majavah: remove jessie-deployment-prep from deployment-deploy01 aptly
* 14:27 Majavah: delete deployment-mediawiki-07 and deployment-parsoid11 [[phab:T278664|T278664]]
* 00:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677388
* 00:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/677382


== 2019-06-28 ==
== 2021-04-06 ==
* 08:58 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/519603
* 20:26 James_F: Zuul: [mediawiki/extensions/D3Loader] Mark as archived [[phab:T277626|T277626]]
* 00:07 James_F: Restoring php-gmp in quibble docker images
* 20:26 James_F: Zuul: [mediawiki/extensions/Gravatar] Add basic quibble and phan jobs [[phab:T279260|T279260]]
* 19:58 James_F: Zuul: Configure the REL1_36 test and gate pipelines [[phab:T279459|T279459]]
* 17:58 James_F: Zuul: [mediawiki/services/function-<nowiki>{</nowiki>orchestr,evalu<nowiki>}</nowiki>ator] Publish images


== 2019-06-27 ==
== 2021-04-05 ==
* 21:27 James_F: Pushing new docker php base images with gmp, and all their descendants
* 18:20 brennen: resizing gitlab-ansible-test to g3.cores8.ram16.disk20
* 17:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519455
* 17:45 brennen: halting gitlab-test for resize
* 17:04 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519454
* 15:29 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424 (try #2)
* 15:22 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424


== 2019-06-26 ==
== 2021-04-02 ==
* 17:57 James_F: Adding the experimental helm-lint job to deployment-charts [[phab:T216049|T216049]]
* 10:53 Majavah: change deployment-wikifeeds01 config to use deployment-mediawiki11
* 17:55 James_F: Pushing initial helm-linter docker image for [[phab:T216049|T216049]]
* 10:47 Majavah: update web proxy parsoid-beta.wmflabs.org to point to deployment-parsoid12
* 01:29 James_F: Moving labs/tools/stewardbots's composer test to php72+ [[phab:T226585|T226585]]


== 2019-06-25 ==
== 2021-04-01 ==
* 22:00 hauskatze: Ran `replication start operations/debs/coredns --wait`
* 16:16 Majavah: hard reboot unresponsive deployment-cache-text06
* 21:07 hauskatze: gerrit: Created new repo: operations/debs/coredns - requested by fsero on mediawiki
* 12:52 Majavah: update floating ip 185.15.56.9 from deployment-parsoid11 to deployment-parsoid12
* 19:11 Krinkle: Update Jenkins collapsible section patterns to account for "quibble.cmd" => "quibble.commands" changing in the output
* 11:00 Majavah: restart changeprop container on deployment-docker-mobileapps01 to pick up config changes
* 15:58 James_F: Adding CentralAuth to FileImporter required extensions in Zuul
* 10:45 hashar: Updating all Jenkins jobs with jjb to deploy https://gerrit.wikimedia.org/r/676298
* 15:51 awight: restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 15:50 awight: deployment-prep restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 14:39 hauskatze: Ran `replication start operations/debs/file-read-backwards --wait` on gerrit.wikimedia.org refs. [[phab:T226449|T226449]]
* 14:34 hauskatze: github: created https://github.com/wikimedia/operations-debs-file-read-backwards mirror refs. [[phab:T226449|T226449]]
* 14:14 hauskatze: gerrit: created `operations/debs/file-read-backwards.git` refs. [[phab:T226449|T226449]]
* 11:42 hashar: Deleting Jenkins jobs puppet-cdh-rake-docker and puppet-cdh4-rake-docker # [[phab:T226474|T226474]]


== 2019-06-24 ==
== 2021-03-30 ==
* 23:31 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 18:05 Majavah: remove <nowiki>{</nowiki>trysty,precise<nowiki>}</nowiki>-deployment-prep repos from deployment-deploy01 aptly
* 22:08 James_F: Zuul: Restore graphoid deploy repo tests, and comment on change-propagation ones.
* 17:51 Majavah: arm deployment-deploy01 keyholder with all the keys
* 22:02 James_F: Pushed graphoid-deploy-npm-node-6-docker into jjb
* 14:50 Majavah: cherry pick 675807 675814 and 675815 to deployment-puppetmaster to unblock work on deployment-deploy03 until sre has merged those [[phab:T278689|T278689]]
* 22:01 James_F: Re-publishing npm-test-graphoid, whoops.
* 14:44 Majavah: remove deployment-puppetmaster04 local patch adding releng/phatality to scap to see if it unbreaks deployment-deploy03 puppet runs
* 15:17 James_F: Reloading Zuul to drop wikiba.se-composer-php56-docker [[phab:T224905|T224905]]
* 13:55 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675802/ on beta to unblock my progress until merged
* 13:04 hashar: Updating git mirrored repos: integration-cumin:~$ sudo cumin --trace --force 'name:docker' 'find /srv/git -name "*.git" -exec git -C {} fetch --prune \;'
* 13:35 Majavah: create and install deployment-deploy03 [[phab:T278689|T278689]]
* 12:00 hashar: Updating Quibble based jobs to use version 0.0.32
* 13:17 Majavah: armed deployment-cumin keyholder, found passphrase at deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 10:09 hashar: Tag Quibble 0.0.32 @ {{Gerrit|c0fe6eb}} # [[phab:T211701|T211701]] [[phab:T218357|T218357]] [[phab:T220199|T220199]] [[phab:T223752|T223752]]
* 07:26 Majavah: shutoff deployment-mediawiki-09 [[phab:T278664|T278664]]
* 09:50 hashar: Building container releng/zuul-cloner:0.2.0 and switching jobs # [[phab:T224908|T224908]]
* 06:25 Majavah: switch w-beta.wmflabs.org web proxy to deployment-mediawiki11
* 08:25 hashar: Removing mysql-server from the CI slaves # https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517095/ {{!}} [[phab:T225735|T225735]]
* 06:18 Majavah: restart restbase on deployment-restbase03 to pick up config changes to use deployment-mediawiki11


== 2019-06-22 ==
== 2021-03-29 ==
* 11:36 hauskatze: github: deleted `wikimedia/mediawiki-extensions-FirstSteps` GitHub mirror refs. [[phab:T226149|T226149]]
* 15:37 Majavah: hard reboot deployment-sessionstore03 [[phab:T263617|T263617]]
* 00:31 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/518344
* 15:16 Majavah: manually run puppet on deployment-sessionstore03, starting Cassandra (which was stopped) [[phab:T263617|T263617]]
* 13:04 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675503/ on deployment-puppetmaster04 ([[phab:T278664|T278664]]), also apply same change on horizon. this will switch traffic from deployment-mediawiki-07 to deployment-mediawiki11
* 10:29 Majavah: remove deployment-mediawiki10, too much live debugging, not in use
* 09:56 Majavah: taavi@deployment-mediawiki10:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/aeffde42.0 && sudo update-ca-certificates
* 09:29 Urbanecm: Manually run puppet on mediawiki10
* 09:28 Urbanecm: Re-enable puppet on mediawiki10
* 08:49 Urbanecm: DIsable puppet on mediawiki10 - investigating failing curl certificate check
* 06:46 Majavah: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357/ on deployment-puppetmaster04 - [[phab:T278664|T278664]]
* 05:40 Majavah: move role::labs::lvm::srv puppet classes from deployment-mediawiki- prefix to current individual appservers, [[phab:T278664|T278664]]


== 2019-06-21 ==
== 2021-03-26 ==
* 20:52 Krinkle: Add "post-build scripts" Jenkins Console section to collapse most unrelated output and cascading failures from artefact readers etc. which are confusing when looking for "the" error
* 08:16 hashar: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/675004
* 20:48 Krinkle: Fix Jenkins Console Section pattern for "setup quibble mw-install" to not eat all of output from phan (differentiate from phpunit, composer-test and phan). regexcomplexity++
* 07:05 Majavah: delete remaining shutdown deployment-prep jessies: deployment-sca[01-02], deplyoment-logstash2 ([[phab:T218729|T218729]])
* 16:58 James_F: Updated all four codehealth jjb jobs to use java8-sonar-scanner 0.4.10
* 16:55 James_F: Docker: Publishing java8-sonar-scanner 0.4.10.
* 06:37 legoktm: deployed https://gerrit.wikimedia.org/r/518205 https://gerrit.wikimedia.org/r/518206
* 06:34 legoktm: deployed https://gerrit.wikimedia.org/r/518204
* 06:04 legoktm: deleted remexhtml-phpunit-coverage-docker-publish jenkins job
* 05:51 legoktm: deployed https://gerrit.wikimedia.org/r/518195 https://gerrit.wikimedia.org/r/518196


== 2019-06-20 ==
== 2021-03-25 ==
* 22:03 James_F: Pushing new java8-sonar-scanner:0.4.9 image.
* 20:20 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/674964
* 21:29 James_F: Pushing new java8-sonar-scanner:0.4.8 image
* 16:24 Majavah: install openssl security update and restart trafficserver-tls on deployment-cache-*
* 20:06 James_F: Add new mwcore-codehealth-* config to Zuul
* 03:24 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/674749
* 20:04 James_F: Pushed new jjb mwcore-codehealth-* jobs and updated mwext-codehealth-* ones
* 19:54 James_F: Pushing new java8-sonar-scanner:0.4.7 image
* 17:45 James_F: Updated JJB definitions for mwext-php7\d-phan(-seccheck)?-docker for new docker-setup-mwext-for-phan macro.
* 17:35 James_F: Updated JJB definitions for wikibase-{client{{!}}repo}-docker to use php72 not php70.
* 17:12 James_F: Updated JJB definitions for all six coverage jobs
* 15:09 hashar: Nuking castor caches for Quibble jobs  integration-castor03 : rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/*quibble*/{composer,npm}
* 14:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/518033/
* 14:16 hashar: building some container for codehealth pipeline  https://gerrit.wikimedia.org/r/518034


== 2019-06-19 ==
== 2021-03-24 ==
* 08:58 hashar: Removed Debian packages for python / tox etc from all slaves {{!}} [[phab:T225735|T225735]] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517092/
* 07:42 Majavah: remove deployment-logstash2 hiera from horizon, instahce was shut off earlier by moritzm [[phab:T238707|T238707]]
* 08:46 hashar: Removed Debian package colordiff  from all slaves {{!}} [[phab:T225735|T225735]] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517091/
* 03:43 legoktm: deploying https://gerrit.wikimedia.org/r/517785 https://gerrit.wikimedia.org/r/517786


== 2019-06-18 ==
== 2021-03-23 ==
* 23:19 James_F: Making extension-seccheck non-voting on all 195 repos, hopefully temporarily. [[phab:T226064|T226064]]
* 20:46 James_F: Zuul: [mediawiki/extensions/CopyToClipboard] Archive per [[phab:T274015|T274015]]
* 22:14 Reedy: Apply https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OATHAuth/+/496451/53/sql/mysql/patch-add_generic_fields.sql to centralauth on beta [[phab:T218210|T218210]]
* 20:44 James_F: Zuul: [mediawiki/extensions/Wikibase] Add experimental Postgres job [[phab:T207226|T207226]]
* 21:59 James_F: Manually pruned some old quibble-coverage Docker images from contint1001.
* 19:33 James_F: Zuul: [operations/homer/public] Add bespoke tox-publish job
* 21:51 James_F: Updating docker-pkg files on contint1001 to fix quibble-stretch-php72
* 04:45 James_F: dockerfiles: [quibble-buster] Switch npm to our own build, and cascade [[phab:T252434|T252434]]
* 21:43 James_F: Zuul: Adding php72 variants of phan-seccheck jobs as experimental
* 02:39 James_F: Zuul: Make php72_buster jobs voting for skin-quibble template [[phab:T252434|T252434]]
* 21:35 James_F: Pushing docker image quibble-stretch-php72:0.3.3
* 02:23 James_F: Zuul: [mediawiki/vendor] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 20:40 James_F: Reload Zuul to point MediaWiki's phan jobs at php72 instead of php70 [[phab:T225456|T225456]]
* 01:35 James_F: Zuul: [mediawiki/core] Make php72_buster jobs voting for master branch [[phab:T252434|T252434]]
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517733/
* 00:03 brennen: re-associating floating IP for gitlab-test to gitlab-ansible-test box for speed & function use
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]]
* 20:22 James_F: Reload Zuul to point Parsoid's experimental phan job at php72 instead of php70 [[phab:T225456|T225456]]
* 20:12 James_F: Reload Zuul to point extension-phan and skin-phan templates at php72 instead of php70 [[phab:T225456|T225456]]
* 19:51 hashar: building containers for php7.2 from SRE # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517684/
* 19:38 James_F: Bumped jjb mwext-codehealth-*, mediawiki-*coverage*docker*, mwext*coverage*docker* jobs to use quibble-coverage:0.0.31-4
* 19:28 James_F: Pushing quibble-coverage:0.0.31-4 docker image.
* 17:02 James_F: Updated jjb jobs for mwext-codehealth-patch and mwext-codehealth-master-non-voting.
* 16:54 James_F: Pushing new quibble-coverage docker image
* 11:32 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-PureWikiDeletion {{!}} [[phab:T225991|T225991]]


== 2019-06-15 ==
== 2021-03-22 ==
* 00:17 Krinkle: Disable "Publish/Notify" for mediawiki/libs/ObjectFactory.git in Diffusion –&nbsp;[[phab:T222409|T222409]], [[phab:T143162|T143162]]
* 23:06 James_F: Zuul: [labs/tools/majavah-bot] Run generic tox tests
* 19:28 James_F: Zuul: [mediawiki/services/function-orchestrator] Add code coverage job
* 12:07 Majavah: delete deployment-restbase[01-02], [[phab:T250574|T250574]]
* 11:36 dcaro: Created subzone svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T276624|T276624]])
* 11:33 dcaro: Created subzone beta.wmcloud.org ([[phab:T276624|T276624]])


== 2019-06-14 ==
== 2021-03-19 ==
* 22:15 James_F: Dropped jjb jobs trending-edits?(-deploy)-npm-node-6-docker
* 15:11 dpifke: Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta.
* 22:14 James_F: Reloading Zuul to archive the trending-edits repos
* 13:35 addshore: reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750)
* 21:50 James_F: Dropped docker-registry.discovery.wmnet/releng/hhvm-jessie:latest, unused
* 12:48 Majavah: shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, [[phab:T218729|T218729]]
* 19:57 James_F: Ran `docker rmi docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-4` on contint1001; unused.
* 11:49 Majavah: remove now-deleted deployment-cumin02 from hiera allowed cumin masters
* 19:41 James_F: Zuul: Deploying Codehealth: Add Translate extension to pipeline
* 08:54 Majavah: remove deployment-restbase02 from cassandra and shut it down [[phab:T250574|T250574]]
* 19:39 James_F: Zuul: Deploying Remove coverage ci step for extensions with codehealth
* 08:45 Majavah: disable puppet and stop restbase service on deployment-restbase02 for [[phab:T250574|T250574]]
* 19:35 James_F: Zuul: Deploying Add sonar CodeHealth checks to Wikibase repos
* 18:44 James_F: Publishing image docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-5
* 16:58 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/517087
* 16:08 James_F: Updating docker-pkg to provide quibble-coverage on php7.2
* 16:04 James_F: Dropped docker-registry.discovery.wmnet/releng/quibble-stretch-php70:0.0.31-1 from contint1001, unused
* 15:50 James_F: Bumped jjb mediawiki-phpunit-coverage-patch-docker


== 2019-06-13 ==
== 2021-03-18 ==
* 22:30 James_F: Deploying quibble-stretch-php70 update.
* 22:29 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw ([[phab:T277109|T277109]])
* 22:17 James_F: Deploying quibble-coverage fix with chmod 0755 files.
* 22:28 marxarelli: staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199
* 21:49 James_F: Reverted JJB jobs mediawiki-core-code-coverage-docker and mediawiki-phpunit-coverage-patch-docker back to quibble-stretch-php70
* 18:27 brennen: updating gitlab-test to 13.9.4-ce
* 21:43 James_F: Updated JJB job mediawiki-phpunit-coverage-patch-docker to point to quibble-coverage
* 18:03 James_F: zuul: [mediawiki/extensions/Wikibase] Use composer not vendor on REL1_35 [[phab:T277750|T277750]]
* 21:42 James_F: Update JJB job mediawiki-core-code-coverage-docker to point to quibble-coverage
* 17:39 marxarelli: deploying https://gerrit.wikimedia.org/r/c/blubber/+/671199 to staging
* 21:10 brennen: restarted jenkins agent on integration-trigger-01 to pick up new root configuration
* 16:48 hashar: Purging openjdk-8 packages from Jenkins agents # [[phab:T269354|T269354]]
* 20:54 hashar: ¡Learning to log one step at a time!
* 16:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/673208 and https://gerrit.wikimedia.org/r/673211 [[phab:T277750|T277750]] (apitests php versions)
* 20:54 James_F: Updating docker-pkg files to create quibble-coverage image
* 15:55 addshore: reload zuul for Introduce query-builder job so it can use npm 6.14.* instead [integration/config] - https://gerrit.wikimedia.org/r/673183 [[phab:T277060|T277060]]
* 20:40 hashar: deleting docker-registry.wikimedia.org/releng/hhvm-jessie-compile from contint1001 # [[phab:T224908|T224908]]
* 15:52 hashar: Disconnecting a bunch of Jenkins agents to upgrade them to Java 11  # [[phab:T269354|T269354]]
* 18:17 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/514016
* 13:20 Majavah: manually systemctl daemon-reload && systemctl start srv-swift\\x2dstorage-lv\\x2da1.mount on deployment-ms-be* nodes for [[phab:T276179|T276179]]
* 17:05 brennen: changing jenkins remote root for integration-trigger-01 to /mnt/home/jenkins-deploy per [[phab:T224069|T224069]]
* 09:10 addshore: reload zuul for Remove mwselenium-quibble-docker [integration/config] - https://gerrit.wikimedia.org/r/673206
* 16:07 James_F: Dropped docker-registry.discovery.wmnet/releng/npm-test-mathoid:latest, unused
* 08:44 Majavah: delete now unused deployment-ircd [[phab:T277081|T277081]]
* 16:06 James_F: Dropped mathoid?(-deploy)-npm-node-6-docker from jjb
* 08:40 Majavah: delete deployment-db06, 07/08 have been working fine for a week now
* 16:01 James_F: Dropped docker-registry.discovery.wmnet/releng/npm6:latest from contint1001, unused
* 15:33 Krinkle: Disable "Publish/Notify" for quibble.git in diffusion, [[phab:T143162|T143162]], [[phab:T196347|T196347]]
* 00:13 James_F: Dropped jjb jobs for (change-propagation{{!}}citoid{{!}}cxserver{{!}}graphoid)?(-deploy)-npm-node-6-docker


== 2019-06-12 ==
== 2021-03-17 ==
* 23:34 brennen: deleting and rebuilding integration-trigger-01 (again) to use jessie instead of stretch
* 20:30 hashar: Reloaded Zuul for {{Gerrit|I2368478e4c4ab8752581f55a7c5ab493fafdeb41}}
* 23:20 James_F: Zuul: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now
* 15:37 Majavah: shutdown deployment-restbase01 for [[phab:T250574|T250574]]
* 22:44 James_F: Enabling service-pipeline-test on four services: 3d2png, chromium-render, recommendation-api, trending-edits
* 15:32 Majavah: taavi@deployment-restbase01:~$ sudo nodetool decomission # [[phab:T250574|T250574]]
* 22:40 James_F: Dropped jjb jobs wikimedia-cz-tracker-npm-node-6-docker fundraising-dash-npm-node-6-docker wikimedia-portals-npm-browser-node-6-docker wikimedia-portals-deploy-npm-browser-node-6-docker
* 14:53 addshore: reload zuul for https://gerrit.wikimedia.org/r/673028 Run more Wikibase tests jobs for REL1_35 branch
* 22:38 brennen: deleting and re-creating integration-trigger-01
* 01:21 James_F: Zuul: [labs/tools/wikisource-ocr] Remove CI
* 22:22 James_F: Dropped mapdata-npm-node-6-docker and marvin-npm-node-6-docker from jjb
* 22:21 James_F: Reloading Zuul to make mpadata use generic-node10-browser-docker
* 21:54 James_F: Reloading Zuul to deploy, umm, 12 patches. 🤷🏽‍♂️
* 21:07 James_F: Updating Zuul with https://gerrit.wikimedia.org/r/516674 https://gerrit.wikimedia.org/r/516675 https://gerrit.wikimedia.org/r/516676
* 18:35 James_F: Reloaded Zuul to use node10 things for OOUI experimentally
* 18:32 James_F: Deployed new jjb job ooui-npm-special-node10-plus-composer-docker
* 16:28 Krinkle: Fix Jenkins Console section "setup quibble mw-install" pattern to not slurp the whole output on 'mediawiki-quibble-composertest' jobs
* 16:27 James_F: Reloading Zuul to add service-pipeline-test?(-and-publish) to mediawiki/services/mobileapps
* 00:49 James_F: Updated Zuul phan dependencies for ContentTranslation, adding GlobalPreferences


== 2019-06-11 ==
== 2021-03-16 ==
* 23:51 James_F: Updated Zuul phan dependencies for WikibaseLexeme, adding Wikibase [[phab:T225524|T225524]]
* 21:37 longma: Updating dev-images docker-pkg files on primary contint
* 23:37 James_F: Dropped unused jjb jobs mwext-jsduck-publish mwext-VisualEditor-jsduck-docker, and mwext-VisualEditor-docker-publish
* 21:22 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/672792
* 23:34 James_F: Updating Zuul to drop extension-jsduck, no longer used (yay)
* 20:06 marxarelli: restarting zuul due to seemingly stuck dependency chain
* 23:14 James_F: Reloading Zuul to have Wikibase use extension-javascript-documentation
* 16:22 James_F: Docker: Publishing quibble-stretch-php72-apache:0.0.46-s1
* 22:39 James_F: PHP55 docker images deleted; needed to specify :0.… and also :latest because we're not just dropping one tag, we're dropping the double pair. All now done. [[phab:T224907|T224907]]
* 10:29 addshore: reload zuul for https://gerrit.wikimedia.org/r/670898 Add configuration for new wikidata/query-builder repo
* 22:34 James_F: PHP55 docker images not deleted because they're still referenced (by what?).
* 10:17 hashar: Building docker-registry.wikimedia.org/releng/sonar-scanner:4.6.0.2311-1 # [[phab:T277527|T277527]]
* 22:25 James_F: Running `docker rmi` for quibble-jessie-php55 ({{Gerrit|938a4b4f6abb}}) composer-test-php55 ({{Gerrit|9731d99e7a6f}}) composer-package-php55 ({{Gerrit|906edee88f4f}}) php55-compile ({{Gerrit|01f3fa5ef542}}) composer-php55 ({{Gerrit|a14836c5fc62}}) and php55 ({{Gerrit|f8b5dc437239}})
* 22:22 James_F: Updating docker-pkg files on contint1001 to drop all php55 images [[phab:T224907|T224907]]
* 22:18 James_F: Dropped from jjb: mwgate-composer-php55-docker, composer-php55-docker, composer-package-php55-docker, php-compile-php55-docker
* 22:06 James_F: Drop unused jjb job integration-composer-check-php55-docker
* 22:06 James_F: Reloading Zuul to drop integration-composer-check-php55-docker from integration/composer
* 22:01 James_F: Reloading Zuul to add integration-composer-check-php72-docker to integration/composer
* 21:59 James_F: Push new jjb job integration-composer-check-php72-docker
* 21:56 James_F: Reloading Zuul to drop all *php55* jobs except integration-composer
* 21:49 James_F: Reloading Zuul to add composer-php56-docker to integration/docroot
* 21:45 James_F: Reloading Zuul to drop quibble*php55*
* 21:42 James_F: Reloading Zuul to add composer-php72-docker to labs/tools/heritage
* 21:07 brennen: creating integration-trigger-01 for [[phab:T224069|T224069]]
* 19:34 hashar: building docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.4.5 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/516511/
* 19:04 halfak: deployment-prep deploying ores {{Gerrit|04fbd58}}
* 19:03 mdholloway: deployment-prep created deployment-wikifeeds01, added role::beta::docker_services and service config
* 18:29 halfak: deployment-prep deploying ores {{Gerrit|86809a4}}
* 13:17 hashar: Building container docker-registry.wikimedia.org/releng/civicrm:0.1.0 # [[phab:T210287|T210287]]


== 2019-06-10 ==
== 2021-03-15 ==
* 19:50 James_F: Restarting zuul to drop quibble experimental php70-phan-docker jobs; php72 suffices
* 08:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/670782
* 19:39 thcipriani: restarting jenkins
* 08:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/672118
* 19:37 James_F: Restarting jenkins with `sudo service restart jenkins` on contint1001 due to out-of-space issue earlier.
* 19:10 Krinkle: Fix "setup quibble mw-install" collapsible section pattern in Jenkins administration, so that it doesn't match the "PHP 7.1+" warning from Phan. The pattern is meant to detect the first line of mw/maint/install.php output
* 19:01 thcipriani: cleaning contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 17:48 James_F: Reloading zuul to set zuul parameters for PHP 7.2 variants of phan jobs too
* 17:41 James_F: Reloading zuul to Add experimental mwext-php72-phan-docker jobs to quibble template
* 17:38 James_F: Pushed new mwext-php72-phan-docker with jjb
* 16:59 James_F: Deployed https://gerrit.wikimedia.org/r/516300
* 16:31 legoktm: deployed https://gerrit.wikimedia.org/r/516298
* 04:38 legoktm: deploying https://gerrit.wikimedia.org/r/516065 https://gerrit.wikimedia.org/r/516066


== 2019-06-08 ==
== 2021-03-13 ==
* 00:49 James_F: Updating docker-pkg files on contint1001 to build mediawiki-phan:0.2.0 based on php72 [[phab:T223847|T223847]]
* 17:10 twentyafterfour: restart apache on gerrit1001


== 2019-06-07 ==
== 2021-03-12 ==
* 23:29 marxarelli: deploying zuul changes for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 22:57 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295
* 23:28 marxarelli: updating jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 22:28 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199
* 13:30 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/515058
* 19:13 Majavah: taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
* 00:10 James_F: jjb: Migrated docker-setup-mwext-for-phan to node10 quibble
* 19:01 legoktm: legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06
* 14:10 addshore: reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ [[phab:T276428|T276428]]


== 2019-06-06 ==
== 2021-03-11 ==
* 18:06 James_F: Reloading Zuul to drop EventBus dependency on CentralNotice for now [[phab:T225195|T225195]]
* 22:11 marxarelli: reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job([[phab:T277236|T277236]])
* 15:12 hashar: reloading zuul for {{Gerrit|I044bedd41091b57a1a188ea243c4769a6e21babb}} [[phab:T222802|T222802]]
* 22:10 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 15:12 hashar: reloading Zuul for {{Gerrit|I8d6ea5b5491fe50d7144926ef8bc69d4704d4fab}} [[phab:T224301|T224301]]
* 22:03 marxarelli: running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs ([[phab:T277236|T277236]])
* 08:13 hashar: Reloading Zuul for {{Gerrit|I764972711843645afd00e196a3bedd17730b4cbe}}  which drops  mwselenium-quibble-docker from Wikibase
* 16:58 Majavah: copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 [[phab:T276419|T276419]]
* 16:49 Majavah: delete deployment-etcd-01 [[phab:T276462|T276462]]
* 13:51 Majavah: shut down deployment-db06, now unused [[phab:T277070|T277070]]
* 13:48 Majavah: set deployment-db07 as r/w [[phab:T277070|T277070]]
* 13:48 Majavah: stop mariadb to ensure reads have stopped on deployment-db06 [[phab:T277070|T277070]]
* 13:41 Majavah: stop slave on deployment-db06 [[phab:T276968|T276968]]
* 13:37 Majavah: make deployment-db06 and deployment-db08 be replicas of deployment-db07 [[phab:T277070|T277070]]
* 13:34 Majavah: stop and reset slave on deployment-db07 [[phab:T277070|T277070]]
* 13:32 Majavah: set deployment-db06 as read only [[phab:T277070|T277070]]
* 09:12 hashar: Updated plugins on https://releases-jenkins.wikimedia.org/


== 2019-06-05 ==
== 2021-03-10 ==
* 19:34 andrewbogott: moving deployment-imagescaler03 to cloudvirt1029
* 19:36 Majavah: shutdown deployment-ircd [[phab:T277081|T277081]]
* 17:35 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547
* 18:46 Majavah: switch floating ip 185.15.56.34 to deployment-ircd02 [[phab:T277081|T277081]]
* 16:01 James_F: Reloading Zuul to deploy {{Gerrit|172a4e7886b285adb48a85dce7287dce5c3778e2}}
* 18:05 Majavah: create deployment-ircd02 for [[phab:T277081|T277081]]
* 09:07 hashar: Pooled in integration-slave-docker-1058 and integration-slave-docker-1059
* 17:26 marxarelli: `rm -rf /srv/dump` on deployment-db06 and reenabling puppet
* 17:25 marxarelli: `rm -rf /srv/restore` on deployment-db08 and reenabling puppet
* 17:24 marxarelli: `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet
* 17:09 Majavah: set beta cluster mediawiki as read write on mw config ([[phab:T276968|T276968]])
* 17:03 Majavah: make deployment-db06 read-write [[phab:T276968|T276968]]
* 16:50 Majavah: `reset slave;` on new master deployment-db06 [[phab:T276968|T276968]]
* 16:49 Majavah: add deployment-db07 as a replica of db06 for [[phab:T276968|T276968]]
* 16:45 Urbanecm: root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # [[phab:T276968|T276968]]
* 16:12 Majavah: deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; ([[phab:T276968|T276968]])
* 16:06 Urbanecm: start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . ([[phab:T276968|T276968]])
* 15:57 Majavah: set deployment-db06 as readonly from mysql side [[phab:T276968|T276968]]
* 15:54 Urbanecm: Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` ([[phab:T276968|T276968]])
* 15:54 Urbanecm: Start mariadb on db08 ([[phab:T276968|T276968]])
* 15:22 Urbanecm: rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 ([[phab:T276968|T276968]])
* 14:52 Majavah: delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199
* 10:16 arturo: briefly stopping deployment-puppetdb03 to disable VMX CPU flag
* 00:28 marxarelli: mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade ([[phab:T276968|T276968]])
* 00:10 marxarelli: restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) ([[phab:T276968|T276968]])


== 2019-06-04 ==
== 2021-03-09 ==
* 20:39 James_F: Updating jjb PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [[phab:T220917|T220917]]
* 21:54 marxarelli: restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround ([[phab:T276968|T276968]])
* 20:31 James_F: Updating docker-pkg files on contint1001 for quibble-stretch-php70 [[phab:T220917|T220917]]
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 ([[phab:T276968|T276968]])
* 17:44 James_F: Updating jjb PHP code coverage jobs to node10 quibble [[phab:T224983|T224983]]
* 20:53 marxarelli: restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127
* 17:31 James_F: Reloading Zuul to make mwext-MobileFrontend-npm-run-lint-modules-docker non-voting [[phab:T224997|T224997]]
* 20:39 marxarelli: doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user ([[phab:T276968|T276968]])
* 16:10 hashar: Deleting  integration-slave-docker-1021 and integration-slave-docker-1049  / too small disk (20G partition) and not enough ram (2G) # [[phab:T221872|T221872]]
* 20:33 Majavah: install mariadb on deployment-db08 [[phab:T276968|T276968]]
* 13:24 hashar: Update all selenium-daily* jobs to use NodeJS 10 instead of NodeJS 6. [[phab:T217545|T217545]]
* 19:59 marxarelli: creating new instance deployment-db08 to use as new beta replica db ([[phab:T276968|T276968]])
* 13:01 hashar: Building docker-registry.discovery.wmnet/releng/node10-test-browser:0.6.0 # [[phab:T217545|T217545]]
* 19:56 marxarelli: deleting deployment-db05 to free up quota for new replica ([[phab:T276968|T276968]])
* 11:32 hashar: Upgrading Jenkins Pipeline plugins
* 19:50 marxarelli: restoring database dump on deployment-db07 ([[phab:T276968|T276968]])
* 11:31 hashar: Upgrading Jenkins Warnings Next Generation Plugin # [[phab:T224745|T224745]]
* 18:49 marxarelli: restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` ([[phab:T276968|T276968]])
* 11:30 hashar: Upgrading Jenkins BlueOcean and all its dependencies
* 18:38 Majavah: installing mariadb 10.4 via role::mariadb::beta to db07 [[phab:T276968|T276968]]
* 18:25 marxarelli: "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 ([[phab:T276968|T276968]])
* 18:21 Majavah: create deployment-db07 as g2.cores8.ram16.disk160 Buster [[phab:T276968|T276968]]
* 18:20 marxarelli: disabled puppet on deployment-db06 and started mysqldump ([[phab:T276968|T276968]])
* 18:09 Majavah: set deployment-db05 to read-only to avoid issues with [[phab:T276968|T276968]]
* 18:04 marxarelli: deleting shut down memc* deployment-prep instances to free up quota for replacement db instances ([[phab:T276968|T276968]])
* 17:25 marxarelli: seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05
* 17:22 marxarelli: restarting deployment-db05 via horizon
* 17:22 marxarelli: deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up
* 16:47 marxarelli: still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors
* 16:27 marxarelli: builds once again being scheduled on deployment-deploy01
* 16:24 marxarelli: cycling gearman plugin on integration.wikimedia.org
* 16:16 marxarelli: taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs
* 13:32 arturo: hard-reboot deployment-db05 because issues related to [[phab:T276922|T276922]]
* 12:34 arturo: briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other


== 2019-06-03 ==
== 2021-03-08 ==
* 19:32 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514089
* 21:38 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159
* 19:29 James_F: Generated mwgate-node10-docker and deployed via jjb.
* 08:57 hashar: Nuked castor cache for labs/striker # [[phab:T276605|T276605]]
* 18:34 hashar: hswitch most Quibble jobs to node 10 [[phab:T222406|T222406]] - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ [[phab:T222406|T222406]]
* 17:40 James_F: Reloading Zuul to switch most Quibble jobs to node 10 [[phab:T222406|T222406]]
* 15:58 hashar: Deleting integration-slave-docker-1055 and integration-slave-docker-1056 . CPU is way too slow [[phab:T223971|T223971]]
* 15:46 hashar: reduce number of executors on all integration slave docker, they are somehow starving on CPU and/or IO when lot of mediawiki builds are running in parallel
* 14:38 James_F: hashar and I are temporarily disabling running selenium tests in CI. See [[phab:T211784|T211784]] [[phab:T222406|T222406]] for more details.


== 2019-06-01 ==
== 2021-03-07 ==
* 22:54 James_F: Reloading Zuul to re-add Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 17:46 James_F: Deleting deployment-snapshot01, shut off since 2020-10-03.
* 21:14 James_F: Reloading Zuul to add phan for CentralNotice and add DannyS712 to whitelist
* 17:43 James_F: Deleting deployment-cumin02, shut off since 2020-10-16.
* 17:18 Majavah: shutdown deployment-memc[04-05] [[phab:T276707|T276707]]
* 16:51 Majavah: cherry pick 669436 and 669436 to deployment-puppetmaster04 [[phab:T276707|T276707]]
* 15:52 Majavah: redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 [[phab:T276707|T276707]]
* 15:44 Majavah: create deployment-memc10 on Buster [[phab:T276707|T276707]], beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted
* 15:28 Majavah: remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, [[phab:T276707|T276707]] [[phab:T250585|T250585]]
* 13:14 Majavah: create deployment-memc09 on Buster [[phab:T276707|T276707]]


== 2019-05-31 ==
== 2021-03-06 ==
* 23:34 James_F: Reloading Zuul to remove Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 19:45 Majavah: restart deployment-logstash03 to see if it fixes it being empty
* 22:12 James_F: Reloading Zuul to add phan for LdapAuthentication
* 09:48 Majavah: cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 [[phab:T276654|T276654]]
* 21:18 James_F: Reloading Zuul to add dependencies for TimedMediaHandler, ReadingLists, JsonConfig, and FlaggedRevs.
* 08:09 Majavah: deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix [[phab:T276652|T276652]]
* 21:13 James_F: Reloading Zuul to add phan for Collection, ContentTranslation, and Jade
* 19:43 James_F: Reloading Zuul to add phan for Math(s)
* 19:31 James_F: Reloading Zuul to add phan for MobileFrontend
* 19:20 James_F: Reloading Zuul to add phan for Sentry
* 18:35 James_F: Reloading Zuul to deploy CentralNotice dependency to EventBus
* 18:17 James_F: Reloading Zuul to deploy extra dependencies for MobileFrontend
* 18:10 James_F: Reloading Zuul to deploy phan for cldr and GlobalPreferences
* 18:02 James_F: Reloading Zuul to deploy {{Gerrit|I0ee1c8166}}
* 17:56 James_F: Reloading Zuul to deploy {{Gerrit|I9c115b1f6}} and {{Gerrit|I0d98f01d}}


== 2019-05-30 ==
== 2021-03-05 ==
* 23:19 James_F: Pruned releng/quibble-fresnel:0.0.31-3 {{Gerrit|8ca03484a2c3}} from contint1001 to save space
* 20:25 James_F: Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 23:02 James_F: Updating docker-pkg files on contint1001 for +quibble-fresnel (0.0.31-4)
* 20:23 James_F: Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more [[phab:T250585|T250585]]
* 21:33 Krinkle: Jenkins admin says ""Your Jenkins data directory /var/lib/jenkins (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full.""
* 19:36 Majavah: release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
* 21:27 greg-g: back in business (ugh)
* 19:30 Majavah: shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week [[phab:T276462|T276462]]
* 21:21 greg-g: doing the zuul deadlock dance again
* 19:15 Majavah: beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref [[phab:T276462|T276462]]
* 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/513168
* 17:50 Majavah: switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref [[phab:T276462|T276462]]
* 02:48 legoktm: deploying https://gerrit.wikimedia.org/r/512474 https://gerrit.wikimedia.org/r/512507 https://gerrit.wikimedia.org/r/513153 https://gerrit.wikimedia.org/r/513157 https://gerrit.wikimedia.org/r/512729 https://gerrit.wikimedia.org/r/513206 https://gerrit.wikimedia.org/r/513214 https://gerrit.wikimedia.org/r/513217 https://gerrit.wikimedia.org/r/513218 https://gerrit.wikimedia.org/r/513211
* 13:40 Majavah: create deployment-etcd02 and sign its puppet certificate [[phab:T276462|T276462]]
* 02:35 legoktm: deployed https://gerrit.wikimedia.org/r/513129 https://gerrit.wikimedia.org/r/513071
* 13:13 Majavah: move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
* 11:48 Majavah: live hack beta puppetmaster to fix hopefully trust store location; [[phab:T276521|T276521]] and possibly others
* 08:32 Majavah: deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair [[phab:T241481|T241481]], directory is different on deployment-logstash2


== 2019-05-29 ==
== 2021-03-04 ==
* 21:46 greg-g: doing the executor deadlock dance: https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Jenkins_execution_lock
* 15:47 hashar: Refreshing jobs based on releng/tox-buster to use latest image.  That brings in tox installed with python3 instead of python2 # [[phab:T276384|T276384]]
* 15:00 Majavah: remove graphoid role from deploymenr-sca[01-02] ref [[phab:T276102|T276102]] and it being decomissioned in [[phab:T242855|T242855]]
* 13:18 Majavah: shutdown deployment-fluorine02 for a scream test for [[phab:T276419|T276419]], I believe everything has been moved to deployment-mwlog01
* 12:38 Majavah: `git rebase origin/production` on deployment-puppetmaster04 to update few settings for [[phab:T276419|T276419]]
* 12:19 Majavah: Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon
* 12:06 Majavah: deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir
* 11:02 Majavah: live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref [[phab:T276419|T276419]]
* 10:51 Majavah: stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is
* 09:20 hashar: Restored analytics/udp2log cause it got to be packaged for Buster # [[phab:T276422|T276422]] [[phab:T180301|T180301]]
* 07:47 legoktm: rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259
* 06:33 Majavah: create Buster VM deployment-mwlog01 to eventually replace deployment-fluorine02 which is still on Stretch


== 2019-05-28 ==
== 2021-03-03 ==
* 12:43 paladox: delete extensions/3D repo in gerrit - [[phab:T224463|T224463]]
* 20:30 legoktm: added Majavah as projectadmin in deployment-prep (Beta Cluster)
* 01:35 legoktm: deployed https://gerrit.wikimedia.org/r/512794
* 19:59 James_F: Zuul: [mediawiki/services/function-schemata] Revert "Use bespoke pipeline jobs"
* 16:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Run phan with Echo
* 11:40 addshore: reload zuul for https://gerrit.wikimedia.org/r/667623
* 10:33 hashar: REplaced java-codehealth-patch job in favor of running sonar:sonar inline in all the java jobs.  Thanks gehel!  # [[phab:T264873|T264873]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/666869/


== 2019-05-26 ==
== 2021-03-02 ==
* 18:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/512544
* 22:22 Krinkle: Run `sudo systemctl restart memcached` on deployment-mediawiki-07
* 22:22 Krinkle: Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git).


== 2019-05-22 ==
== 2021-03-01 ==
* 07:21 hashar: Updating Jenkins job to have castor use rsync --delay-updates # [[phab:T203506|T203506]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/511672/
* 18:25 marxarelli: deleting unused docker-registry-uploader jenkins credential
* 14:41 andrewbogott: changed profile::redis::multidc::discovery from 'false' to "" to comply with strict typing in the deployment-memc puppet prefix.


== 2019-05-21 ==
== 2021-02-27 ==
* 22:05 marxarelli: generating new blubber-pipeline-* and trigger-blubber-pipeline-* jenkins jobs defined by https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 22:03 Reedy: re-armed beta keyholder... I think...
* 11:23 hashar: Depooling integration-slave-docker-1055 and integration-slave-docker-1056 : CPU is too slow # [[phab:T223971|T223971]]
* 08:24 hashar: Updated phan jobs to no more install mediawiki development dependencies (potentially conflicting with the extensions code) https://gerrit.wikimedia.org/r/#/c/integration/config/+/511447/ # [[phab:T223397|T223397]]
* 07:29 legoktm: deployed https://gerrit.wikimedia.org/r/511627
* 07:10 legoktm: manually rebuilding mediawiki-core-code-coverage-docker with 5 hour timeout


== 2019-05-20 ==
== 2021-02-26 ==
* 13:11 hashar: updating phan jobs to use docker-registry.wikimedia.org/releng/mediawiki-phan:0.1.15 # [[phab:T219114|T219114]]
* 19:47 James_F: Zuul: [mediawiki/services/geoshapes] Add typescript service CI [[phab:T274380|T274380]]
* 01:43 legoktm: deploying https://gerrit.wikimedia.org/r/511361
* 01:11 legoktm: update credentials in https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ for new ci-build user ([[phab:T275559|T275559]])


== 2019-05-19 ==
== 2021-02-24 ==
* 15:43 hashar: Purging webperformance.integration.eqiad.wmflabs  /tmp directory (inodes full)
* 22:47 James_F: Docker: Actually re-building Rust images for 1.50.0
* 15:43 hashar: Purging all images on Docker CI slaves
* 22:16 legoktm: rebuilding Rust docker images
* 08:03 legoktm: deployed https://gerrit.wikimedia.org/r/511122


== 2019-05-18 ==
== 2021-02-23 ==
* 18:14 Amir1: cherry-picking 511078 on puppetmaster
* 18:20 James_F: Zuul: [mediawiki/services/function-schemata] Add generic pipeline CI
* 12:07 Reedy: reload zuul to deploy https://gerrit.wikimedia.org/r/510967
* 16:24 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Test with Echo (Notifications)


== 2019-05-17 ==
== 2021-02-20 ==
* 22:55 paladox: created mediawiki/extensions/Scribe gerrit repo - [[phab:T223662|T223662]]
* 18:35 James_F: Zuul: [mediawiki/services/function-evaluator] Drop direct CI; uses pipeline
* 00:35 brennen: Updating dev-images docker-pkg files on contint1001
* 18:13 James_F: Zuul: [mediawiki/extensionos/LockAuthor] Enable basic quibble CI


== 2019-05-16 ==
== 2021-02-19 ==
* 09:36 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/composer-test-php72:0.1.0 # [[phab:T223428|T223428]]
* 13:51 hashar: Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897  did not get merged
* 07:35 hashar: integration-slave-jessie-1002: purging all php Debian packages and rerunning puppet. There is some package conflict somewhere :-(
* 13:49 hashar: Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/651791/


== 2019-05-15 ==
== 2021-02-18 ==
* 11:55 hashar: Bring back https://integration.wikimedia.org/ci/computer/integration-castor03/ to restore the central cache behavior. In turn unblocking a wide range of builds
* 01:50 Urbanecm: Kill stuck beta-scap-eqiad job and start a new one to sync beta
* 11:32 hashar: bringing back castor, integration-castor03 is back
* 00:06 brennen: gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata
* 09:35 hashar: Regenerating all CI jobs to disable castor saving entirely
* 02:35 Krenair: Logged into deployment-sca0[12] as root and given them the correct nameservers to try to unbreak things. [[phab:T221654|T221654]]
* 01:00 thcipriani: /usr/local/sbin/keyholder arm


== 2019-05-14 ==
== 2021-02-17 ==
* 09:43 hashar: Upgraded all tox jobs to tox 3.10.0
* 20:05 hashar: Updating all Jenkins jobs for https://gerrit.wikimedia.org/r/664897 # [[phab:T275049|T275049]]
* 09:19 hashar: Building docker containers releng/tox-*:0.4.0
* 17:59 hashar: Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/664680  # [[phab:T275049|T275049]]
* 03:26 James_F: Zuul: [mediawiki/core] PHP 8.0 version of composertest job to experimental


== 2019-05-13 ==
== 2021-02-16 ==
* 20:21 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/509930
* 20:33 brennen: updating gitlab-test.wmcloud.org to 13.8.4-ce.0
* 19:47 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/502606
* 19:58 hashar: Updating Jenkins job wikimedia-fundraising-civicrm-docker to stop cloning the drupal repository # [[phab:T273822|T273822]]
* afk: updating docker-pkg images on contint1001 for https://gerrit.wikimedia.org/r/508019
* 17:02 greg-g: doing the https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update dance
* 14:48 hashar: if you build Docker containers, there is a long delay between it being build/published and it actually being available https://phabricator.wikimedia.org/T222210#5176863  known issue
* 04:18 James_F: Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site
* 13:49 hashar: Building docker image releng/tox:0.4.0
* 04:00 James_F: Zuul: Add Tim Abdullin from S&F to CI allow list
* 10:37 hashar: Rolling CI config change https://gerrit.wikimedia.org/r/508512 which caused some patches to not be processed last week # https://wikitech.wikimedia.org/wiki/Incident_documentation/20190506-zuul / [[phab:T105474|T105474]]


== 2019-05-10 ==
== 2021-02-15 ==
* 18:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/509483
* 15:58 hashar: Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # [[phab:T209953|T209953]]
* 18:08 Reedy: Reloading Zuul to deploy dependancies, new tests, phan
* 17:53 Reedy: Reloading Zuul to deploy 5 more phan patches
* 17:37 Reedy: Reloading Zuul to deploy 5 phan enabling patches
* 17:23 Reedy: Reloading Zuul to deploy various phan additions and one dependancy
* 17:08 Reedy: Reloading Zuul to deploy patches adding phab
* 16:46 Reedy: Reloading Zuul to deploy various dependancy patches


== 2019-05-09 ==
== 2021-02-14 ==
* 08:17 elukey: remove mediawiki memcached nutcracker config from deployment-prep (should be unused) - [[phab:T214275|T214275]]
* 21:31 James_F: Zuul: Add 'check php' support for library repos
* 00:19 thcipriani: updating docker images on contint1001 for https://gerrit.wikimedia.org/r/508929
* 20:12 James_F: Zuul: [mediawiki/services/graphoid] Archive [[phab:T274738|T274738]]
* 00:19 thcipriani: clean docker images on contint1001


== 2019-05-07 ==
== 2021-02-13 ==
* 22:23 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/507871
* 03:50 James_F: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library
* 10:57 hashar: Upgraded Zuul to 2.5.1-wmf8 # [[phab:T105474|T105474]]  [[phab:T140297|T140297]]


== 2019-05-06 ==
== 2021-02-12 ==
* 21:57 thcipriani: update docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508086
* 17:19 brennen: Publishing from dev-images docker-pkg files on primary contint for fr-tech images
* 19:07 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/508012
* 12:05 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 18:37 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508370
* 17:44 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508036
* 13:31 hashar: Jenkins: installed Least Load plugin {{!}} [[phab:T218458|T218458]]


== 2019-05-03 ==
== 2021-02-11 ==
* 20:58 thcipriani: updating docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508006
* 21:44 Krinkle: Logstash in beta is not receiving any events [[phab:T274593|T274593]]
* 20:58 thcipriani: clean docker images from contint1001
* 17:36 James_F: Zuul: [mediawiki/extensions/Acrolinx] Disable running selenium tests
* 17:02 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/507992
* 17:14 James_F: Zuul: [mediawiki/extensions/GoogleAppEngine] Archive the extension [[phab:T274069|T274069]]
* 09:22 hashar: removed zuul debian package from integration-castor03
* 09:50 hashar: Successfully build Docker images for Quibble 0.0.46
* 08:30 hashar: Building Docker image releng/gradle:0.1.0  {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/507872/ {{!}} [[phab:T222199|T222199]]
* 09:07 hashar: Building Quibble 0.0.46 Docker images on contint1001 (it is faster than contint2001)
* 01:24 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/663339


== 2019-05-01 ==
== 2021-02-10 ==
* 17:53 halfak: deploying ores:52e9759
* 22:55 longma: Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796
* 20:59 brennen: Attempting one more update from dev-images docker-pkg on contint2001 for [[phab:T274306|T274306]]
* 18:36 Urbanecm: deployment-prep: Run scap sync-world as jenkins-deploy
* 18:36 Urbanecm: deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension


== 2019-04-30 ==
== 2021-02-09 ==
* 18:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/494778
* 22:27 brennen: once more, with feeling: attempting docker-pkg run for dev-images again
* 17:01 bearND: (beta): Update mobileapps to {{Gerrit|142ba30}}
* 22:09 brennen: attempting to run docker-pkg manually for dev-images on contint2001
* 02:01 hashar: Polled in integration-slave-docker-1055 and ntegration-slave-docker-1056
* 21:08 brennen: Updating dev-images docker-pkg files on primary contint for [[gerrit:635361]] and [[gerrit:632173]]
* 01:35 hashar: Deleting integration-slave-docker-1037 (bigram)  it is too slow for some reason # [[phab:T222023|T222023]]
* 20:10 James_F: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld
* 19:31 James_F: Docker: Building and publishing tox-buster &c. with tox 3.21.4 [[phab:T274232|T274232]]
* 04:43 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref [[phab:T273247|T273247]]
* 04:42 Krinkle: Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify


== 2019-04-29 ==
== 2021-02-08 ==
* 14:04 godog: add dsharpe user
* 21:40 brennen: enabled administrative approval for new account signups on gitlab-test
* 21:40 brennen: upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0
* 17:22 hashar: Built image docker-registry.discovery.wmnet/releng/quibble-buster-php72:0.0.45-s4
* 17:17 hashar: Building some docker images on contint.wikimedia.org
* 16:51 hashar: Now really reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 16:43 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/662729
* 14:55 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658946  # [[phab:T272863|T272863]]
* 09:06 _joe_: cherry-picked  https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 in deployment-prep


== 2019-04-26 ==
== 2021-02-06 ==
* 13:33 Krenair: shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. [[phab:T218729|T218729]]
* 21:13 Reedy: unstuck beta jobs


== 2019-04-25 ==
== 2021-02-04 ==
* 16:35 Krenair: shutting down deployment-ms-fe02 and deployment-poolcounter04 [[phab:T218729|T218729]]
* 11:40 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$
* 00:17 James_F: Zuul: [mediawiki/libs/Minify] Install initial CI [[phab:T273247|T273247]]


== 2019-04-23 ==
== 2021-02-03 ==
* 15:33 paladox: merging the 2.15.13 release into stable-2.15 following https://wikitech.wikimedia.org/wiki/Gerrit#Update_our_repository
* 21:38 James_F: Zuul: Archive VirtualKeyboard extension [[phab:T273801|T273801]]
* 14:41 Krenair: Shut down deployment-ms-be03 and deployment-ms-be04 [[phab:T218729|T218729]]
* 00:30 James_F: Docker: Publish rust images with default-libmysqlclient-dev
* 10:22 Amir1: ores:060fc37 going beta
* 00:24 James_F: Zuul: [mediawiki/extensions/UseResource] Rename from TemplateScripts
* 00:17 James_F: Zuul: Enable CI for mediawiki/libs/Dodo and mediawiki/libs/WebIDL [[phab:T273295|T273295]]


== 2019-04-19 ==
== 2021-02-02 ==
* 19:56 mutante: phab1003 - editing /srv/deployment/phabricator/deployment-cache/.config manually to replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet to fix git cloning issue on first puppet run on new host where somehow tin.eqiad still shows up. fixes puppet run on [[phab:T221389|T221389]]
* 19:43 hashar: Pruning dangling Docker images on contint2001
* 19:39 hasharDinner: Pruning dangling Docker images on contint1001
* 19:28 James_F: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [[phab:T61245|T61245]]
* 11:27 hashar: gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour
* 10:59 hashar: Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # [[phab:T273599|T273599]]


== 2019-04-18 ==
== 2021-02-01 ==
* 19:25 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/504824
* 16:04 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta.
* 10:55 Urbanecm: deployment-prep: Create beta votewiki ([[phab:T272608|T272608]])
* 09:15 hashar: devtools: switched gerrit-prod-1001 to local puppetmaster


== 2019-04-17 ==
== 2021-01-29 ==
* 17:17 andrewbogott: cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504580/ to move off of soon-to-be-shutdown dns recursors
* 18:51 hashar: CI slightly overloaded due to a surge of library updates but is otherwise processing changes


== 2019-04-16 ==
== 2021-01-28 ==
* 17:36 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf {{!}} tee [[phab:T221107|T221107]].php
* 22:15 marxarelli: deleting unused integration-registry-1003 instance
* 17:46 marxarelli: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339
* 17:42 marxarelli: updating jenkins jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/659339


== 2019-04-15 ==
== 2021-01-27 ==
* 10:16 Amir1: ores:8f01d40 going beta
* 10:29 apergos: decommissioned deployment-snapshot01 at last, long since replaced by deployment-snapshot02
* 08:51 hashar: castor: nuked /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwselenium-quibble-docker  # [[phab:T220948|T220948]]


== 2019-04-13 ==
== 2021-01-26 ==
* 21:05 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. [[phab:T91410|T91410]]
* 18:27 marxarelli: restarting jenkins on releases-jenkins.wikimedia.org following plugin updates
* 21:00 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul.
* 18:26 marxarelli: updating pipeline plugins on releases-jenkins.wikimedia.org
* 21:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503669
* 18:26 marxarelli: updating git plugins on releases-jenkins.wikimedia.org
* 18:52 Krinkle: "Your JENKINS_HOME (/var/lib/jenkins) is almost full. "
* 16:06 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449
* 18:16 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/503664
* 05:43 legoktm: reloaded zuul for https://gerrit.wikimedia.org/r/658355
* 17:44 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/502807 (postgres php72)
* 00:06 Krenair: transferred /home from deployment-cache-upload04 to deployment-cache-upload05 and shut down old one
* 00:06 Krenair: transferred /home from deployment-cumin to deployment-cumin02 and shut down old one


== 2019-04-12 ==
== 2021-01-25 ==
* 15:09 Krenair: upload traffic now through cache-upload05
* 15:57 Lucas_WMDE: canceled one beta-scap-eqiad job per https://w.wiki/J5$


== 2019-04-11 ==
== 2021-01-22 ==
* 18:29 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/480463
* 08:07 legoktm: manually started mwcore-phpunit-coverage-master job with 6hr timeout
* 18:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/499334
* 17:50 Reedy: deployed 'quibble-donationinterface-REL1_31-php70-docker' to jenkins
* 16:31 Krenair: deleting deployment-db04, unused and shut down since 28th March. [[phab:T219087|T219087]]
* 16:18 mateusbs17: deployment-chromium0[1-2] Update Proton on to {{Gerrit|8988283}} ([[phab:T213362|T213362]], [[phab:T216191|T216191]], [[phab:T212322|T212322]])


== 2019-04-10 ==
== 2021-01-19 ==
* 23:32 James_F: Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. [[phab:T220653|T220653]]
* 17:11 James_F: Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI [[phab:T271761|T271761]]
* 22:36 James_F: Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches.
* 20:27 paladox: create operations/software/gerrit/plugins/MassBranchCreation repository
* 14:16 hashar: contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 14:13 hashar: contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 13:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/502790
* 12:46 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502786
* 12:40 hashar: contint2001: stopped puppet and zuul-merger for debugging


== 2019-04-09 ==
== 2021-01-18 ==
* 23:08 Krinkle: Reloading Zuul to deploy https://phabricator.wikimedia.org/T220561
* 23:38 James_F: Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker [[phab:T272320|T272320]]
* 17:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502560
* 16:38 Krenair: deleting deployment-db03, unused since [[phab:T216635|T216635]] and shut down since 24th March. [[phab:T219087|T219087]]
* 16:21 bearND: (beta): Update mobileapps to {{Gerrit|3edfcad}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]]) - 3rd time is the charm


== 2019-04-08 ==
== 2021-01-17 ==
* 22:46 hashar: cleaned docker images on  integration-slave-docker-1030 and  integration-slave-docker-1043 :)
* 03:44 James_F: Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job [[phab:T248925|T248925]]
* 20:06 bearND: (beta): Update mobileapps to {{Gerrit|cdb9928}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]])
* 14:47 hashar: hard rebooting integration-slave-docker-1053 OOM / deadlocked
* 13:39 hashar: Deleting integration-slave-docker-1045 again. It uses Stretch instead of Jessie
* 13:21 hashar: integration: fix cumin profiles that got renamed by {{Gerrit|9e0aa8264659799e74c3e815ea39c640e5f05393}}
* 10:15 hashar: Building Docker image  releng/quibble-stretch-php73:0.0.31-2 # [[phab:T220237|T220237]]


== 2019-04-07 ==
== 2021-01-16 ==
* 00:30 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501793
* 23:24 James_F: Docker: Building cascade of new php-ast image [[phab:T271428|T271428]]


== 2019-04-06 ==
== 2021-01-14 ==
* 02:17 legoktm: rebuilding npm-php image again https://gerrit.wikimedia.org/r/501848
* 19:08 James_F: Zuul: [mediawiki/extensions/HeadScript] Add quibble job
* 01:52 legoktm: rebuilding npm-php image https://gerrit.wikimedia.org/r/501847
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/501786 https://gerrit.wikimedia.org/r/501714 https://gerrit.wikimedia.org/r/501707 https://gerrit.wikimedia.org/r/501782 https://gerrit.wikimedia.org/r/501709 https://gerrit.wikimedia.org/r/500111 https://gerrit.wikimedia.org/r/500106 https://gerrit.wikimedia.org/r/500127 https://gerrit.wikimedia.org/r/500119
* 00:53 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/501813


== 2019-04-05 ==
== 2021-01-13 ==
* 23:53 Krinkle: Beta cluster puppetmaster is stalled behind origin/production as of 24 hours ago (57 patches behind) due to a local merge conflict
* 15:26 hashar: Pruned Docker containers and images on all Docker based Jenkins agents
* 22:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501789
* 22:02 legoktm: rebuilding mediawiki-phan docker image https://gerrit.wikimedia.org/r/501794
* 20:11 thcipriani: updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/501465
* 08:56 hashar: Reloaded Zuul for operations/software/gerrit/plugins/barricade https://gerrit.wikimedia.org/r/#/c/integration/config/+/501507/
* 08:21 legoktm: deploying https://gerrit.wikimedia.org/r/501416


== 2019-04-04 ==
== 2021-01-12 ==
* 23:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501442
* 21:15 brennen: Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/640567
* 22:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/500360
* 20:13 James_F: Zuul: Remove Disambiguator from Parsoid dependencies (again)
* 21:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501428
* 20:06 James_F: Zuul: Add parsoid as a dependency of the Disambiguator extension
* 20:47 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/501407
* 20:03 James_F: Zuul: Allow parsoid to be added to dependency and gatedextensions lists
* 17:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/500719 (
* 19:33 James_F: Zuul: Revert Parsoid integration job injection.
* 02:35 James_F: Zuul: [mediawiki/vendor] Experimental composer-php80 job, not 72
* 01:11 James_F: Zuul: Ensure Parsoid's integration job tests against the Disambiguator extension [[phab:T237538|T237538]]
* 01:07 James_F: Zuul: [labs/tools/stewardbots] Enable PHP 8.0 jobs; drop special template


== 2019-04-03 ==
== 2021-01-11 ==
* 06:13 hashar: gerrit: renamed group "scholarships" to "wikimedia-wikimania-scholarships". Made it owned by "Gerrit Managers" # [[phab:T218864|T218864]]
* 08:59 hashar: gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # [[phab:T271683|T271683]]


== 2019-04-02 ==
== 2021-01-09 ==
* 22:24 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/PageAssessments/maintenance/purgeUnusedProjects.php --wiki=enwikivoyage {{!}} [[phab:T219935|T219935]]
* 04:30 James_F: Zuul: [mediawiki/libs/RemexHtml] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]
* 09:46 hashar: Upgrading CI Quibble jobs to 0.0.31
* 04:30 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.0 jobs, now passing [[phab:T271575|T271575]]


== 2019-04-01 ==
== 2021-01-08 ==
* 21:42 hauskatze: Imported tool-ldap from Diffusion to Gerrit with full history {{!}} [[phab:T219703|T219703]]
* 02:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/654964
* 21:33 hauskatze: Created https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/ldap {{!}} [[phab:T219703|T219703]]
* 20:53 hashar: ssh contint1001.wikimedia.org sudo rm /tmp/docker-pkg-build.log
* 20:44 hashar: Building Quibble 0.0.31 containers again # [[phab:T219647|T219647]]  [[phab:T219786|T219786]]
* 20:10 hashar: gerrit: flush-caches --cache git_tags  # some tag got stalled when querying over https -  [[phab:T219786|T219786]]
* 18:50 hauskatze: Created mediawiki/extensions/ContributionCredits.git per request on mediawiki.org
* 18:00 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/#/c/integration/config/+/500504/
* 17:25 Krinkle: fresnel-node10-browser-docker failing with ENOMEM. Depooled integration-slave-docker-1049 as precaution.
* 17:04 hashar: Building CI docker images for Quibble 0.0.31 (yes it is a long day...)
* 13:50 hashar: Reverted CI Jenkins jobs to Quibble 0.0.28 # [[phab:T219647|T219647]]
* 13:11 hashar: Upgraded CI Jenkins jobs to Quibble 0.0.30 # [[phab:T219647|T219647]]
* 10:50 hashar: Manually triggering postmerge step of citoid due to [[phab:T219017|T219017]] for mvolz. On contint1001: zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/citoid --change 497315,1
* 08:08 hashar: Rebuilding Quibble Jessie containers that failed to build last week due to wikimedia-jessie container. # [[phab:T219647|T219647]]
* 08:07 hashar: Rebuilding container docker-registry.wikimedia.org/wikimedia-jessie # [[phab:T219683|T219683]]


== 2019-03-29 ==
== 2021-01-07 ==
* 19:12 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/497975/ + https://gerrit.wikimedia.org/r/498887/
* 18:34 James_F: Zuul: Provide experimental PHP 7.2-on-buster jobs [[phab:T252434|T252434]]
* 17:30 hashar: some quibble 0.0.30 images fail to build for an unknown reason . Left to figure out after the week-end has passed..
* 18:32 James_F: Zuul: [labs/tools/massmailer] Add gate-and-submit-l10n jobs [[phab:T271426|T271426]]
* 17:22 hashar: Building Ci docker images for quibble 0.0.30 [[phab:T219647|T219647]] [[phab:T219645|T219645]]
* 16:01 hashar: Tag Quibble 0.0.46 @ {{Gerrit|df9e75329ab}} # [[phab:T225218|T225218]] [[phab:T266441|T266441]] [[phab:T263500|T263500]]
* 12:03 Krenair: added alaasarhan to deployment-prep [[phab:T219621|T219621]]


== 2019-03-28 ==
== 2021-01-06 ==
* 18:49 Krenair: shut off deployment-db04 instance per [[phab:T219087|T219087]]
* 19:36 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/620729 in beta puppet.
* 18:20 thcipriani: reload zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/
* 18:33 dpifke: Doing a test run in beta of the scap commands I'm going to run in today's backport window to roll out profiler changes.
* 17:36 paladox: forking plugins/quota from upstream
* 17:24 Krenair: deployment-prep [[phab:T219087|T219087]] beginning master switch
* 10:45 hashar: Tagged Quibble 0.0.30 {{Gerrit|6ddc6d508cb554e6443ff72648da3ea8a3253fff}}
* 08:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/499539 (no-op) and https://gerrit.wikimedia.org/r/499717 (SecurePoll phan)


== 2019-03-27 ==
== 2021-01-05 ==
* 22:27 James_F: Altered Wikimedia GitHub settings to require 2FA; see [[phab:T198810|T198810]]
* 23:55 dpifke: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/651267 and https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/621095/3 in beta.  Should be a no-op (removes hopefully unused code).
* 14:16 hashar: contint: refreshed all git caches manually from cumin:  cumin --force 'name:docker' 'find /srv/git -name '*.git' -type d -print -exec git -C {} fetch --prune \;'
* 14:16 hashar: contint: added repositories to the git caches https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499482/


== 2019-03-26 ==
== 2021-01-04 ==
* 23:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499344 (part 1, 2 and 3)
* 22:48 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.0 composer job [[phab:T269719|T269719]]
* 20:36 paladox: create gerrit repo operations/software/gerrit/plugins/WikimediaBlocks [[phab:T219300|T219300]]
* 22:30 hasharAway: IRC notifications from Jenkins / wmf-insecte disabled  for now due to [[phab:T271122|T271122]]
* 20:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499312
* 21:08 hasharAway: Change Jenkins IRC login to mw-jenkinsbot # [[phab:T271122|T271122]]
* 17:43 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499262
* 17:33 thcipriani: fixed beta-scap-eqiad by removing local mwdeploy user/group using vipw/vigr and chown -R mwdeploy:mwdeploy /srv/mediawiki for deployment-prep hosts
* 17:42 Krinkle: Reloading Zuul to deploy merged-but-not-deployed patches https://gerrit.wikimedia.org/r/#/c/integration/config/+/497800/ and https://gerrit.wikimedia.org/r/#/c/498684/


== 2019-03-25 ==
== 2021-01-02 ==
* 14:46 mateusbs17: sunset deployment-maps03
* 19:13 James_F: Zuul: Add CI for the Mirage skin [[phab:T270979|T270979]]


== 2019-03-24 ==
== 2021-01-01 ==
* 16:06 Krenair: shut off old deployment-db03 instance per [[phab:T219087|T219087]]
* 18:26 James_F: zuul: Try in a second way to only run mwext coverage jobs on master [[phab:T270976|T270976]]
* 05:39 Krenair: cleaned up old puppet certs/nodes -certcentral-testclient03 -certcentral-testdns -certcentral03 -zotero01 -eventgate-analytics -t153468-test -rd3-cptest-master01 -maps05
* 18:13 James_F: zuul: [mediawiki/extensions/AbuseFilter] Make sqlite tests voting [[phab:T251967|T251967]]
* 04:12 Krenair: removed php7.0-fpm package (conflicting with php7.2-fpm) and removed /etc/nginx/sites-enabled/default (conflicting with apache, puppet will remove the available copy too) from -deploy02, -jobrunner03, -mwmaint01, and -mediawiki-07 hosts to try to get puppet there happy again
 
== 2019-03-23 ==
* 23:44 legoktm: deploying https://gerrit.wikimedia.org/r/498681 https://gerrit.wikimedia.org/r/498682 https://gerrit.wikimedia.org/r/498683
* 23:33 legoktm: deploying https://gerrit.wikimedia.org/r/498247 https://gerrit.wikimedia.org/r/498249 https://gerrit.wikimedia.org/r/498680
* 20:52 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/498664 / [[phab:T215562|T215562]]
 
== 2019-03-22 ==
* 19:59 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/498466/ # [[phab:T219017|T219017]]
* 19:34 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/494548/
* 19:24 thcipriani: clean old docker-pkg image from contint1001
* 03:08 legoktm: rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T218963|T218963]]
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T215562|T215562]])
 
== 2019-03-21 ==
* 23:34 legoktm: rebuilding mediawiki-phan docker image for https://gerrit.wikimedia.org/r/498266
* 22:34 legoktm: onlined integration-slave-jessie-1002
* 22:33 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 21:03 legoktm: deploying https://gerrit.wikimedia.org/r/498178 https://gerrit.wikimedia.org/r/498185 https://gerrit.wikimedia.org/r/498187 https://gerrit.wikimedia.org/r/498189 https://gerrit.wikimedia.org/r/498182
* 15:22 <hashar>: pruning images/containers on integration-slave-docker-1021
* 10:13 <hashar>: deployment-deploy01: sudo rm -fR /tmp/mw-cache-master | files were from Mar 15 10:21
* 00:33 <legoktm>: deploying https://gerrit.wikimedia.org/r/497600
 
== 2019-03-20 ==
* 13:20 <hashar>: Scheduled update of Diffusion repository wikibase-termbox via https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ on request of Pablo_WMDE
 
== 2019-03-19 ==
* 22:47 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497322 / T218553
* 21:00 <paladox>: hiding operations/software/gerrit/plugins/WikimediaWebSessions from normal users
* 20:59 <paladox>: create operations/software/gerrit/plugins/WikimediaWebSessions project T218739
* 20:23 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/preferences.d/jessie_mitaka_pinning_*' # T218559
* 19:56 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list' # T218559
* 16:56 <Reedy>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/494802
* 15:48 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497437
* 14:01 <hashar>: Removed Zuul "check" pipeline | https://gerrit.wikimedia.org/r/#/c/integration/config/+/493188/ | T192217
* 02:21 <Krenair>: sudo aptly publish --architectures="all,amd64" --skip-signing repo buster-deployment-prep
* 02:09 <Krenair>: sudo aptly repo create -component="main" -distribution="buster-deployment-prep" buster-deployment-prep
 
== 2019-03-18 ==
* 16:39 Krenair: yet*
* 16:38 Krenair: created deployment-acme-chief01 and a client instance for further acme-chief testing + dev. used stretch, would be buster like prod but not sure that's easily available outside testlabs yes
* 10:09 hashar: contint1001: rm -fR /srv/doc1001.eqiad.wmnet
* 10:06 hashar: github: deleting https://github.com/wikimedia/wikidata-gremlin # archived [[phab:T155829|T155829]]
* 09:58 hashar: arming keyholder on integration-cumin
* 09:55 hashar: deleting shutdowned instance integration-publisher02 , we do not use it anymore since doc publishing got overhauled ( [[phab:T137890|T137890]] )  # [[phab:T218146|T218146]]
* 09:12 hashar: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
* 07:37 legoktm: deployed https://gerrit.wikimedia.org/r/496610
 
== 2019-03-17 ==
* 16:29 Krenair: deactivated and cleaned certs for deployment-redis3-changeprop02 and deployment-prometheus01 (which no longer appear to exist but were causing cumin to be upset)
* 16:27 Krenair: armed deployment-cumin keyholder
* 00:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496887
 
== 2019-03-16 ==
* 21:56 Krinkle: krinkle@contint1001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/OOJsUIAjaxLogin --change 490979,1
* 21:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497077
* 21:41 legoktm: deploying https://gerrit.wikimedia.org/r/496507 https://gerrit.wikimedia.org/r/496524 https://gerrit.wikimedia.org/r/496520 https://gerrit.wikimedia.org/r/496516 https://gerrit.wikimedia.org/r/496510
* 21:21 Krinkle: krinkle@contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/EventLogging --change 264494,4
* 21:20 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/497076
* 21:20 Krinkle: Removing doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-extensions-EventLogging (created by accident)
 
== 2019-03-15 ==
* 21:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496978
* 21:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496882
* 20:10 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/496688
* 18:21 thcipriani: clean old docker images from contint1001
* 17:40 thcipriani: rearm beta keyholder
* 16:54 thcipriani: reenable beta-scap-eqiad
* 16:08 thcipriani: disable beta-scap-eqiad to test new php, back shortly
 
== 2019-03-14 ==
* 23:48 Krinkle: Abort job quibble-vendor-mysql-hhvm-docker/39874/ for mwext-CentralAuth (stuck after 59 minutes)
* 21:25 hashar: Manually triggered tests for 12 ContentTranslation changes that had label:verified=-1 # [[phab:T216689|T216689]]
* 21:21 hashar: Updated quibble-vendor-mysql-hhvm-docker with latest libc6 hopefully fixing HHVM segfault within libpthread # [[phab:T216689|T216689]]
* 19:14 ebernhardson: restart logstash on deployment-logstash2 to re-read and re-create apifeatureusage template
* 16:34 hashar: rollback quibble-vendor-mysql-hhvm-docker job to no more capture core files, we have enough and a good lead ( reverting https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ ) # [[phab:T216689|T216689]]
* 12:31 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/4 # [[phab:T216689|T216689]]
* 12:12 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/3 # [[phab:T216689|T216689]]
* 10:49 hashar: triggering tests for all ContentTranslation pending changes # [[phab:T216689|T216689]]
* 09:54 hashar: ci: live hacked job https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/ in attempt to capture 'core' files from hhvm {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ {{!}} [[phab:T216689|T216689]]
 
== 2019-03-13 ==
* 21:26 thcipriani: pool new bigram CI instance integration-slave-docker-1054
* 20:58 thcipriani: deleting bigram CI instance integration-slave-docker-1046 due to corrupt disk cf: [[phab:T218245|T218245]]
* 20:44 thcipriani: marking integration-slave-docker-1046 offline
* 20:38 hashar: Added integration-slave-docker-1045 to Jenkins. The instance existed in WMCS but was not in Jenkins
* 20:13 bearND: (beta): Update mobileapps to {{Gerrit|5865552}} ({{Gerrit|7074964}} {{Gerrit|d6dc3cd}} {{Gerrit|fbc6262}})
* 19:13 hashar: integration-slave-docker-1046 is back online # [[phab:T218245|T218245]]
* 19:04 hashar: hard rebooting integration-slave-docker-1046 , not reachable over ssh # [[phab:T218245|T218245]]
* 18:48 hashar: Building containers releng/quibble-jessie-hhvm and releng/quibble-stretch-hhvm with HHVM core_dump_report enabled # [[phab:T216689|T216689]]
* 18:09 ebernhardson: restart elasticsearch on deployment-elastic* to deploy apifeature usage fix ([[phab:T183156|T183156]])
* 10:39 hashar: Triggered tests for ContentTranslation changes that had label:verified=-1 # [[phab:T217654|T217654]]
* 10:03 hashar: Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # [[phab:T217654|T217654]]
 
== 2019-03-12 ==
* 18:38 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/479558  / [[phab:T203506|T203506]]
* 18:23 thcipriani: bring integration-slave-jessie-1001 back online, /srv disk space now at 20% (not sure if someone cleared disk and forgot to repool)
* 16:45 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/495931
* 13:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/493139/
 
== 2019-03-11 ==
* 15:59 Krenair: previous !log was for deployment-prep
* 15:56 Krenair: added MSantos to projectadmin, per chat in -infrastructure he'll use it to work on maps. user is a new foundation software engineer
* 01:20 legoktm: deploying https://gerrit.wikimedia.org/r/495593
 
== 2019-03-10 ==
* 02:50 legoktm: deploying https://gerrit.wikimedia.org/r/495443 https://gerrit.wikimedia.org/r/493741
* 02:45 legoktm: deploying https://gerrit.wikimedia.org/r/495332 https://gerrit.wikimedia.org/r/495470
 
== 2019-03-08 ==
* 22:18 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490950/
* 20:20 legoktm: deploying https://gerrit.wikimedia.org/r/495302
* 20:16 legoktm: deploying https://gerrit.wikimedia.org/r/495300 https://gerrit.wikimedia.org/r/495298 https://gerrit.wikimedia.org/r/493502
* 11:36 hasharLunch: integration: deleting old java8 docker images:  sudo cumin --force 'name:docker' 'docker images{{!}}grep java8{{!}}awk "{ print $3 }"{{!}}xargs docker rmi'
* 10:04 hashar: Updating maven based jobs to latest java8 containers {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/495022/ {{!}} [[phab:T208938|T208938]]
 
== 2019-03-06 ==
* 18:25 hasharAway: contint1001: restart Jenkins for plugins upgrade
* 18:24 hasharAway: reloading zuul for {{Gerrit|I21d7fa2939f507441f7d130f8207541e28d762ad}}
* 18:02 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org
* 17:37 hashar: Reloading Zuul for  mediawiki/libs/Zest https://gerrit.wikimedia.org/r/#/c/integration/config/+/494609/2/zuul/layout.yaml
* 16:27 Krinkle: doc1001 had bad permissions set on /org/wikimedia/doc (chmod 755 instead of 775). Making it impossible to git pull in the way that post-merge Jenkins job on integration/docroot recommends. Fixed with `sudo -u doc-uploader chmod 775 /srv/wikimedia/org/wikimedia/doc`.
* 15:58 andrewbogott: deleting deployment-prometheus01 on Filippo's advice
* 13:31 gehel: upgrading logstash to 5.6.14 on deployment-logstash2
* 13:15 gehel: upgrading elasticsearch to 5.6.14 on deployment-logstash2
* 11:33 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mediawiki-09:~$ sudo systemctl restart php7.2-fpm # [[phab:T217323|T217323]]
 
== 2019-03-05 ==
* 20:10 thcipriani: reenable beta-scap-eqiad
* 19:17 thcipriani: disable beta-scap-eqiad due to [[phab:T217587|T217587]]
* 10:13 hashar: integration: fixed erroneous ssh key restriction for cumin {{!}} [[phab:T217642|T217642]]
 
== 2019-03-04 ==
* 17:36 hashar: Build Docker containers for chromium=v71 pin ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/494243/ )
* 15:14 herron: cherry picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/492390/ to deployment-puppetmaster03
* 13:49 hauskatze: GitHub: deleted wikimedia/mediawiki-extensions-UploadLocal {{!}} [[phab:T213011|T213011]]
* 09:59 hashar: cleaned docker images on integration-slave-docker-1037
* 07:38 legoktm: deploying https://gerrit.wikimedia.org/r/494167
 
== 2019-03-03 ==
* 02:26 Krinkle: tried rebooting or shutting down integration-slave-docker-1021, no response on horizon. Did pause/resume instead, which did work, after which shutdown/start worked. Jenkins agent has been relaunched and seems online again.
* 02:20 Krinkle: integration-slave-docker-1021 (ci1.medium) has jobs failing on it due to ENOMEM. Horizon shows in log: integration-slave-docker-1021 login: [4961938.696837] Out of memory: Kill process 21770 (chromium) score 841 or sacrifice child; [4961938.699176] Killed process 21770 (chromium) total-vm:3171496kB, anon-rss:1379288kB, file-rss:0kB, shmem-rss:1636kB
 
== 2019-03-02 ==
* 22:12 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/493959
* 20:44 hauskatze: Renamed https://github.com/wikimedia/wikimedia-github-community-health-defaults to https://github.com/wikimedia/.github
* 20:42 hauskatze: ssh -p 29418 gerrit.wikimedia.org replication start wikimedia/github-community-health-defaults --wait
* 20:40 hauskatze: github created https://github.com/wikimedia/wikimedia-github-community-health-defaults
* 20:31 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/493881
* 20:30 Krinkle: Failure on integration-slave-docker-1021 (ENOMEM) https://integration.wikimedia.org/ci/job/fresnel-node10-browser-docker/61/console
* 19:51 legoktm: deploying https://gerrit.wikimedia.org/r/493872
* 19:37 legoktm: deploying https://gerrit.wikimedia.org/r/493862
* 18:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493808
* 18:21 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493837
 
== 2019-03-01 ==
* 19:17 thcipriani: integration-slave-docker-1021:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '{print $3}')
* 17:02 thcipriani: integration-slave-jessie-1004 back online
* 16:58 thcipriani: integration-slave-jessie-1002 back online (disk space looked fine); rebooting integration-slave-jessie-1004 -- can't ssh to machine
* 16:11 Lucas_WMDE: delete refs/master and refs/gerrit/master on WikibaseQualityConstraints repository [[phab:T217408|T217408]]
* 15:49 hashar: wikidata/query/blazegraph  change Gerrit config to require a change-id # [[phab:T216855|T216855]]
* 14:28 hashar: Upgrading integration/jenkins-job-builder to version 2.0.2  + one custom hack 11aa5de4...a06d173e  # [[phab:T143731|T143731]]
* 14:18 hashar: integration/jenkins-job-builder : importing upstream code to new branch "upstream".  Push all upstream tags to our repository
 
== 2019-02-28 ==
* 18:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493261
* 03:42 duh: deploying https://gerrit.wikimedia.org/r/493357
* 03:42 duh: deploying
* 03:30 duh: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/493355
 
== 2019-02-27 ==
* 22:02 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/
* 19:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/492758
* 19:08 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758 (take II)
* 18:27 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758
 
== 2019-02-26 ==
* 22:52 ebernhardson: delete logstash logs in /var/log/logstash generated prior to 2019
* 22:51 ebernhardson: restart logstash on deployment-logstash2 while hacking around to see why apifeatureusage doesn't work
* 18:01 dcausse: deployement-prep: installing elastic 6.5.4 to deployment-elastic* machines
* 16:46 addshore: added Cparle to deployment-prep
* 16:11 hashar: Generating 1.33.0-wmf.19 deploy notes https://integration.wikimedia.org/ci/job/train-deploy-notes/9/console {{!}} [[phab:T206673|T206673]]
* 09:32 godog: remove now-merged node-exporter timer disable, cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/492632
 
== 2019-02-25 ==
* 23:32 twentyafterfour: root@deployment-db05# mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata # [[phab:T216067|T216067]]
* 22:14 thcipriani: docker rmi images without "latest" tag on contint1001 to free space -- should have kept all current docker-pkg images as well as images with children -- [[phab:T217094|T217094]]
* 13:39 hashar: Rebuilding some CI Docker images using PHP sury.org to switch the sury.org component from jessie to stretch ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/492666/ )
 
== 2019-02-24 ==
* 21:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/492561, [[phab:T216964|T216964]]
* 04:54 legoktm: rebuilding docker image for https://gerrit.wikimedia.org/r/485241
* 04:41 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/tools$ sudo -u zuul rm -rf phan
 
== 2019-02-23 ==
* 22:25 legoktm: deploying https://gerrit.wikimedia.org/r/492497
* 02:32 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/492427
* 01:59 Krinkle: Reloading Zuul to deploy ttps://gerrit.wikimedia.org/r/492425
 
== 2019-02-22 ==
* 20:59 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/492377
* 16:48 paladox: created "block-users" group for https://bugs.chromium.org/p/gerrit/issues/detail?id=10507 (reported by a wmf user)
* 13:38 Reedy: restarting zuul because nothing is being run
 
== 2019-02-21 ==
* 21:02 paladox: branch deploy/wmf/stable-2.16 from {{Gerrit|0e0ea0ff735da0a494347884917fc48881d7e545}} in operations/software/gerrit
* 19:49 Amir1: ores:5d937b1 is going beta
* 18:22 Amir1: ores:2d84709 going beta
* 17:45 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/483225/
* 16:35 hashar: Adjusting IP of https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ and adding it back # [[phab:T216513|T216513]]
* 14:10 Amir1: deploying ores:5d50713 to beta
 
== 2019-02-20 ==
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status  # [[phab:T216635|T216635]]
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status
* 17:26 hashar: For beta cluster the MySQL master database has some innodb issue [[phab:T216635|T216635]] , the MySQL slave has an issue as well [[phab:T216067|T216067]]
* 17:09 hashar: reloading zuul for {{Gerrit|Id1e3afd0afba9b388778066b9b6e8e564a25826b}}
* 17:09 hashar: contint1001: fix broken root ownership on zuul git deploy repo: sudo find /etc/zuul/wikimedia/.git -not -user zuul -exec chown zuul:zuul {} +
* 17:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490606/
* 16:57 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490559
* 16:36 hauskatze: Ran replication start mediawiki/extensions/PageViewInfo --wait on gerrit.wikimedia to populate GitHub mirror (success messages afterwards) {{!}} [[phab:T180864|T180864]]
* 00:25 greg-g: disabled beta-update-databases-eqiad in the jenkins UI - [[phab:T216067|T216067]]
 
== 2019-02-19 ==
* 21:23 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/491577, also deploys  https://gerrit.wikimedia.org/r/490492  / [[phab:T198495|T198495]]
* 20:28 hashar: jenkins: disable Android Emulator plugin {{!}} [[phab:T198495|T198495]]
* 20:27 hashar: gerrit: archived repository integration/jenkinsci/android-emulator-plugin {{!}} [[phab:T198495|T198495]]
* 20:23 hashar: Deleting Jenkins job apps-android-wikipedia-periodic-test {{!}} [[phab:T198495|T198495]]
* 15:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491513
* 15:46 hashar: Creating integration-slave-jessie-android m1.large it got deleted last week {{!}} [[phab:T216517|T216517]]
* 08:55 hashar: Cleaning contint1001 / partition
* 04:55 bd808: Removed stale cherry-pick of {{Gerrit|Ic7e726768701fefdee68622b08e3f2995779fe5a}} from deployment-puppetmaster03 that was blocking rebase on origin/production
* 04:50 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491405
* 04:14 Krinkle: Disabled spam account on Phab - https://phabricator.wikimedia.org/people/manage/18915/
* 03:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491403
* 00:54 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491381  / {{Gerrit|bc8e4198961cb73}} /  [[phab:T133646|T133646]]
 
== 2019-02-18 ==
* 22:32 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491379  / [[phab:T211784|T211784]]
* 21:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/491378 / [[phab:T133646|T133646]])
* 21:37 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/459268 / [[phab:T133646|T133646]]
* 15:38 elukey: kill/spawn deployment-aqs0[2,3] in deployment-prep with Debian Stretch
* 14:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490590
* 12:00 Krenair: [[phab:T216067|T216067]] Stopping mysql on -db04 to begin copy to -db05. Note crashed tables centralauth.globaluser and centralauth.localuser
* 11:57 elukey: kill/spawn deployment-aqs01 with Debian Stretch in deployment-prep
* 11:45 arturo: manually start deployment-db03 per Krenair request
* 11:29 hasharAway: beta: tried to start instance deployment-db03 172.16.5.23 --> ERROR  {{!}} [[phab:T216067|T216067]]
 
== 2019-02-17 ==
* 07:21 legoktm: deploying https://gerrit.wikimedia.org/r/491029
* 07:10 legoktm: Building image docker-registry.discovery.wmnet/releng/tox-acme-chief:0.3.4
* 06:28 legoktm: building new tox-acme-chief docker image https://gerrit.wikimedia.org/r/489725
* 01:12 Krinkle: beta-scap-eqiad (cron) failing with "sudo: a password is required"
 
== 2019-02-16 ==
* 19:44 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490937 / [[phab:T216275|T216275]])
* 17:23 thcipriani: installed php7.0-curl on deployment-deploy01 (why was that suddenly necessary?)
 
== 2019-02-15 ==
* 17:28 thcipriani: integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
 
== 2019-02-14 ==
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T216102|T216102]]
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T133646|T133646]]
* 13:24 thcipriani: rearm keyholder on deployment-deploy01: sudo keyholder arm, passwords on https://wikitech.wikimedia.org/wiki/Keyholder
 
== 2019-02-13 ==
* 21:32 marxarelli: dduvall@integration-slave-jessie-1001:/mnt/home/jenkins-deploy$ `rm -rf .gradle/ .m2/` due to full disk
* 21:21 marxarelli: bringing integration-slave-docker-1046 and integration-slave-jessie-1001 back online
* 21:20 marxarelli: dduvall@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
* 21:15 marxarelli: removing old docker images on integration-slave-docker-1046
* 21:10 marxarelli: starting migrated integration-slave-docker-1046 instance
* 21:01 marxarelli: pooling new jenkins node for integration-slave-docker-1052
* 20:46 marxarelli: pooling jenkins node for integration-slave-docker-1051
* 20:45 marxarelli: launching replacement instance integration-slave-docker-1052
* 20:35 marxarelli: launching replacement instance integration-slave-docker-1051
* 20:32 marxarelli: pooling jenkins node for integration-slave-docker-1050
* {{safesubst:SAL entry|1=20:15 marxarelli: integration-slave-docker-{1044,1046,1047} unresponsiveness due to cloudvirt failure. 1046 is being moved already by CS. deleting 1044 and 1047}}
* {{safesubst:SAL entry|1=19:57 marxarelli: seeing jenkins agent connection failures for integration-slave-docker-{1044,1046,1047}}}
* 19:48 marxarelli: pooling replacement jenkins node integration-slave-docker-1049
* 19:34 marxarelli: deleting integration-slave-jessie-android jenkins node and instance
* 19:33 marxarelli: deleting integration-slave-jessie-1003 jenkins node and instance
* 19:32 marxarelli: deleting integration-slave-docker-1033 jenkins node and instance
* 19:25 marxarelli: deleting integration-slave-docker-1017 jenkins node and instance
* 18:45 Krinkle: integration-slave-jessie-1003 seems to be consitently unable to start jobs, marking as offline manually
* 18:32 thcipriani: bringing up new integration-castor03, re-enabling castor-save* jobs
* 18:15 marxarelli: adding new jenkins node integration-slave-docker-1048
* 18:02 marxarelli: launching new integration-slave-docker-1048 instance
* 17:59 marxarelli: deleting integration-slave-docker-1038 node and deleting instance
* 17:50 marxarelli: bringing integration-slave-docker-1033 back online after clearing out old docker images
* 17:33 thcipriani: rebuilding integration-castor03
* 17:21 thcipriani: stopping rsync server on castor03
* 17:21 twentyafterfour: stopped rsync on castor03
* 17:16 twentyafterfour: disconnected castor03 from jenkins
* 16:48 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/487880/
* 14:34 thcipriani: modified castor-save-workspace-cache to exit 0 and run on blubber nodes while integration-castor03 is down
* 14:26 dcausse: deployement-prep: upgrading to elastic 5.6.14
 
== 2019-02-12 ==
* 23:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490130/1
 
== 2019-02-11 ==
* 21:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/489689
* 20:39 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/489688/
* 19:46 marxarelli: installing/enabling HTTP Request jenkins plugin on integration.wikimedia.org/ci to support https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/480689 changes
* 16:04 addshore: bring integration-slave-docker-1040 back online
* 16:04 addshore: addshore@integration-slave-docker-1040:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.52GB
* 10:47 godog: shut deployment-prometheus01, unused now
* 03:06 Reedy: graceful restart of zuul as no jobs were running
 
== 2019-02-10 ==
* 03:22 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489434  (Create quibble-stretch-hhvm, replacing jessie)
* 02:06 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489430
 
== 2019-02-08 ==
* 20:20 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config, ref [[phab:T91410|T91410]]
* 15:59 addshore: this reload also included "Switch npm-audit job to node10"? [[phab:T211784|T211784]], which did touch the zuul file
* 15:58 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/489241/
* 03:10 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config
* 00:28 Krinkle: krinkle@doc1001: sudo -u doc-uploader chmod 775 /srv/docroot/org/wikimedia/doc/
* 00:12 marxarelli: removed old docker images on contint1001 to free up space
 
== 2019-02-07 ==
* 23:17 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
* 23:15 thcipriani: integration-slave-docker-1033:sudo docker image prune and bring back online
* 22:28 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/467550 (
* 19:09 paladox: created integration/zuul/build gerrit repo for [[phab:T215458|T215458]]
* 19:05 paladox: created integration/zuul/wheels gerrit repo for [[phab:T215458|T215458]]
* 15:48 addshore: brought integration-slave-docker-1043 back online
* 15:48 addshore: addshore@integration-slave-docker-1043:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 14.86GB
* 08:49 hashar: cleaning docker images on integration-slave-docker-1021
 
== 2019-02-06 ==
* 22:34 shdubsh: Deploy node-exporter 0.17 [[phab:T213708|T213708]]
* 14:12 godog: shut off deployment-prometheus01 - [[phab:T215272|T215272]]
* 14:00 godog: switch beta-prometheus to deployment-prometheus02 - [[phab:T215272|T215272]]
 
== 2019-02-05 ==
* 20:07 ebernhardson: jobrunner port 9006 is firewalled, revert to 9005 and created [[phab:T215339|T215339]] to fix job queue in beta cluste
* 19:36 ebernhardson: Update profile::cpjobqueue::{jobrunner,videoscaler}_host in horizon hiera from port 9005 to 9006 to match new restrictions in gerrit.wikimedia.org/r/481866
* 16:29 addshore: [[phab:T215288|T215288]] added mirrys to deployment-prep as a user
* 15:32 addshore: [[phab:T215278|T215278]] addshore@integration-slave-docker-1037:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.59GB
 
== 2019-02-04 ==
* 23:13 thcipriani: integration-slave-docker-1040:sudo docker image prune and bring back online
* 23:12 thcipriani: integration-slave-docker-1038:sudo docker image prune and bring back online
* 21:48 ebernhardson: restart logstash on deployment-logstash2
* 15:25 hashar: removed Jenkins user "nodepoolmanager" as well as related authorizations {{!}} [[phab:T209361|T209361]]
 
== 2019-02-03 ==
* 06:15 legoktm: deployed https://gerrit.wikimedia.org/r/487627
* 05:44 legoktm: deployed https://gerrit.wikimedia.org/r/485967
* 04:36 legoktm: deploying https://gerrit.wikimedia.org/r/487534
 
== 2019-02-02 ==
* 22:17 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
 
== 2019-01-31 ==
* 15:03 thcipriani: rearm keyholder on deployment-deploy01
* 12:05 arturo: VM instances deployment-deploy01,deployment-deploy02,deployment-fluorine02,deployment-kafka-jumbo-2,deployment-kafka-main-1,deployment-maps04,deployment-mcs01,deployment-mediawiki-09,deployment-memc04,deployment-ms-be03,deployment-ms-fe02,deployment-parsoid09,deployment-sca04,deployment-webperf12, were stopped briefly due to issue in hypervisor ([[phab:T215012|T215012]])
 
== 2019-01-30 ==
* 08:35 legoktm: deploying https://gerrit.wikimedia.org/r/486439 https://gerrit.wikimedia.org/r/481570 https://gerrit.wikimedia.org/r/481571
 
== 2019-01-29 ==
* 07:41 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * b/c full disk
 
== 2019-01-28 ==
* 16:33 hashar: contint1001: cleaning up disk space on /
* 13:07 addshore: bringing integration-slave-docker-1041 back online
* 13:07 addshore: addshore@integration-slave-docker-1041:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.12GB
* 09:37 Amir1: ores:ad160b0 is going beta
 
== 2019-01-27 ==
* 19:57 addshore: bringing integration-slave-docker-1034 back online
* 19:50 addshore: addshore@integration-slave-docker-1034:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.12GB
 
== 2019-01-26 ==
* 22:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486791
* 21:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486737
 
== 2019-01-25 ==
* 19:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486503/
* 19:07 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486501/
* 14:56 hashar: contint1001: systemctl stop zuul-merger && find /srv/zuul/git -name .git -type d -print -execdir git gc --prune=now \;
* 13:35 hashar: flake8 broken under python2.7 due to configparser==3.5.2  https://github.com/jaraco/configparser/issues/27 https://github.com/jaraco/configparser/issues/27
* 00:58 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* clean gradle cache, bring back online
 
== 2019-01-23 ==
* 22:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/486170/
 
== 2019-01-22 ==
* 23:46 thcipriani: Updating (more) tox docker image on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485971/
* 23:24 thcipriani: integration-slave-docker-1017 clean docker images and repool
* 21:06 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485895/
* 20:18 legoktm: rebuilding php, composer, phan images for apt security update https://gerrit.wikimedia.org/r/485883
* 19:59 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485882/
* 19:28 legoktm: rebuilding ci support images for apt security upgrade https://gerrit.wikimedia.org/r/485874
* 19:09 legoktm: rebuilding ci-stretch/ci-jessie images https://gerrit.wikimedia.org/r/485873
* 19:08 legoktm: manually docker pulled docker-registry.wikimedia.org/wikimedia-jessie and docker-registry.wikimedia.org/wikimedia-stretch on contint1001
* 18:56 bearND: (beta) Update mobileapps to {{Gerrit|0aac268}} (fix pronunciation detection in mobile-sections [[phab:T214338|T214338]])
* 18:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/485870/ and https://gerrit.wikimedia.org/r/#/c/integration/config/+/485331/
* 14:31 addshore: reloaded zuul for REVERT: Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:26 addshore: reloaded zuul for Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:03 addshore: reload zuul for Add ArticlePlaceholder to Wikibase Tests [integration/config] - https://gerrit.wikimedia.org/r/485753
 
== 2019-01-21 ==
* 19:49 hashar: integration: update sudo rule for debian-glue to keep env variable EXTRAPACKAGES. Would let us get eatmydata included {{!}} [[phab:T214328|T214328]]
* 15:40 hashar: contint1001: removing all generated doc/cover from /srv/org/wikimedia/doc {{!}} [[phab:T137890|T137890]]
 
== 2019-01-18 ==
* 23:22 hashar: contint1001: sudo docker image prune  # Total reclaimed space: 3.592GB
* 23:00 Krinkle: Some docker builds on integration-slave-docker-1021 failing with ENOMEM
* 23:00 mutante: contint1001 - gzipping more files in /var/log/zuul/
* 22:57 mutante: contint1001 - moved zuul logs from 2018 and gzipped zuul logs from /var/log/zuul to /srv/logs/zuul to free disk space on /
* 22:39 mutante: contint1001 - apt-get clean - disk space low
* 22:31 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482527  / [[phab:T212602|T212602]]
 
== 2019-01-17 ==
* 19:34 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 08:39 legoktm: deploying composer docker image - https://gerrit.wikimedia.org/r/484853
 
== 2019-01-16 ==
* 21:11 bearND: (beta): Update mobileapps to {{Gerrit|258d76b}} page summary changes
 
== 2019-01-15 ==
* 09:00 hashar: Deleting Docker images on integration-slave-docker-1021
 
== 2019-01-14 ==
* 22:02 bearND: (beta): Update mobileapps to {{Gerrit|f2658de}}
* 21:47 mutante: deployment-mcs01 - sudo su deploy-service; cd /srv/deployment/mobileapps/deploy-cache/revs/1182b3b8f288df0221257b929ca43fb86862c2f8/scap ; touch log  (for debugging permission problem reported by bearND)
* 14:31 hashar: Nuked Castor cache for all *tox* jobs. Some might have cached binary wheels compiled against a lib that is no more existing (eg libmysqlclient.so.18 for mysql-python). Follow up the jessie -> stretch upgrade # [[phab:T191764|T191764]]
* 14:28 hashar: Deleted Castor cache for wikimedia-cz/tracker  mysql-python got cached as  a wheel but compiled against libmysqlclient.so.18. That fails with the new tox...:0.3.0 containers which uses mariadb / libmysqlclient.so compat symlink
 
== 2019-01-11 ==
* 20:48 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
* 19:29 legoktm: deploying https://gerrit.wikimedia.org/r/483832
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/483824
* 04:37 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/483674
* 03:48 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483671
* 03:15 twentyafterfour: mediawiki-1.32.0 uploaded to releases.wikimedia.org (archives should be downloadable in a few minutes after puppet runs and the proxy's cache expires)
* 03:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483667 Switch unicodejs jobs to Node 10  - [[phab:T211784|T211784]]
* 02:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483666
* 02:01 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483656
* 01:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483644
* 00:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482755 - [[phab:T211784|T211784]]
 
== 2019-01-09 ==
* 19:59 legoktm: deploying https://gerrit.wikimedia.org/r/482766
* 18:05 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/483189 https://gerrit.wikimedia.org/r/#/c/integration/config/+/481947/
* 13:27 hasharAway: shutting down instance integration-publishing02 [172.16.4.5]  No more used {{!}} [[phab:T137890|T137890]]
* 13:26 hasharAway: deleting https://integration.wikimedia.org/ci/job/publish-on-contint1001/ no more used {{!}} [[phab:T137890|T137890]]
 
== 2019-01-08 ==
* 21:52 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/476600/
* 21:31 Hauskatze: github: @niedzielski updated @jdlrobson permission on Wikimedia from `read` to `admin`
* 21:30 Hauskatze: github:
* 20:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/482855/
* 19:53 mutante: deployment-prep adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 19:53 mutante: adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 14:25 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org/
* 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T 101131
* 05:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482752
* 02:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482751
 
== 2019-01-07 ==
* 20:29 Krinkle: Reloading Zuul to apply https://gerrit.wikimedia.org/r/482695
* 20:12 Krinkle: Delete job 'fresnel-npm6-browser-node-6-docker' and 'fresnel-node-10-docker'
* 20:08 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/482572
* 19:32 awight: [[phab:T212530|T212530]]: ORES revscoring 2.3.0
* 19:22 legoktm: building node10 docker images w/ https://gerrit.wikimedia.org/r/482573 this time
* 00:30 legoktm: deploying https://gerrit.wikimedia.org/r/482392 / create node10 mage
 
== 2019-01-06 ==
* 00:39 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482392
 
== 2019-01-03 ==
* 20:15 Hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/SparkPost
* 20:03 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-SparkPost
* 18:46 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests, part 2
* 18:32 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests
* 16:44 thcipriani: remove old docker images and bring integration-slave-docker-1021 back online