You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(James_F: Zuul: [PropertySuggester] Add phan dependency T224774)
imported>Stashbot
(mutante: contint* - PHP upgraded to version 7.4)
 
(926 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== 2019-09-20 ==
== 2023-06-05 ==
* 01:00 James_F: Zuul: [PropertySuggester] Add phan dependency [[phab:T224774|T224774]]
* 23:09 mutante: contint* - PHP upgraded to version 7.4
* 23:02 James_F: Zuul: [integration/docroot] Drop PHP 7.3 CI, box now runs 7.4
* 22:22 James_F: Dockerfiles: [maven-java8] Update gerrit.wikimedia.org IP  for [[phab:T326368|T326368]]
* 12:41 hashar: Reloaded Zuul to enable SonarQube codehealth on WikimediaMessages, WikimediaIncubator, WikimediaEvents and WikimediaMaintenance # [[phab:T321837|T321837]]
* 09:15 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/927122 # [[phab:T337749|T337749]]


== 2019-09-19 ==
== 2023-06-02 ==
* 16:52 hashar: Building releng/helm-linter:0.2.0 [[phab:T233291|T233291]]
* 21:44 mutante: based on [[phab:T336427|T336427]]#8890714 pending a response, everything already copied to gerrit1003, and extra paths being added to Bacule...  shell access to gerrit1001 will be removed next week
* 09:31 hashar: Upgrading Jenkins plugins on CI and release hosts
* 18:16 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/926593


== 2019-09-18 ==
== 2023-06-01 ==
* 19:22 hashar: Keeping https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/5393/ forever for [[phab:T233143|T233143]]
* 13:48 James_F: doc: `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/GWToolset/` for [[phab:T337062|T337062]]
* 18:54 James_F: Zuul: Drop extension-seccheck-non-voting and skin-seccheck-non-voting, unused
* 13:34 James_F: Zuul: [wikidata/query/flink-rdf-streaming-updater] Archive
* 18:47 James_F: [[phab:T231928|T231928]] terminating endlessly-running attempts to build kartotherian and tilerator
* 13:18 Daimona: Dropping campaign_events.event_tracking_tool_id and campaign_events.event_tracking_tool_event_id on beta wikishared # [[phab:T336362|T336362]]
* 17:19 James_F: Zuul: [[phab:T231928|T231928]] Migrate kartotherian to custom pipeline jobs
* 07:43 hashar: Raised integration-castor03 egress trafic shaping from 50mbps to 100mbps # [[phab:T213223|T213223]]
* 07:36 hashar: Raised integration-castor03 egress trafic shaping from 50mbps to 100mbps # [[phab:T232644|T232644]]


== 2019-09-17 ==
== 2023-05-26 ==
* 20:52 brennen: Updating dev-images docker-pkg files on contint1001 for new restbase image
* 15:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/923647
* 18:32 James_F: jjb: Updated the rest of the quibble jobs to 0.0.35.
* 18:14 James_F: jjb: Upgrading wmf-quibble-vendor-mysql-php72-docker
* 17:56 James_F: jjb: Upgrading mediawiki-quibble-vendor-mysql-php72-docker
* 17:51 James_F: jjb: Updating parsoidsvc-parsertests-docker
* 17:49 James_F: jjb: Updating mwext-phpunit-coverage*
* 17:40 James_F: jjb: Updated wikibase-client-docker, wikibase-repo-docker
* 17:36 James_F: jjb: Updating mediawiki-fresnel-patch-docker
* 17:30 James_F: jjv: Updated mwext-php72-phan-docker, mwext-php72-phan-seccheck-docker, mwskin-php72-phan-docker, mwskin-php72-phan-seccheck-docker
* 17:29 James_F: jjb: Updated mwext-codehealth-patch and mwext-codehealth-master-non-voting
* 17:16 James_F: jjv: Updating mwcore-phpunit-coverage-master,mwcore-phpunit-coverage-patch
* 16:58 James_F: jjb: Updating mwcore-codehealth-patch experimentally
* 16:51 James_F: jjb: Experimentally updated mwext-codehealth-master-non-voting
* 16:46 James_F: jjb: Experimentally updated mwext-php72-phan-seccheck-docker-non-voting
* 16:04 hashar: Quibble 0.0.35 containers for CI have been build.
* 15:40 hashar: Building docker images for Quibble 0.0.35
* 15:32 hashar: Tagging quibble 0.0.35 {{Gerrit|48f6b1d8db9eed8572f255fe86f920753149153c}} # [[phab:T225730|T225730]] [[phab:T231862|T231862]] [[phab:T219694|T219694]]
* 14:31 hashar: Raise egress filter trafic shaping on integration-castor03 from 240Mbit to 400Mbit ( [[phab:T232644|T232644]] / [[phab:T188375|T188375]] )
* 12:11 hashar: Remove Jenkins CI webproxy configuration , that breaks jobs using the git plugin!
* 11:26 hashar: Configured CI jenkins to use a webproxy for the update center due to mysterious "connection reset" when downloading plugins
* 09:24 hashar: Manually triggered https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9920/ (switch to php7.2)
* 09:16 hashar: Rebuilding releng/doxygen container  to use php7.2
* 09:15 hashar: On doxygen jobs, change container to use php7.2 instead of php7.0
* 08:19 hashar: Building docker container releng/quibble-stretch-bundle:0.0.34-2  (switch to php7.2)
* 00:29 James_F: Zuul: Dropping php70 and php71 from quibble jobs for master and REL1_34 [[phab:T228342|T228342]]


== 2019-09-16 ==
== 2023-05-25 ==
* 23:54 James_F: Zuul: Initial CI for WebToolsManager
* 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # [[phab:T334521|T334521]]
* 18:52 hashar: Gerrit: tweaked members of the Analytics group # [[phab:T233026|T233026]]
* 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` # [[phab:T334521|T334521]]
* 18:51 hashar: Gerrit: tweaked members of the Analytics group # [[phab:T233026|T233026]]
* 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait  # [[phab:T334521|T334521]]
* 16:00 James_F: Zuul: Restore maps/kartotherian/package to unarchived CI-less midden
* 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd


== 2019-09-14 ==
== 2023-05-24 ==
* 20:45 Krenair: shifted deployment-cache-upload05 to using unified cert from acme-chief
* 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, [[phab:T319477|T319477]]
* 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # [[phab:T336362|T336362]]


== 2019-09-13 ==
== 2023-05-23 ==
* 23:40 Krinkle: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
* 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
* 16:19 James_F: Zuul: Add initial jobs for mediawiki/tools/api-testing [[phab:T230340|T230340]]
* 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` [[phab:T337172|T337172]]
* 12:39 hashar: integration-slave-docker-1043 : killed stall container # [[phab:T232706|T232706]]


== 2019-09-12 ==
== 2023-05-21 ==
* 23:37 James_F: Zuul: Define pipelines for REL1_34 [[phab:T232025|T232025]]
* 10:54 Lucas_WMDE: aborted stuck job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/37265/console which was blocking gate-and-submit
* 22:19 hashar: integration-slave-jessie-1002: rm /srv/pbuilder/build/* /srv/pbuilder/aptcache/*/*
* 22:16 hashar: Pooling back integration-slave-jessie-1002 in Jenkins ( /srv was 95% full )
* 22:15 hashar: integration-slave-jessie-1002: apt-get clean
* 21:45 James_F: Temporarily taking deployment-deploy01 offline as CI is stuck
* 20:54 James_F: Zuul: Drop operations-mw-config-hhvm-composer-test-docker, the PHP72 one suffices
* 20:31 James_F: Zuul: Add seccheck to PageNotice
* 16:36 halfak: deploying ores {{Gerrit|7d45b80}}
* 15:53 hauskatze: Phabricator: Switched articlequality, drafttopic, draftquality and editquality Diffusion (ORES project) to mirror to Gerrit via SSH/K19 refs. [[phab:T232661|T232661]]
* 08:14 hashar: Marking integration-slave-docker-1043 offline , triggering mwcore-phpunit-coverage-master again # [[phab:T232706|T232706]]


== 2019-09-11 ==
== 2023-05-17 ==
* 22:31 Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - [[phab:T143162|T143162]]
* 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # [[phab:T336840|T336840]]
* 21:06 James_F: Zuul: Reload to add operations-mw-config-php72-composer-test-docker and operations-mw-config-hhvm-composer-lint-docker
* 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)
* 14:27 hashar: puppet cert clean integration-agent-docker-1001.integration.eqiad.wmflabs # [[phab:T232619|T232619]] {{!}} might unbreak puppet


== 2019-09-10 ==
== 2023-05-16 ==
* 11:57 hashar: Gerrit: deleting mediawiki/services/service-template-node and mediawiki/services/service-runner (unused) # [[phab:T231953|T231953]]
* 15:11 hashar: Killing stall https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ job
* 08:03 hashar: Updated Phan jobs for [[phab:T232413|T232413]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/535507 phan: stop installing MediaWiki
* 15:04 hashar: CI Jenkins: updating IRC plugin from 2.42 to 3.829.v12d4b_d1f7650  and instant-messaging plugin from 1.48 to 2.666.va_6c1e97cc252  # [[phab:T283009|T283009]]
* 11:30 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919873 {{!}} [[phab:T336218|T336218]]
* 10:55 taavi: update operations-puppet-tests-buster-docker jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/914785


== 2019-09-09 ==
== 2023-05-12 ==
* 15:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535198  Shibboleth disable selenium tests
* 15:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919347/ {{!}} [[phab:T333861|T333861]]
* 12:26 hashar: Reloading Zuul to add multiple extension dependencies
* 12:15 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/535161 " FundraiserLandingPage requires EventLogging"


== 2019-09-06 ==
== 2023-05-11 ==
* 19:59 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/534872
* 21:20 RhinosF1: beta scap is paused while MatmaRex tests some changes
* 19:51 hashar: Reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534862/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534863/ https://gerrit.wikimedia.org/r/#/c/integration/config/+/534864/
* 20:41 thcipriani: pausing beta-code-update-eqiad/beta-mediawiki-config-update-eqiad/beta-scap-sync-world for debugging train
* 19:37 hashar: Reloading zuul for {{Gerrit|Id07a109accc88b138193463a8e7335a60e501910}} Add UniversalLanguageSelector to ExternalGuidance's dependencies
* 16:25 hashar: Reloading Zuul
* 07:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918970 {{!}} [[phab:T333125|T333125]]
* 07:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918474 {{!}} [[phab:T333125|T333125]]


== 2019-09-05 ==
== 2023-05-10 ==
* 19:36 James_F: Zuul: Drop debian-glue-non-voting from cassandra-table-properties
* 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot ([[phab:T327424|T327424]])


== 2019-09-04 ==
== 2023-05-09 ==
* 18:48 James_F: Zuul: Move quibble and mwgate basic test jobs from php70 to php72
* 16:36 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project analytics/datahub --change 916483,4
* 16:49 James_F: Zuul: Replacing basic quibble HHVM test with PHP72 https://gerrit.wikimedia.org/r/533973
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/917868 {{!}} [[phab:T329514|T329514]]
* 16:25 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534481/
* 09:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/916513/ and https://gerrit.wikimedia.org/r/c/integration/config/+/916793/
* 12:20 hashar: Dropped Phan jobs for extensions from release branches  [[phab:T226945|T226945]]
* 08:04 hashar: Reloading Zuul to enable Sonar Codehhealth on TitleBlacklist, TextExtracts, TemplateData and SyntaxHighlight_GeSHI Mediawiki extensions # [[phab:T321837|T321837]]


== 2019-09-03 ==
== 2023-05-05 ==
* 21:59 James_F: Zuul: Move ORES and CirrusSearch to codehealth [[phab:T224990|T224990]]
* 23:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916634
* 20:47 James_F: Zuul: Suspend trigger-mediawiki-pipeline-dev for now [[phab:T218360|T218360]]
* 17:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916527
* 15:57 James_F: Zuul: Enable the publish pipeline for mediawiki/services/kartotherian  [[phab:T223275|T223275]]
* 15:36 James_F: Zuul: Add phan dependencies for MachineVision
* 14:34 James_F: Zuul: And wmf/ branches for VisualEditor core.
* 14:21 James_F: Zuul: Drop EducationProgram from dependencies
* 09:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/533916 "Apply REL based pipelines"
* 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533905/ "Remove wmf pipelines from non mediawiki repos"
* 08:32 hashar: err
* 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533909/ "zuul: add pipelines for MediaWiki releases"


== 2019-09-02 ==
== 2023-05-04 ==
* 17:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533885/ zuul: introduce test-wmf pipeline
* 13:51 _joe_: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/915652
* 17:05 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533880/ "zuul: rename gate-and-submit-swat to use -wmf"
* 12:00 _joe_: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/914782
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533536/ "zuul: drop skip-if for fundraising/REL branch"
* 08:22 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533532/ "zuul: pipelines for fundraising branches"


== 2019-08-31 ==
== 2023-05-03 ==
* 16:20 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/533628/
* 09:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/914715
* 12:28 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]
* 12:22 hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/prometheus-swagger-exporter {{!}} T 231696
* 12:18 hauskatze: github: created https://github.com/wikimedia/operations-debs-prometheus-swagger-exporter {{!}} [[phab:T231696|T231696]]


== 2019-08-30 ==
== 2023-05-02 ==
* 18:28 James_F: Zuul: Move Popups and MobileFrontend to codehealth
* 16:55 dancy: scap version "4.52.0" installed on 15 hosts
* 15:40 Krinkle: Updated "Prod Error" task form to have Title at the top and Trace below Description.
* 08:15 hashar: Reloaded Zuul to enable Sonar Codehealth on CategoryTree https://gerrit.wikimedia.org/r/909331 and PageImages https://gerrit.wikimedia.org/r/909330 # [[phab:T321837|T321837]]
* 10:53 hashar: deployment-mediawiki-07: nuke /mnt/mediawiki  , last touched in April 2018 and using space on /
* 08:14 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/913541 https://gerrit.wikimedia.org/r/913925 https://gerrit.wikimedia.org/r/913191


== 2019-08-29 ==
== 2023-05-01 ==
* 23:23 James_F: Zuul: Move all archived repos to their own section at the end (no-op)
* 17:20 brennen: experimenting with ldap user sync on gitlab.devtools
* 18:38 James_F: Zuul: Adding Wikibase back into dependencies for MachineVision
* 15:47 James_F: Zuul: Drop Wikibase dependency from MachineVision
* 02:32 Krinkle: Delete erroneous branch 'REL1_34-5' ({{Gerrit|108cb95c57a6e02b8}}) from mediawiki/extensions/SoftRedirector. ref https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/


== 2019-08-28 ==
== 2023-04-28 ==
* 23:00 James_F: Zuul: add tests for the labs/tools/maintain-kubeusers repo [[phab:T228499|T228499]]
* 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for  [[phab:T335602|T335602]]
* 23:00 James_F: Zuul: [TimedMediaHandler] Run phan job [[phab:T224766|T224766]] (merged but not deployed?)
* 21:08 thcipriani: restarting zuul server since it's out-of-sync with reality
* 16:11 hashar: Upgraded Jenkins for security update (2.176.3)
* 16:08 hashar: integration: docker container prune -f ; docker image prune -f
* 00:07 James_F: JJB: Delete mwext-npm-doc-publish, no longer used
* 00:07 James_F: Zuul: Migrate MobileFrontend to extension-javascript-documentation [[phab:T230831|T230831]]


== 2019-08-27 ==
== 2023-04-27 ==
* 20:25 James_F: Zuul: TwnMainPage: switch to extension-quibble-composer-nohhvm [[phab:T231289|T231289]]
* 17:10 James_F: Zuul: [wvui] Mark repo as archived for [[phab:T310244|T310244]]


== 2019-08-26 ==
== 2023-04-26 ==
* 22:31 James_F: Zuul: Move Popups to extension-javascript-documentation
* 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list
* 22:12 James_F: Zuul: Move MinervaNeue to extension-javascript-documentation
* 18:23 James_F: Zuul: [MinervaNeue, MobileFrontend, Popups] Add extension-javascript-documentation template
* 15:35 hashar: deployment-mwmaint01: /var/cache/hhvm/cli.hhbc.sq3 (1.3GB) - [[phab:T161598|T161598]]
* 13:51 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/532379


== 2019-08-25 ==
== 2023-04-25 ==
* 21:27 Reedy: centralauth.globalblocks `ALTER TABLE /*_*/globalblocks MODIFY gb_reason varbinary(767) NOT NULL;
* 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False  (manually it works, but that's the same command that puppet is supposed to run !?) - [[phab:T257317|T257317]] [[phab:T326368|T326368]]
* 16:43 Urbanecm: Revert 'Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]'
* 16:33 Urbanecm: Revert {{Gerrit|45831e6}} locally on beta cluster to test possible root cause of [[phab:T231162|T231162]]


== 2019-08-23 ==
== 2023-04-24 ==
* 11:12 Urbanecm: deployment-prep: run scap sync to ensure everything is deployed ([[phab:T231058|T231058]])
* 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - [[phab:T327853|T327853]]
* 07:12 rxy: Applied SQL queries per [[phab:T231058#5433197]]


== 2019-08-21 ==
== 2023-04-23 ==
* 15:49 James_F: Zuul: Add phan for LiquidThreads
* 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%<nowiki>{</nowiki>facts.fqdn<nowiki>}</nowiki>' instead of performance.discovery.wmnet as the latter doesn't resolve  / would be an invalid cert for https://deployment-webperf21, ref [[phab:T291015|T291015]]
* 08:41 hashar: Build docker-registry.discovery.wmnet/releng/quibble-coverage:0.0.34-4  # [[phab:T230423|T230423]]


== 2019-08-20 ==
== 2023-04-22 ==
* 15:55 James_F: Zuul: Update Shreyas Minocha's whitelist e-mail
* 21:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/908941
* 15:45 urandom: creating Parsoid/PHP storage schema in deployment-prep -- [[phab:T230792|T230792]]
* 21:47 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/909624
* 09:39 James_F: Zuul: [kartotherian] Disable pipeline on gate as it doesn't pass [[phab:T223275|T223275]]
* 19:35 Krinkle: Create excimer_ui_server db user in Beta Cluster on deployment-db13 based on prod grants. Password stored in deployment-puppetmaster04:/var/lib/git/labs/private/ (passwords::excimer_ui_server::$excimer_db_pass). Picked db13 because recommendationapi is here (also prod m2), and because mdb (created for this purpose originally) appears broken since several OS iterations (likely forgotten due to unusual name) [[phab:T301637|T301637]], [[phab:T331956|T331956]]
* 18:05 Krinkle: Fix database hostname dns error at https://performance.wikimedia.beta.wmflabs.org/xhgui/, switch from mdb01 to mdb02, ref [[phab:T301637|T301637]]
* 18:00 Krinkle: Move deployment-xhgui03 config from to deployment-xhgui prefix, https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/1d03d4cf84a148bff7e055d9939c44f30c618d85/deployment-prep/deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud.yaml https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ab1c9d2968ba3c292b81ba76f8c0775151630f94%5E%21/, ref [[phab:T301637|T301637]]


== 2019-08-19 ==
== 2023-04-21 ==
* 18:43 thcipriani: git -C /srv/mediawiki-staging/php-master/extensions/ConfirmEdit checkout maintenance/GenerateFancyCaptchas.php
* 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # [[phab:T335181|T335181]]


== 2019-08-15 ==
== 2023-04-20 ==
* 12:13 James_F: Zuul: Make GrowthExperiments depend on Echo for non-Phan too.
* 09:59 taavi: update helm-lint jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/910433/
* 10:50 James_F: Zuul: Make GrowthExperiments depend on Echo
* 09:13 joe: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/909996
* 09:05 James_F: Zuul: Kartotherian pipeline enablement [[phab:T223275|T223275]]


== 2019-08-14 ==
== 2023-04-19 ==
* 18:40 hauskatze: github: Created https://github.com/wikimedia/operations-debs-prometheus-ipsec-exporter && forced replication to mirror changes # [[phab:T230443|T230443]]
* 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for [[phab:T334998|T334998]]
* 18:29 hauskatze: Created new gerrit project: `operations/debs/prometheus-ipsec-exporter.git` # [[phab:T230443|T230443]]
* 10:56 joe: Updating docker-pkg files on contint primary for helm-linter
* 14:42 James_F: Move all BlueSpice extensions to not run HHVM [[phab:T230482|T230482]]


== 2019-08-13 ==
== 2023-04-18 ==
* 22:56 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/530012
* 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
* 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for [[phab:T334397|T334397]]


== 2019-08-08 ==
== 2023-04-17 ==
* 17:14 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/529108
* 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for [[phab:T281500|T281500]]
* 03:18 thcipriani: cancelled stuck beta-scap-eqiad job to unblock postmerge


== 2019-08-07 ==
== 2023-04-12 ==
* 16:00 thcipriani: restarting jenkins for updates
* 15:39 jnuche: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/908248
* 09:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/899523


== 2019-08-06 ==
== 2023-04-10 ==
* 21:24 James_F: Zuul: [TheWikipediaLibrary] Enable plan
* 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
* 21:05 James_F: Zuul: [TheWikipediaLibrary] Enable phan-seccheck
* 18:44 thcipriani: restart zuul-merger
* 21:01 James_F: Zuul: [TheWikipediaLibrary] Add phan dependencies
* 19:26 paladox: ssh -p 29418 gerrit.wikimedia.org replication start --url github mediawiki/core --wait --now
* 17:09 accraze: deploying ores {{Gerrit|d08fa628aacb82529dbb4be357b68dd55c15fdee}}
* 11:42 paladox: ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --wait


== 2019-08-05 ==
== 2023-04-09 ==
* 22:24 accraze: deploying ores {{Gerrit|4270244d4a54c520f581dd33b312aa52f9a4c736}}
* 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2
* 19:22 James_F: Zuul: [mediawiki/services/chromium-render] Enable pipeline testing and publishing [[phab:T217114|T217114]]


== 2019-08-04 ==
== 2023-04-06 ==
* 12:11 hauskatze: gerrit: ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/OATHAuth --wait # [[phab:T229756|T229756]]
* 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list


== 2019-08-02 ==
== 2023-04-05 ==
* 17:30 James_F: Zuul: [operations/mediawiki-config] Stop running php70 jobs"
* 18:05 James_F: Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep, for [[phab:T333661|T333661]]
* 18:05 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production, for [[phab:T331718|T331718]]
* 09:44 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905664


== 2019-08-01 ==
== 2023-04-04 ==
* 20:19 Krinkle: Resolve Jenkins admin warning about legacy token by revoking it from the only user id "<unknown>" that seems to have one
* 05:04 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905262
* 19:00 James_F: Zuul: [BlueSpiceExtendedSearch] Disable HHVM tests, this doesn't work there


== 2019-07-31 ==
== 2023-04-03 ==
* 19:20 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$  mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki ([[phab:T225056|T225056]])
* 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
* 18:27 James_F: Zuul: Move three repos to generic tox-docker job
* 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 {{!}} https://gerrit.wikimedia.org/r/905190
* 17:50 James_F: Zuul: Move cassandra-table-properties to use cassandra-table-properties-tox-docker
* 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
* 16:27 James_F: Zuul: Archive SemanticTitle [[phab:T229422|T229422]]
* 10:20 hashar: Tag Quibble 1.5.4 @ {{Gerrit|0a37ce9367672e973b49b8293f57ebffe40abdf6}} # [[phab:T332930|T332930]] [[phab:T331280|T331280]] [[phab:T333412|T333412]]
* 15:55 James_F: Docker: Pushing operations-puppet:0.5.4
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T333586|T333586]]
* 15:46 James_F: Docker: Pushing quibble-coverage:0.0.34-3
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T33586|T33586]]
* 14:21 andrewbogott: moving deployment-sca02 to cloudvirt1030
* 12:59 andrewbogott: moving deployment-elastic05, deployment-kafka-main-2, deployment-mx02, deployment-webperf11 to new cloudvirts


== 2019-07-30 ==
== 2023-03-31 ==
* 21:17 James_F: Docker: Push operations-puppet:0.5.3
* 16:24 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904818
* 16:17 James_F: Docker: Pushing quibble-coverage:0.0.34-2
* 16:20 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904810
* 15:39 cdanis: manually set dbctl etcd key to dummy value: cdanis@deployment-etcd-01:~$ etcdctl -u root -C http://127.0.0.1:2378 set /conftool/v1/mediawiki-config/eqiad/dbconfig
* 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904805/
* 14:44 James_F: Docker: Pushing quibble-coverage:0.0.34-1
* 14:34 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 14:26 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]


== 2019-07-29 ==
== 2023-03-29 ==
* 18:23 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Make PHP72 job voting, drop HHVM one
* 06:09 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/903839
* 18:13 James_F: Zuul: [translatewiki] [[phab:T228376|T228376]] Provide experimental PHP72 job to replace HHVM one
* 00:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/903539


== 2019-07-27 ==
== 2023-03-23 ==
* 20:38 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-SemanticHighcharts refs. [[phab:T199421|T199421]]
* 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.


== 2019-07-26 ==
== 2023-03-22 ==
* 21:10 legoktm: deploying https://gerrit.wikimedia.org/r/525871
* 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # [[phab:T242031|T242031]]
* 18:38 James_F: Zuul: operations/software/cassandra-table-properties initial jobs
* 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql {{!}} sql centralauth --write # [[phab:T242031|T242031]]
* 16:02 hauskatze: Deleted https://github.com/wikimedia/mediawiki-extensions-Copyvio {{!}} refs. [[phab:T228087|T228087]]
* 16:00 James_F: Zuul: [Copyvio] Mark as archived for CI [[phab:T228087|T228087]]
* 15:49 Krinkle: Fix Jenkins Console output section "Selenium" to account for Quibble 0.34 changes (multiple Selenium sections, one implicitly closing the other)
* 15:45 James_F: doc1001: sudo -u doc-uploader rm -rf /srv/docroot/org/wikimedia/doc/cover-extensions/Copyvio/ # [[phab:T228087|T228087]]
* 04:43 James_F: Zuul: [mediawiki/libs/LangConv] Make this PHP7.2+ only
* 04:36 James_F: Zuul: Add testing for mediawiki/libs/LangConv


== 2019-07-25 ==
== 2023-03-20 ==
* 23:12 hashar: Purging all docker images on Jenkins instances: sudo cumin --trace --force 'name:docker' 'docker images{{!}} awk "{ print \$3}"{{!}}xargs docker rmi'
* 21:43 brennen: phabricator: setting sbassett to administrator per [[phab:T331928|T331928]]
* 22:52 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for  https://gerrit.wikimedia.org/r/525673
* 22:30 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525673
* 21:28 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 (second attempt) for [[phab:T229020|T229020]]  using https://gerrit.wikimedia.org/r/#/c/integration/config/+/525570/
* 21:00 James_F: Zuul: Add phan to EventBus
* 18:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525608 - [[phab:T227004|T227004]])
* 17:38 greg-g: Added Jforrester to the Release Engineering group in Gerrit
* 17:17 hashar: Upgrading Jenkins jobs to Quibble 0.0.34 for [[phab:T229020|T229020]]
* 16:56 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/525591
* 16:54 hashar: Tagged Quibble 0.0.34 {{Gerrit|ff8d86ffca97d17521f96d534f1e43780e6a22d5}} # [[phab:T229020|T229020]]
* 16:41 James_F: Zuul: [analytics-wikistats2] Drop node 6, switch to node10-browser-docker [[phab:T211784|T211784]]
* 16:20 hashar: Rolling back Quibble based jobs to 0.0.32 :-(
* 15:38 hashar: Upgrading Jenkins jobs to Quibble 0.0.33
* 15:09 hashar: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/525559
* 14:52 hashar: Tagged Quibble 0.0.33 {{Gerrit|b2f9e36}}  {{!}} [[phab:T193824|T193824]] [[phab:T87781|T87781]] [[phab:T199116|T199116]]


== 2019-07-24 ==
== 2023-03-19 ==
* 20:00 James_F: Zuul: Mark VIKI, VikiSemanticTitle, and VikiTitleIcon as broken.
* 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in [[phab:T332507|T332507]]
* 10:55 hauskatze: Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` [[phab:T322469|T322469]]
* 10:54 hauskatze: deployment-prep Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. [[phab:T228655|T228655]]
* 14:43 hashar: deployment-prep: databases update hourly job times out, filed [[phab:T332507|T332507]]


== 2019-07-23 ==
== 2023-03-17 ==
* 17:01 James_F: Zuul: [trigger-mediawiki-pipeline-dev] Only trigger on master
* 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 ([[phab:T332455|T332455]])
* 16:00 hashar: gerrit: adding Jakob_WMDE to wmf-deployment group ( https://gerrit.wikimedia.org/r/#/admin/groups/21,members ) # [[phab:T227193|T227193]]
* 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 ([[phab:T332455|T332455]])
* 08:00 hashar: Restore puppet class contint::packages::javascript on permanent slave ( reverting https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524221/ )  # [[phab:T228639|T228639]]
* 10:36 mutante: disabled phab user 'Cloudduck1233'


== 2019-07-22 ==
== 2023-03-16 ==
* 22:43 James_F: Zuul: [wikimedia/iegreview] Adding mediawiki-i18n-check-docker template
* 10:48 hashar: Updating all jobs to switch Castor from Stretch to Bullseye based image {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/899508/ {{!}} [[phab:T278203|T278203]]
* 10:49 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master/extensions$ sudo rm -rf Wikidata
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based) [[phab:T278203|T278203]]
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based)
* 08:20 hashar: github: created https://github.com/wikimedia/mediawiki-services-machinetranslation for replication from Gerrit # [[phab:T331256|T331256]]


== 2019-07-19 ==
== 2023-03-15 ==
* 19:11 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/524565
* 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
* 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/898807 {{!}} [[phab:T331061|T331061]], [[phab:T331621|T331621]]
* 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034


== 2019-07-18 ==
== 2023-03-14 ==
* 22:18 James_F: Zuul: Archive maps/kartotherian and maps/tilerator
* 16:54 hashar: Building Docker images for Quibble 1.5.3
* 21:17 mutante: rebooting gerrit server
* 16:41 hashar: Tag Quibble 1.5.3 @ {{Gerrit|df49e8a04f5}} # [[phab:T331621|T331621]]  [[phab:T331061|T331061]]
* 19:18 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.1
* 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
* 19:03 James_F: Zuul: [analytics-mediawiki-storage] Drop node 6, switch to node10-browser-docker
* 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 18:25 mutante: contint2001 - puppet run reinstalled python-dateutil and ruby-safe-yeaml, stopped zuul service but no more failures now
* 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 18:25 mutante: contint2001 /etc/init.d/zuul-merger stop; apt install zuul (this time it worked to configure zuul after also stopping zuul-merge) ; apt-get autoremove (removed unused python packages); puppet agent -tv
* 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 18:22 mutante: contint2001 /usr/sbin/dpkg-reconfigure: zuul is broken or not fully installed
* 02:44 James_F: MediaWiki being branched to REL1_40 from {{Gerrit|fd3ce7633984d0120288d4a6fe48343cf9ba86a2}} for [[phab:T329079|T329079]]
* 18:22 mutante: contint2001 systemctl stop zuul ; apt install zuul ... still fails
* 18:10 James_F: Zuul: [wikimedia/services/kartotherian] Switch to custom job [[phab:T228170|T228170]]
* 18:06 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.0
* 13:51 hashar: integration: in hiera project config, remove  classes: [ contint::packages::apt ]  {{!}} [[phab:T225735|T225735]]
* 13:48 hashar: integration: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523150/  for apt unattended_upgrades {{!}} [[phab:T225735|T225735]]
* 13:37 hashar: integration: rm /etc/apt/sources.list.d/sury-php.list  and remove php packages {{!}} https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523148/ {{!}} [[phab:T225735|T225735]]


== 2019-07-17 ==
== 2023-03-13 ==
* 16:49 James_F: Zuul: [PageForms] Add phan dependencies on AdminLinks, Cargo, PageSchemas [[phab:T228155|T228155]]
* 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging [[phab:T326192|T326192]]
* 16:48 James_F: Zuul: Register initial tests for eventgate-wikimedia [[phab:T226668|T226668]]
* 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging [[phab:T326192|T326192]]
* 16:08 hashar: Jenkins: updating wikidata-query-gui-build to get rid of slave-scripts/bin/npm-setup.sh which requires an obsolete/non existing tmpfs  # [[phab:T227818|T227818]]
* 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior
* 09:48 hashar: Reloaded Zuul [Convert2Wiki] Archive extension
* 00:42 James_F: Docker: Deploy ci-buster and lintr:0.2.0
* 00:04 James_F: Docker: Complete rebuild finished and published; only took 2.5 hours. But now [[phab:T228196|T228196]] is fixed elsewise anyway, oh well.


== 2019-07-16 ==
== 2023-03-12 ==
* 21:58 hauskatze: GitHub: deleted `wikimedia/mediawiki-extensions-Convert2Wiki` refs. [[phab:T228198|T228198]]
* 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
* 21:29 James_F: Docker: Publishing a whole new set of RelEng images for [[phab:T228196|T228196]]
* 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
* 20:25 James_F: Docker: Running a general rebuild for all missing RelEng images [[phab:T228196|T228196]]
* 22:57 James_F: Zuul: Add Want to the CI whitelist
* 16:49 James_F: Zuul: [ContentTranslation] Adding CentralAuth to phan dependencies
* 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 [[phab:T328240|T328240]]
* 16:48 James_F: Zuul: Adding first tasks for mediawiki/services/kartotherian [[phab:T228170|T228170]]
* 15:19 James_F: Ran manual scap pull on deployment-deploy01
* 15:10 James_F: beta-scap-eqiad failing because mw-config repo is out of date?
* 14:57 James_F: Taking deployment-deploy01 offline
* 14:50 James_F: Jenkins deadlock? Beta jenkins jobs haven't successfully run for ~20 hours.


== 2019-07-15 ==
== 2023-03-11 ==
* 22:40 James_F: Zuul: [CentralAuth] Add phan [[phab:T189371|T189371]]
* 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
* 18:05 James_F: Zuul: [MinervaNeue] Make phan voting [[phab:T133664|T133664]]
* 17:00 James_F: Zuul: [GrowthExperiments] Add MobileFrontend to phan dependencies
* 16:49 James_F: Zuul: [MinervaNeue] Add phan as non-voting [[phab:T133664|T133664]]
* 16:04 Urbanecm: Run urbanecm@deployment-deploy01:~$ mwscript migrateUserGroup.php --wiki=commonswiki Image-reviewer image-reviewer ([[phab:T216406|T216406]])
* 15:59 James_F: Zuul: Adding Echo and MobileFrontend dependencies to skins/MinervaNeue [[phab:T133664|T133664]]
* 15:38 James_F: Zuul: [SecurePoll] Make seccheck voting [[phab:T205972|T205972]]
* 15:37 James_F: Zuul: [CentralAuth] Add seccheck [[phab:T189227|T189227]]
* 13:30 hashar: Deleting compiler1003.puppet-diffs.eqiad.wmflabs .. insanely slow for some reason (was on cloudvirt1002) # [[phab:T221969|T221969]] )
* 12:55 hashar: Creating compiler1003.puppet-diffs.eqiad.wmflabs [172.16.2.46] m1.large / 4vCPUs  # [[phab:T221969|T221969]]


== 2019-07-14 ==
== 2023-03-10 ==
* 00:55 Krinkle: krinkle@doc1001: Remove https://doc.wikimedia.org/cover-extensions/$DOC_BASENAME/  - a failed build for GrowthExperiments
* 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
* 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
* 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]
* 14:37 TheresNoTime: (deployment-prep) clear lock
* 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`


== 2019-07-12 ==
== 2023-03-09 ==
* 23:07 James_F: Zuul: Instead of hacking around for WebAuthn, define bespoke quibble php72+ template
* 20:53 brennen: scheduled tentative phabricator update window for tuesday 2023-03-14 @ 21:00 UTC
* 22:10 James_F: Zuul: Make seccheck voting on WebAuthn [[phab:T226737|T226737]]
* 11:37 hashar: Rolling back selenium-daily Jenkins jobs back to NodeJS 14 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/896026/ {{!}} [[phab:T330426|T330426]]
* 19:28 James_F: Zuul: Add MobileFrontend and MinervaNeue to Wikibase's dependencies
* 10:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895839
* 16:56 James_F: Zuul: Add Flow as phan dependency to GrowthExperiments
* 00:32 James_F: Zuul: Migrate from php70-phan-seccheck to php72-phan-seccheck [[phab:T226420|T226420]]


== 2019-07-11 ==
== 2023-03-08 ==
* 23:33 James_F: Pushing docker mediawiki-phan-seccheck:0.4.0
* 20:12 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/893074
* 21:34 James_F: Pushing new php-based docker images with php-bcmath extension for [[phab:T227043|T227043]]
* 10:36 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895698 {{!}} [[phab:T331505|T331505]]
* 20:30 mutante: add project member cdanis
* 06:26 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895142
* 18:57 Krinkle: Fix "setup quibble prepare" Console Section pattern in Jenkins to account for changes in 'mediawiki-quibble-composertest-php72-docker' job
* 06:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895329 and https://gerrit.wikimedia.org/r/c/integration/config/+/895333 {{!}} [[phab:T321837|T321837]]
* 18:22 James_F: Zuul: Archive ZeroBanner and ZeroPortal; drop from dependencies
* 17:50 James_F: jforrester@doc1001:/srv/docroot/org/wikimedia/doc/cover-extensions$ rm -rf WikibaseQuality/ # [[phab:T205065|T205065]]
* 16:15 James_F: Zuul: Archiving WikibaseQuality
* 09:48 hashar: jenkins: add more log details to hudson.model.Queue (FINER > FINEST) https://integration.wikimedia.org/ci/log/Jenkins%20Queue/configure  # [[phab:T72597|T72597]]
* 09:06 hashar: beta cluster jobs are dead locked. Taking a thread dump in case it helps figure out what is going on. [[phab:T72597|T72597]]


== 2019-07-10 ==
== 2023-03-07 ==
* 21:33 James_F: Zuul: Fix gerrit-docker pointer
* 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, [[phab:T326163|T326163]]
* 18:27 James_F: Zuul: Adding phan to StructuredDiscussions [[phab:T224759|T224759]]
* 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 17:55 James_F: Zuul: Disabling mwgate-rake for mediawiki/extensions/CentralNotice
* 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893765
* 13:30 hashar: on zuul-merger: rm /srv/zuul/git/operations/software/gerrit  due to [[phab:T157818|T157818]]  # [[phab:T189549|T189549]]


== 2019-07-09 ==
== 2023-03-06 ==
* 18:47 James_F: Zuul: Move mediawiki-quibble-composertest from php70 to php72
* 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~`  [[phab:T327853|T327853]]
* 18:18 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move back to php72 as primary
* 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small {{!}} [[phab:T327853|T327853]]
* 18:08 James_F: Push docker composer-package-php72:0.2.2-s2
* 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
* 17:21 James_F: Push docker mediawiki-phan-seccheck:0.3.6-2
* 12:24 hashar: Building Docker images for Quibble 1.5.2 {{!}} https://gerrit.wikimedia.org/r/894617
* 16:18 James_F: Push docker image +mediawiki-phan-seccheck:0.3.6-1
* 12:15 hashar: Tag Quibble 1.5.2 @ {{Gerrit|932b48cab06c}} # [[phab:T331236|T331236]] [[phab:T331061|T331061]] [[phab:T323750|T323750]]
* 15:53 James_F: Pushing docker images php-ast:1.0.1-2 and mediawiki-phan-seccheck:0.3.6
* 15:29 thcipriani: contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to [[phab:T157818|T157818]]
* 15:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/520437
* 14:08 James_F: Pushing mediawiki-phan-seccheck:0.3.6 docker image.
* 09:48 hashar: contint1001: removing /var/lib/docker , no more needed since we now use /mnt/docker # [[phab:T207707|T207707]]
* 00:04 James_F: Pushing php-ast:1.0.1-1 and related composer-package* docker image updates


== 2019-07-08 ==
== 2023-03-05 ==
* 23:54 James_F: Zuul: Move wikimania-scholarships from php56 to php72
* 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons ([[phab:T331214|T331214]]#8666432)
* 23:35 James_F: Dropping mwext-MobileFrontend-npm-run-lint-modules-docker from Zuul [[phab:T224997|T224997]]
* 21:39 thcipriani: reenable jenkins jobs service-pipeline-test and service-pipeline-test-and-publish following docker mount move
* 21:27 thcipriani: docker-pkg failed building docker-registry.discovery.wmnet/releng/composer-php56:0.1.5
* 21:18 thcipriani: fab deploy_docker to pull down ci docker-pkg images (I hope)
* 21:15 thcipriani: restart docker on contint1001 to pickup config change for https://gerrit.wikimedia.org/r/520738/
* 21:14 thcipriani: disabled jenkins jobs service-pipeline-test and service-pipeline-test-and-publish pre-contint1001 docker restart
* 13:38 hashar: Build docker-registry.wikimedia.org/releng/bazel:0.1.0  # [[phab:T189549|T189549]]


== 2019-07-06 ==
== 2023-03-03 ==
* 11:20 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --wiki=deploymentwiki {{!}} [[phab:T209565|T209565]]
* 19:52 zabe: deployment-db13: start replication from deployment-db11 # [[phab:T331019|T331019]]
* 11:09 hauskatze: beta cluster: maurelio@deployment-deploy01 foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --dry-run {{!}} [[phab:T209565|T209565]]
* 19:52 zabe: deployment-db13: import dump into mariadb # [[phab:T331019|T331019]]
* 08:56 hashar: Updating Jenkins jobs with JJB 4.0.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893667/
* 00:37 zabe: take deployment-prep out of read-only # [[phab:T331019|T331019]]


== 2019-07-05 ==
== 2023-03-02 ==
* 21:17 James_F: Adding node10-docker for react.ooui
* 22:01 zabe: enable read-only mode and create dump of all databases # [[phab:T331019|T331019]]
* 19:08 James_F: Pushing java8-sonar-scanner:0.4.11
* 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T331019|T331019]]
* 16:35 zabe: create volume db13 and attach to deployment-db13 # [[phab:T331019|T331019]]
* 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # [[phab:T331019|T331019]]
* 16:24 zabe: shutoff deployment-db09 # [[phab:T331019|T331019]]
* 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # [[phab:T331019|T331019]]
* 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
* 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was {{Gerrit|ed287d58a1e9a2ea39f2eb5bbba31d8fc54269c2}} ) # [[phab:T330752|T330752]]
* 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # [[phab:T330752|T330752]]
* 12:13 zabe: delete deploymentwiki # [[phab:T198673|T198673]]


== 2019-07-03 ==
== 2023-03-01 ==
* 21:01 accraze: deploying ores {{Gerrit|676f7ba43c23e7b6dfa20b3b3c03924cb49acb90}}
* 19:30 hashar: Building Docker images for Quibble 1.5.1 ( https://gerrit.wikimedia.org/r/c/integration/config/+/893528/ )
* 13:12 Urbanecm: deployment-prep Manually ran sudo -u jenkins-deploy scap sync-file wmf-config/CommonSettings-labs.php 'Not synced yet?' to sync file that's mysteriously not synced, but present on deployment host
* 19:17 hashar: Tagged Quibble 1.5.1 @ {{Gerrit|9a163094a855}} # [[phab:T330750|T330750]]
* 18:30 brennen: gitlab: enable custom issue-tracker instance-wide
* 17:37 hashar: Deploying `device-analytics-*` jobs for for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 16:32 taavi: deployment-prep: block automatic/update.php execution of UpdateForMultipleDevicesSupport maintenance script in OATHAuth by pre-inserting rows in updatelog tables, will be run manually instead. [[phab:T242031|T242031]]
* 15:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 14:07 hashar: integration: removed krenair from Puppet config `passwords::root::extra_keys`


== 2019-07-02 ==
== 2023-02-28 ==
* 20:34 James_F: Zuul: Adding quibble to the PopupAlerts, UnblockMe and Scribe extensions
* 15:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892548/
* 14:43 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892993 {{!}} [[phab:T271418|T271418]]
* 14:08 hashar: Updating Quibble jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/891742 (node 14 shipped from the node14 images instead of being manually installed at build time)
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892546 {{!}} [[phab:T311462|T311462]]
* 13:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892952 {{!}} [[phab:T296348|T296348]]
* 12:11 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892919 {{!}} [[phab:T299263|T299263]]
* 10:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892910/ {{!}} [[phab:T299262|T299262]]
* 09:16 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892530 {{!}} [[phab:T299158|T299158]] {{!}} [[phab:T299161|T299161]]


== 2019-07-01 ==
== 2023-02-27 ==
* 23:24 James_F: Pushing quibble-coverage:0.0.32-1 image
* 15:08 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892458 {{!}} [[phab:T299283|T299283]]
* 22:30 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 13:02 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892435 {{!}} [[phab:T327583|T327583]]
* 20:40 James_F: Rebuilt codehealth jjbs using node10-test with https://gerrit.wikimedia.org/r/519652
* 12:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891968
* 18:49 James_F: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental
* 18:43 James_F: Reloading Zuul for mediawiki/tools/phan/SecurityCheckPlugin php72 tests


== 2019-06-30 ==
== 2023-02-25 ==
* 17:03 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/519793
* 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
* 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)


== 2019-06-28 ==
== 2023-02-24 ==
* 08:58 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/519603
* 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- [[phab:T289883|T289883]]
* 00:07 James_F: Restoring php-gmp in quibble docker images
* 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)


== 2019-06-27 ==
== 2023-02-23 ==
* 21:27 James_F: Pushing new docker php base images with gmp, and all their descendants
* 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
* 17:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519455
* 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
* 17:04 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519454
* 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
* 15:29 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424 (try #2)
* 14:00 taavi: update quibble* jenkins jobs for 891303
* 15:22 legoktm: building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424
* 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config


== 2019-06-26 ==
== 2023-02-22 ==
* 17:57 James_F: Adding the experimental helm-lint job to deployment-charts [[phab:T216049|T216049]]
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/891281 [[phab:T328240|T328240]]
* 17:55 James_F: Pushing initial helm-linter docker image for [[phab:T216049|T216049]]
* 14:19 hashar: Updating all jobs to add `QUIBBLE_API_KEY` to the Quibble jobs {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/891278/ {{!}} [[phab:T323750|T323750]]
* 01:29 James_F: Moving labs/tools/stewardbots's composer test to php72+ [[phab:T226585|T226585]]
* 02:48 James_F: Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 for [[phab:T314470|T314470]]
* 02:45 James_F: Zuul: Migrate all MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]
* 02:43 James_F: Zuul: Migrate all non-MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]


== 2019-06-25 ==
== 2023-02-21 ==
* 22:00 hauskatze: Ran `replication start operations/debs/coredns --wait`
* 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
* 21:07 hauskatze: gerrit: Created new repo: operations/debs/coredns - requested by fsero on mediawiki
* 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs [[phab:T329034|T329034]]
* 19:11 Krinkle: Update Jenkins collapsible section patterns to account for "quibble.cmd" => "quibble.commands" changing in the output
* 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
* 15:58 James_F: Adding CentralAuth to FileImporter required extensions in Zuul
* 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
* 15:51 awight: restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation {{!}} [[phab:T321837|T321837]]
* 15:50 awight: deployment-prep restart php7.2-fpm for wikidiff2 upgrade ([[phab:T223391|T223391]])
* 14:39 hauskatze: Ran `replication start operations/debs/file-read-backwards --wait` on gerrit.wikimedia.org refs. [[phab:T226449|T226449]]
* 14:34 hauskatze: github: created https://github.com/wikimedia/operations-debs-file-read-backwards mirror refs. [[phab:T226449|T226449]]
* 14:14 hauskatze: gerrit: created `operations/debs/file-read-backwards.git` refs. [[phab:T226449|T226449]]
* 11:42 hashar: Deleting Jenkins jobs puppet-cdh-rake-docker and puppet-cdh4-rake-docker # [[phab:T226474|T226474]]


== 2019-06-24 ==
== 2023-02-20 ==
* 23:31 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 00:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/890197
* 22:08 James_F: Zuul: Restore graphoid deploy repo tests, and comment on change-propagation ones.
* 22:02 James_F: Pushed graphoid-deploy-npm-node-6-docker into jjb
* 22:01 James_F: Re-publishing npm-test-graphoid, whoops.
* 15:17 James_F: Reloading Zuul to drop wikiba.se-composer-php56-docker [[phab:T224905|T224905]]
* 13:04 hashar: Updating git mirrored repos: integration-cumin:~$ sudo cumin --trace --force 'name:docker' 'find /srv/git -name "*.git" -exec git -C {} fetch --prune \;'
* 12:00 hashar: Updating Quibble based jobs to use version 0.0.32
* 10:09 hashar: Tag Quibble 0.0.32 @ {{Gerrit|c0fe6eb}} # [[phab:T211701|T211701]] [[phab:T218357|T218357]] [[phab:T220199|T220199]] [[phab:T223752|T223752]]
* 09:50 hashar: Building container releng/zuul-cloner:0.2.0 and switching jobs # [[phab:T224908|T224908]]
* 08:25 hashar: Removing mysql-server from the CI slaves # https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517095/ {{!}} [[phab:T225735|T225735]]


== 2019-06-22 ==
== 2023-02-16 ==
* 11:36 hauskatze: github: deleted `wikimedia/mediawiki-extensions-FirstSteps` GitHub mirror refs. [[phab:T226149|T226149]]
* 10:31 jnuche: restarted Zuul to clear stale events
* 00:31 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/518344


== 2019-06-21 ==
== 2023-02-15 ==
* 20:52 Krinkle: Add "post-build scripts" Jenkins Console section to collapse most unrelated output and cascading failures from artefact readers etc. which are confusing when looking for "the" error
* 20:07 taavi: unbroke puppet on deployment-ms-*
* 20:48 Krinkle: Fix Jenkins Console Section pattern for "setup quibble mw-install" to not eat all of output from phan (differentiate from phpunit, composer-test and phan). regexcomplexity++
* 18:12 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/889499
* 16:58 James_F: Updated all four codehealth jjb jobs to use java8-sonar-scanner 0.4.10
* 16:03 thcipriani: restart ci jenkins for updates
* 16:55 James_F: Docker: Publishing java8-sonar-scanner 0.4.10.
* 16:02 thcipriani: restart releases jenkins for updates
* 06:37 legoktm: deployed https://gerrit.wikimedia.org/r/518205 https://gerrit.wikimedia.org/r/518206
* 06:34 legoktm: deployed https://gerrit.wikimedia.org/r/518204
* 06:04 legoktm: deleted remexhtml-phpunit-coverage-docker-publish jenkins job
* 05:51 legoktm: deployed https://gerrit.wikimedia.org/r/518195 https://gerrit.wikimedia.org/r/518196


== 2019-06-20 ==
== 2023-02-14 ==
* 22:03 James_F: Pushing new java8-sonar-scanner:0.4.9 image.
* 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er [[phab:T329000|T329000]]
* 21:29 James_F: Pushing new java8-sonar-scanner:0.4.8 image
* 15:02 zabe: deployment-db12: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 20:06 James_F: Add new mwcore-codehealth-* config to Zuul
* 14:04 zabe: delete deployment-db10 and volume db10 # [[phab:T329577|T329577]]
* 20:04 James_F: Pushed new jjb mwcore-codehealth-* jobs and updated mwext-codehealth-* ones
* 11:38 zabe: deployment-db11: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 19:54 James_F: Pushing new java8-sonar-scanner:0.4.7 image
* 02:19 zabe: deployment-db12: import dump into mariadb # [[phab:T329577|T329577]]
* 17:45 James_F: Updated JJB definitions for mwext-php7\d-phan(-seccheck)?-docker for new docker-setup-mwext-for-phan macro.
* 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # [[phab:T329577|T329577]]
* 17:35 James_F: Updated JJB definitions for wikibase-{client{{!}}repo}-docker to use php72 not php70.
* 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T329577|T329577]]
* 17:12 James_F: Updated JJB definitions for all six coverage jobs
* 01:50 zabe: create volume db12 and attach to deployment-db12 # [[phab:T329577|T329577]]
* 15:09 hashar: Nuking castor caches for Quibble jobs  integration-castor03 : rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/*quibble*/{composer,npm}
* 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 14:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/518033/
* 01:14 zabe: deployment-db11: import dump into mariadb # [[phab:T329577|T329577]]
* 14:16 hashar: building some container for codehealth pipeline  https://gerrit.wikimedia.org/r/518034
* 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # [[phab:T329577|T329577]]
* 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # [[phab:T329577|T329577]]


== 2019-06-19 ==
== 2023-02-13 ==
* 08:58 hashar: Removed Debian packages for python / tox etc from all slaves {{!}} [[phab:T225735|T225735]] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517092/
* 23:48 zabe: create volume db11 and attach to deployment-db11 # [[phab:T329577|T329577]]
* 08:46 hashar: Removed Debian package colordiff  from all slaves {{!}} [[phab:T225735|T225735]] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517091/
* 23:44 zabe: shutoff deployment-db10 # [[phab:T329577|T329577]]
* 03:43 legoktm: deploying https://gerrit.wikimedia.org/r/517785 https://gerrit.wikimedia.org/r/517786
* 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 23:20 zabe: created backup of all databases on deployment-db09 # [[phab:T329577|T329577]]
* 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
* 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm


== 2019-06-18 ==
== 2023-02-10 ==
* 23:19 James_F: Making extension-seccheck non-voting on all 195 repos, hopefully temporarily. [[phab:T226064|T226064]]
* 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for [[phab:T277440|T277440]]
* 22:14 Reedy: Apply https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OATHAuth/+/496451/53/sql/mysql/patch-add_generic_fields.sql to centralauth on beta [[phab:T218210|T218210]]
* 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for [[phab:T324732|T324732]]
* 21:59 James_F: Manually pruned some old quibble-coverage Docker images from contint1001.
* 21:51 James_F: Updating docker-pkg files on contint1001 to fix quibble-stretch-php72
* 21:43 James_F: Zuul: Adding php72 variants of phan-seccheck jobs as experimental
* 21:35 James_F: Pushing docker image quibble-stretch-php72:0.3.3
* 20:40 James_F: Reload Zuul to point MediaWiki's phan jobs at php72 instead of php70 [[phab:T225456|T225456]]
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517733/
* 20:24 hashar: Switching jobs to SRE php7.2 package # [[phab:T226028|T226028]]
* 20:22 James_F: Reload Zuul to point Parsoid's experimental phan job at php72 instead of php70 [[phab:T225456|T225456]]
* 20:12 James_F: Reload Zuul to point extension-phan and skin-phan templates at php72 instead of php70 [[phab:T225456|T225456]]
* 19:51 hashar: building containers for php7.2 from SRE # [[phab:T226028|T226028]] https://gerrit.wikimedia.org/r/#/c/integration/config/+/517684/
* 19:38 James_F: Bumped jjb mwext-codehealth-*, mediawiki-*coverage*docker*, mwext*coverage*docker* jobs to use quibble-coverage:0.0.31-4
* 19:28 James_F: Pushing quibble-coverage:0.0.31-4 docker image.
* 17:02 James_F: Updated jjb jobs for mwext-codehealth-patch and mwext-codehealth-master-non-voting.
* 16:54 James_F: Pushing new quibble-coverage docker image
* 11:32 hauskatze: github: deleted https://github.com/wikimedia/mediawiki-extensions-PureWikiDeletion {{!}} [[phab:T225991|T225991]]


== 2019-06-15 ==
== 2023-02-08 ==
* 00:17 Krinkle: Disable "Publish/Notify" for mediawiki/libs/ObjectFactory.git in Diffusion –&nbsp;[[phab:T222409|T222409]], [[phab:T143162|T143162]]
* 23:17 James_F: Zuul: Add Nik.xyz.in e-mail to CI allow list
* 21:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/887829 # [[phab:T329231|T329231]]
* 18:11 Krinkle: Disable spam account https://phabricator.wikimedia.org/people/manage/35991/
* 12:37 James_F: Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job
* 09:16 hashar: deployment-prep:  deployment-deploy03 `systemctl stop ferm`  to allow jenkins to connect to the instance # [[phab:T329056|T329056]]


== 2019-06-14 ==
== 2023-02-07 ==
* 22:15 James_F: Dropped jjb jobs trending-edits?(-deploy)-npm-node-6-docker
* 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for [[phab:T329077|T329077]]
* 22:14 James_F: Reloading Zuul to archive the trending-edits repos
* 21:50 James_F: Dropped docker-registry.discovery.wmnet/releng/hhvm-jessie:latest, unused
* 19:57 James_F: Ran `docker rmi docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-4` on contint1001; unused.
* 19:41 James_F: Zuul: Deploying Codehealth: Add Translate extension to pipeline
* 19:39 James_F: Zuul: Deploying Remove coverage ci step for extensions with codehealth
* 19:35 James_F: Zuul: Deploying Add sonar CodeHealth checks to Wikibase repos
* 18:44 James_F: Publishing image docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-5
* 16:58 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/517087
* 16:08 James_F: Updating docker-pkg to provide quibble-coverage on php7.2
* 16:04 James_F: Dropped docker-registry.discovery.wmnet/releng/quibble-stretch-php70:0.0.31-1 from contint1001, unused
* 15:50 James_F: Bumped jjb mediawiki-phpunit-coverage-patch-docker


== 2019-06-13 ==
== 2023-02-06 ==
* 22:30 James_F: Deploying quibble-stretch-php70 update.
* 09:30 hashar: Built tox Docker images to include `bats` {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/871178/
* 22:17 James_F: Deploying quibble-coverage fix with chmod 0755 files.
* 21:49 James_F: Reverted JJB jobs mediawiki-core-code-coverage-docker and mediawiki-phpunit-coverage-patch-docker back to quibble-stretch-php70
* 21:43 James_F: Updated JJB job mediawiki-phpunit-coverage-patch-docker to point to quibble-coverage
* 21:42 James_F: Update JJB job mediawiki-core-code-coverage-docker to point to quibble-coverage
* 21:10 brennen: restarted jenkins agent on integration-trigger-01 to pick up new root configuration
* 20:54 hashar: ¡Learning to log one step at a time!
* 20:54 James_F: Updating docker-pkg files to create quibble-coverage image
* 20:40 hashar: deleting docker-registry.wikimedia.org/releng/hhvm-jessie-compile from contint1001 # [[phab:T224908|T224908]]
* 18:17 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/514016
* 17:05 brennen: changing jenkins remote root for integration-trigger-01 to /mnt/home/jenkins-deploy per [[phab:T224069|T224069]]
* 16:07 James_F: Dropped docker-registry.discovery.wmnet/releng/npm-test-mathoid:latest, unused
* 16:06 James_F: Dropped mathoid?(-deploy)-npm-node-6-docker from jjb
* 16:01 James_F: Dropped docker-registry.discovery.wmnet/releng/npm6:latest from contint1001, unused
* 15:33 Krinkle: Disable "Publish/Notify" for quibble.git in diffusion, [[phab:T143162|T143162]], [[phab:T196347|T196347]]
* 00:13 James_F: Dropped jjb jobs for (change-propagation{{!}}citoid{{!}}cxserver{{!}}graphoid)?(-deploy)-npm-node-6-docker


== 2019-06-12 ==
== 2023-02-03 ==
* 23:34 brennen: deleting and rebuilding integration-trigger-01 (again) to use jessie instead of stretch
* 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old
* 23:20 James_F: Zuul: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now
* 22:44 James_F: Enabling service-pipeline-test on four services: 3d2png, chromium-render, recommendation-api, trending-edits
* 22:40 James_F: Dropped jjb jobs wikimedia-cz-tracker-npm-node-6-docker fundraising-dash-npm-node-6-docker wikimedia-portals-npm-browser-node-6-docker wikimedia-portals-deploy-npm-browser-node-6-docker
* 22:38 brennen: deleting and re-creating integration-trigger-01
* 22:22 James_F: Dropped mapdata-npm-node-6-docker and marvin-npm-node-6-docker from jjb
* 22:21 James_F: Reloading Zuul to make mpadata use generic-node10-browser-docker
* 21:54 James_F: Reloading Zuul to deploy, umm, 12 patches. 🤷🏽‍♂️
* 21:07 James_F: Updating Zuul with https://gerrit.wikimedia.org/r/516674 https://gerrit.wikimedia.org/r/516675 https://gerrit.wikimedia.org/r/516676
* 18:35 James_F: Reloaded Zuul to use node10 things for OOUI experimentally
* 18:32 James_F: Deployed new jjb job ooui-npm-special-node10-plus-composer-docker
* 16:28 Krinkle: Fix Jenkins Console section "setup quibble mw-install" pattern to not slurp the whole output on 'mediawiki-quibble-composertest' jobs
* 16:27 James_F: Reloading Zuul to add service-pipeline-test?(-and-publish) to mediawiki/services/mobileapps
* 00:49 James_F: Updated Zuul phan dependencies for ContentTranslation, adding GlobalPreferences


== 2019-06-11 ==
== 2023-02-02 ==
* 23:51 James_F: Updated Zuul phan dependencies for WikibaseLexeme, adding Wikibase [[phab:T225524|T225524]]
* 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for [[phab:T321309|T321309]]
* 23:37 James_F: Dropped unused jjb jobs mwext-jsduck-publish mwext-VisualEditor-jsduck-docker, and mwext-VisualEditor-docker-publish
* 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete  "' research/navigation-vectors'"`
* 23:34 James_F: Updating Zuul to drop extension-jsduck, no longer used (yay)
* 23:14 James_F: Reloading Zuul to have Wikibase use extension-javascript-documentation
* 22:39 James_F: PHP55 docker images deleted; needed to specify :0.… and also :latest because we're not just dropping one tag, we're dropping the double pair. All now done. [[phab:T224907|T224907]]
* 22:34 James_F: PHP55 docker images not deleted because they're still referenced (by what?).
* 22:25 James_F: Running `docker rmi` for quibble-jessie-php55 ({{Gerrit|938a4b4f6abb}}) composer-test-php55 ({{Gerrit|9731d99e7a6f}}) composer-package-php55 ({{Gerrit|906edee88f4f}}) php55-compile ({{Gerrit|01f3fa5ef542}}) composer-php55 ({{Gerrit|a14836c5fc62}}) and php55 ({{Gerrit|f8b5dc437239}})
* 22:22 James_F: Updating docker-pkg files on contint1001 to drop all php55 images [[phab:T224907|T224907]]
* 22:18 James_F: Dropped from jjb: mwgate-composer-php55-docker, composer-php55-docker, composer-package-php55-docker, php-compile-php55-docker
* 22:06 James_F: Drop unused jjb job integration-composer-check-php55-docker
* 22:06 James_F: Reloading Zuul to drop integration-composer-check-php55-docker from integration/composer
* 22:01 James_F: Reloading Zuul to add integration-composer-check-php72-docker to integration/composer
* 21:59 James_F: Push new jjb job integration-composer-check-php72-docker
* 21:56 James_F: Reloading Zuul to drop all *php55* jobs except integration-composer
* 21:49 James_F: Reloading Zuul to add composer-php56-docker to integration/docroot
* 21:45 James_F: Reloading Zuul to drop quibble*php55*
* 21:42 James_F: Reloading Zuul to add composer-php72-docker to labs/tools/heritage
* 21:07 brennen: creating integration-trigger-01 for [[phab:T224069|T224069]]
* 19:34 hashar: building docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.4.5 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/516511/
* 19:04 halfak: deployment-prep deploying ores {{Gerrit|04fbd58}}
* 19:03 mdholloway: deployment-prep created deployment-wikifeeds01, added role::beta::docker_services and service config
* 18:29 halfak: deployment-prep deploying ores {{Gerrit|86809a4}}
* 13:17 hashar: Building container docker-registry.wikimedia.org/releng/civicrm:0.1.0 # [[phab:T210287|T210287]]


== 2019-06-10 ==
== 2023-01-31 ==
* 19:50 James_F: Restarting zuul to drop quibble experimental php70-phan-docker jobs; php72 suffices
* 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
* 19:39 thcipriani: restarting jenkins
* 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
* 19:37 James_F: Restarting jenkins with `sudo service restart jenkins` on contint1001 due to out-of-space issue earlier.
* 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
* 19:10 Krinkle: Fix "setup quibble mw-install" collapsible section pattern in Jenkins administration, so that it doesn't match the "PHP 7.1+" warning from Phan. The pattern is meant to detect the first line of mw/maint/install.php output
* 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for [[phab:T326372|T326372]]
* 19:01 thcipriani: cleaning contint1001 docker images using https://phabricator.wikimedia.org/P8604
* 17:48 James_F: Reloading zuul to set zuul parameters for PHP 7.2 variants of phan jobs too
* 17:41 James_F: Reloading zuul to Add experimental mwext-php72-phan-docker jobs to quibble template
* 17:38 James_F: Pushed new mwext-php72-phan-docker with jjb
* 16:59 James_F: Deployed https://gerrit.wikimedia.org/r/516300
* 16:31 legoktm: deployed https://gerrit.wikimedia.org/r/516298
* 04:38 legoktm: deploying https://gerrit.wikimedia.org/r/516065 https://gerrit.wikimedia.org/r/516066


== 2019-06-08 ==
== 2023-01-30 ==
* 00:49 James_F: Updating docker-pkg files on contint1001 to build mediawiki-phan:0.2.0 based on php72 [[phab:T223847|T223847]]
* 23:23 James_F: Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' for [[phab:T327332|T327332]]
* 18:35 James_F: Zuul: [mediawiki/extensions/PushAll] Add basic tests
* 14:39 hashar: Building Docker image for https://gerrit.wikimedia.org/r/884904
* 10:59 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/884835/


== 2019-06-07 ==
== 2023-01-27 ==
* 23:29 marxarelli: deploying zuul changes for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 20:42 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/884372
* 23:28 marxarelli: updating jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 19:35 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/884365/
* 13:30 godog: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/515058
* 14:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/884143 {{!}} [[phab:T315621|T315621]]
* 00:10 James_F: jjb: Migrated docker-setup-mwext-for-phan to node10 quibble
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884309
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884283
* 08:39 hashar: devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master {{!}} [[phab:T325128|T325128]]


== 2019-06-06 ==
== 2023-01-25 ==
* 18:06 James_F: Reloading Zuul to drop EventBus dependency on CentralNotice for now [[phab:T225195|T225195]]
* 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner.  [[phab:T327949|T327949]]
* 15:12 hashar: reloading zuul for {{Gerrit|I044bedd41091b57a1a188ea243c4769a6e21babb}} [[phab:T222802|T222802]]
* 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for [[phab:T327949|T327949]]
* 15:12 hashar: reloading Zuul for {{Gerrit|I8d6ea5b5491fe50d7144926ef8bc69d4704d4fab}} [[phab:T224301|T224301]]
* 08:13 hashar: Reloading Zuul for {{Gerrit|I764972711843645afd00e196a3bedd17730b4cbe}}  which drops  mwselenium-quibble-docker from Wikibase


== 2019-06-05 ==
== 2023-01-24 ==
* 19:34 andrewbogott: moving deployment-imagescaler03 to cloudvirt1029
* 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [[phab:T326372|T326372]]
* 17:35 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547
* 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for [[phab:T326372|T326372]]
* 16:01 James_F: Reloading Zuul to deploy {{Gerrit|172a4e7886b285adb48a85dce7287dce5c3778e2}}
* 17:19 thcipriani: restarting ci jenkins for updates
* 09:07 hashar: Pooled in integration-slave-docker-1058 and integration-slave-docker-1059
* 17:15 thcipriani: restarted releases-jenkins for plugin updates


== 2019-06-04 ==
== 2023-01-23 ==
* 20:39 James_F: Updating jjb PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [[phab:T220917|T220917]]
* 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
* 20:31 James_F: Updating docker-pkg files on contint1001 for quibble-stretch-php70 [[phab:T220917|T220917]]
* 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for [[phab:T292818|T292818]]
* 17:44 James_F: Updating jjb PHP code coverage jobs to node10 quibble [[phab:T224983|T224983]]
* 17:31 James_F: Reloading Zuul to make mwext-MobileFrontend-npm-run-lint-modules-docker non-voting [[phab:T224997|T224997]]
* 16:10 hashar: Deleting  integration-slave-docker-1021 and integration-slave-docker-1049  / too small disk (20G partition) and not enough ram (2G) # [[phab:T221872|T221872]]
* 13:24 hashar: Update all selenium-daily* jobs to use NodeJS 10 instead of NodeJS 6. [[phab:T217545|T217545]]
* 13:01 hashar: Building docker-registry.discovery.wmnet/releng/node10-test-browser:0.6.0 # [[phab:T217545|T217545]]
* 11:32 hashar: Upgrading Jenkins Pipeline plugins
* 11:31 hashar: Upgrading Jenkins Warnings Next Generation Plugin # [[phab:T224745|T224745]]
* 11:30 hashar: Upgrading Jenkins BlueOcean and all its dependencies


== 2019-06-03 ==
== 2023-01-20 ==
* 19:32 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514089
* 06:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881665 {{!}} [[phab:T327301|T327301]]
* 19:29 James_F: Generated mwgate-node10-docker and deployed via jjb.
* 18:34 hashar: hswitch most Quibble jobs to node 10 [[phab:T222406|T222406]] - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ [[phab:T222406|T222406]]
* 17:40 James_F: Reloading Zuul to switch most Quibble jobs to node 10 [[phab:T222406|T222406]]
* 15:58 hashar: Deleting integration-slave-docker-1055 and integration-slave-docker-1056 . CPU is way too slow [[phab:T223971|T223971]]
* 15:46 hashar: reduce number of executors on all integration slave docker, they are somehow starving on CPU and/or IO when lot of mediawiki builds are running in parallel
* 14:38 James_F: hashar and I are temporarily disabling running selenium tests in CI. See [[phab:T211784|T211784]] [[phab:T222406|T222406]] for more details.


== 2019-06-01 ==
== 2023-01-19 ==
* 22:54 James_F: Reloading Zuul to re-add Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 20:01 James_F: Zuul: Add Izno to the CI allowlist
* 21:14 James_F: Reloading Zuul to add phan for CentralNotice and add DannyS712 to whitelist
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # [[phab:T299954|T299954]]
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
* 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
* 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
* 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907


== 2019-05-31 ==
== 2023-01-18 ==
* 23:34 James_F: Reloading Zuul to remove Kartographer dependency for JsonConfig [[phab:T224785|T224785]]
* 21:33 dancy: Rebooting deployment-jobrunner04 for [[phab:T327329|T327329]]
* 22:12 James_F: Reloading Zuul to add phan for LdapAuthentication
* 17:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881457 {{!}} [[phab:T327301|T327301]]
* 21:18 James_F: Reloading Zuul to add dependencies for TimedMediaHandler, ReadingLists, JsonConfig, and FlaggedRevs.
* 15:42 thcipriani: deployment-jobrunner04: sudo rm -rf /srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~
* 21:13 James_F: Reloading Zuul to add phan for Collection, ContentTranslation, and Jade
* 15:31 andrewbogott: shutting down deployment-imagescaler03 as it is long-overdue for replacement.  See [[phab:T294148|T294148]] for details.
* 19:43 James_F: Reloading Zuul to add phan for Math(s)
* 10:10 arturo: bump trove quotas ([[phab:T326674|T326674]])
* 19:31 James_F: Reloading Zuul to add phan for MobileFrontend
* 19:20 James_F: Reloading Zuul to add phan for Sentry
* 18:35 James_F: Reloading Zuul to deploy CentralNotice dependency to EventBus
* 18:17 James_F: Reloading Zuul to deploy extra dependencies for MobileFrontend
* 18:10 James_F: Reloading Zuul to deploy phan for cldr and GlobalPreferences
* 18:02 James_F: Reloading Zuul to deploy {{Gerrit|I0ee1c8166}}
* 17:56 James_F: Reloading Zuul to deploy {{Gerrit|I9c115b1f6}} and {{Gerrit|I0d98f01d}}


== 2019-05-30 ==
== 2023-01-17 ==
* 23:19 James_F: Pruned releng/quibble-fresnel:0.0.31-3 {{Gerrit|8ca03484a2c3}} from contint1001 to save space
* 12:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880951/
* 23:02 James_F: Updating docker-pkg files on contint1001 for +quibble-fresnel (0.0.31-4)
* 10:05 hashar: Updating sonar-scanner jobs https://gerrit.wikimedia.org/r/c/integration/config/+/880893/
* 21:33 Krinkle: Jenkins admin says ""Your Jenkins data directory /var/lib/jenkins (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full.""
* 09:52 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-4
* 21:27 greg-g: back in business (ugh)
* 08:32 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880503 {{!}} [[phab:T321837|T321837]]
* 21:21 greg-g: doing the zuul deadlock dance again
* 00:21 James_F: Zuul: [operations/debs/cadvisor] Add debian-glue-unstable CI for [[phab:T325557|T325557]]
* 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/513168
* 00:21 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] +AbuseFilter phan dep
* 02:48 legoktm: deploying https://gerrit.wikimedia.org/r/512474 https://gerrit.wikimedia.org/r/512507 https://gerrit.wikimedia.org/r/513153 https://gerrit.wikimedia.org/r/513157 https://gerrit.wikimedia.org/r/512729 https://gerrit.wikimedia.org/r/513206 https://gerrit.wikimedia.org/r/513214 https://gerrit.wikimedia.org/r/513217 https://gerrit.wikimedia.org/r/513218 https://gerrit.wikimedia.org/r/513211
* 02:35 legoktm: deployed https://gerrit.wikimedia.org/r/513129 https://gerrit.wikimedia.org/r/513071


== 2019-05-29 ==
== 2023-01-16 ==
* 21:46 greg-g: doing the executor deadlock dance: https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Jenkins_execution_lock
* 22:29 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/880541
* 00:19 Krinkle: Reloading Zuul


== 2019-05-28 ==
== 2023-01-13 ==
* 12:43 paladox: delete extensions/3D repo in gerrit - [[phab:T224463|T224463]]
* 10:41 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/879754
* 01:35 legoktm: deployed https://gerrit.wikimedia.org/r/512794
* 01:35 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879687


== 2019-05-26 ==
== 2023-01-12 ==
* 18:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/512544
* 21:49 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879649
* 10:42 hashar: Deployed docroot change https://gerrit.wikimedia.org/r/878860 "zuul: Link to report_url if available"


== 2019-05-22 ==
== 2023-01-10 ==
* 07:21 hashar: Updating Jenkins job to have castor use rsync --delay-updates # [[phab:T203506|T203506]] {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/511672/
* 21:14 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/27
* 16:50 James_F: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf for [[phab:T326653|T326653]]
* 15:28 James_F: Zuul: Add experimental jobs for fundraising/REL1_39 for [[phab:T326372|T326372]]


== 2019-05-21 ==
== 2023-01-09 ==
* 22:05 marxarelli: generating new blubber-pipeline-* and trigger-blubber-pipeline-* jenkins jobs defined by https://gerrit.wikimedia.org/r/c/integration/config/+/510602
* 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
* 11:23 hashar: Depooling integration-slave-docker-1055 and integration-slave-docker-1056 : CPU is too slow # [[phab:T223971|T223971]]
* 16:30 wm-bot2: Increased quotas by 6 cores ([[phab:T326568|T326568]]) - cookbook ran by arturo@nostromo
* 08:24 hashar: Updated phan jobs to no more install mediawiki development dependencies (potentially conflicting with the extensions code) https://gerrit.wikimedia.org/r/#/c/integration/config/+/511447/ # [[phab:T223397|T223397]]
* 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests
* 07:29 legoktm: deployed https://gerrit.wikimedia.org/r/511627
* 07:10 legoktm: manually rebuilding mediawiki-core-code-coverage-docker with 5 hour timeout


== 2019-05-20 ==
== 2023-01-08 ==
* 13:11 hashar: updating phan jobs to use docker-registry.wikimedia.org/releng/mediawiki-phan:0.1.15 # [[phab:T219114|T219114]]
* 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
* 01:43 legoktm: deploying https://gerrit.wikimedia.org/r/511361
* 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix


== 2019-05-19 ==
== 2023-01-07 ==
* 15:43 hashar: Purging webperformance.integration.eqiad.wmflabs  /tmp directory (inodes full)
* 22:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876327
* 15:43 hashar: Purging all images on Docker CI slaves
* 08:03 legoktm: deployed https://gerrit.wikimedia.org/r/511122


== 2019-05-18 ==
== 2023-01-06 ==
* 18:14 Amir1: cherry-picking 511078 on puppetmaster
* 16:59 James_F: Docker: Publishing new CI images with composer 2.5.1 for [[phab:T319257|T319257]]
* 12:07 Reedy: reload zuul to deploy https://gerrit.wikimedia.org/r/510967
* 11:43 hashar: Refreshing Jenkins jobs to update Quibble jobs with php-excimer {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/876190


== 2019-05-17 ==
== 2023-01-05 ==
* 22:55 paladox: created mediawiki/extensions/Scribe gerrit repo - [[phab:T223662|T223662]]
* 09:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868193 {{!}} Make CentralAuth a clone dependency for MassMessage {{!}} [[phab:T145430|T145430]]
* 00:35 brennen: Updating dev-images docker-pkg files on contint1001


== 2019-05-16 ==
== 2023-01-04 ==
* 09:36 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/composer-test-php72:0.1.0 # [[phab:T223428|T223428]]
* 19:42 MatmaRex: Ran maintenansce script refreshGlobalimagelinks.php for [[phab:T322588|T322588]]
* 07:35 hashar: integration-slave-jessie-1002: purging all php Debian packages and rerunning puppet. There is some package conflict somewhere :-(
* 02:47 TimStarling: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/874959


== 2019-05-15 ==
== 2023-01-03 ==
* 11:55 hashar: Bring back https://integration.wikimedia.org/ci/computer/integration-castor03/ to restore the central cache behavior. In turn unblocking a wide range of builds
* 19:14 dduvall: restarting zuul
* 11:32 hashar: bringing back castor, integration-castor03 is back
* 19:12 Krinkle: Disable publishing of rPMSM, ref [[phab:T143162|T143162]], [[phab:T326112|T326112]]
* 09:35 hashar: Regenerating all CI jobs to disable castor saving entirely
* 17:38 hashar: gerrit: reactivated account for `jmando` using `gerrit set-account 9431 --active` {{!}} [[phab:T326159|T326159]]
* 02:35 Krenair: Logged into deployment-sca0[12] as root and given them the correct nameservers to try to unbreak things. [[phab:T221654|T221654]]
* 14:20 hashar: integration: restarting all instances for Linux kernel upgrade
* 01:00 thcipriani: /usr/local/sbin/keyholder arm
* 07:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # [[phab:T322368|T322368]]


== 2019-05-14 ==
* 09:43 hashar: Upgraded all tox jobs to tox 3.10.0
* 09:19 hashar: Building docker containers releng/tox-*:0.4.0
== 2019-05-13 ==
* 20:21 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/509930
* 19:47 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/502606
* afk: updating docker-pkg images on contint1001 for https://gerrit.wikimedia.org/r/508019
* 14:48 hashar: if you build Docker containers, there is a long delay between it being build/published and it actually being available https://phabricator.wikimedia.org/T222210#5176863  known issue
* 13:49 hashar: Building docker image releng/tox:0.4.0
* 10:37 hashar: Rolling CI config change https://gerrit.wikimedia.org/r/508512 which caused some patches to not be processed last week # https://wikitech.wikimedia.org/wiki/Incident_documentation/20190506-zuul / [[phab:T105474|T105474]]
== 2019-05-10 ==
* 18:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/509483
* 18:08 Reedy: Reloading Zuul to deploy dependancies, new tests, phan
* 17:53 Reedy: Reloading Zuul to deploy 5 more phan patches
* 17:37 Reedy: Reloading Zuul to deploy 5 phan enabling patches
* 17:23 Reedy: Reloading Zuul to deploy various phan additions and one dependancy
* 17:08 Reedy: Reloading Zuul to deploy patches adding phab
* 16:46 Reedy: Reloading Zuul to deploy various dependancy patches
== 2019-05-09 ==
* 08:17 elukey: remove mediawiki memcached nutcracker config from deployment-prep (should be unused) - [[phab:T214275|T214275]]
* 00:19 thcipriani: updating docker images on contint1001 for https://gerrit.wikimedia.org/r/508929
* 00:19 thcipriani: clean docker images on contint1001
== 2019-05-07 ==
* 22:23 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/507871
* 10:57 hashar: Upgraded Zuul to 2.5.1-wmf8 # [[phab:T105474|T105474]]  [[phab:T140297|T140297]]
== 2019-05-06 ==
* 21:57 thcipriani: update docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508086
* 19:07 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/508012
* 18:37 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508370
* 17:44 marxarelli: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/508036
* 13:31 hashar: Jenkins: installed Least Load plugin {{!}} [[phab:T218458|T218458]]
== 2019-05-03 ==
* 20:58 thcipriani: updating docker-pkg on contint1001 for https://gerrit.wikimedia.org/r/508006
* 20:58 thcipriani: clean docker images from contint1001
* 17:02 thcipriani: reloading zuul to deploy  https://gerrit.wikimedia.org/r/507992
* 09:22 hashar: removed zuul debian package from integration-castor03
* 08:30 hashar: Building Docker image releng/gradle:0.1.0  {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/507872/ {{!}} [[phab:T222199|T222199]]
== 2019-05-01 ==
* 17:53 halfak: deploying ores:52e9759
== 2019-04-30 ==
* 18:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/494778
* 17:01 bearND: (beta): Update mobileapps to {{Gerrit|142ba30}}
* 02:01 hashar: Polled in integration-slave-docker-1055 and ntegration-slave-docker-1056
* 01:35 hashar: Deleting integration-slave-docker-1037 (bigram)  it is too slow for some reason # [[phab:T222023|T222023]]
== 2019-04-29 ==
* 14:04 godog: add dsharpe user
== 2019-04-26 ==
* 13:33 Krenair: shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. [[phab:T218729|T218729]]
== 2019-04-25 ==
* 16:35 Krenair: shutting down deployment-ms-fe02 and deployment-poolcounter04 [[phab:T218729|T218729]]
== 2019-04-23 ==
* 15:33 paladox: merging the 2.15.13 release into stable-2.15 following https://wikitech.wikimedia.org/wiki/Gerrit#Update_our_repository
* 14:41 Krenair: Shut down deployment-ms-be03 and deployment-ms-be04 [[phab:T218729|T218729]]
* 10:22 Amir1: ores:060fc37 going beta
== 2019-04-19 ==
* 19:56 mutante: phab1003 - editing /srv/deployment/phabricator/deployment-cache/.config manually to replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet to fix git cloning issue on first puppet run on new host where somehow tin.eqiad still shows up. fixes puppet run on [[phab:T221389|T221389]]
== 2019-04-18 ==
* 19:25 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/504824
== 2019-04-17 ==
* 17:17 andrewbogott: cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504580/ to move off of soon-to-be-shutdown dns recursors
== 2019-04-16 ==
* 17:36 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf {{!}} tee [[phab:T221107|T221107]].php
== 2019-04-15 ==
* 10:16 Amir1: ores:8f01d40 going beta
* 08:51 hashar: castor: nuked /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwselenium-quibble-docker  # [[phab:T220948|T220948]]
== 2019-04-13 ==
* 21:05 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. [[phab:T91410|T91410]]
* 21:00 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul.
* 21:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503669
* 18:52 Krinkle: "Your JENKINS_HOME (/var/lib/jenkins) is almost full. "
* 18:16 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/503664
* 17:44 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/502807 (postgres php72)
* 00:06 Krenair: transferred /home from deployment-cache-upload04 to deployment-cache-upload05 and shut down old one
* 00:06 Krenair: transferred /home from deployment-cumin to deployment-cumin02 and shut down old one
== 2019-04-12 ==
* 15:09 Krenair: upload traffic now through cache-upload05
== 2019-04-11 ==
* 18:29 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/480463
* 18:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/499334
* 17:50 Reedy: deployed 'quibble-donationinterface-REL1_31-php70-docker' to jenkins
* 16:31 Krenair: deleting deployment-db04, unused and shut down since 28th March. [[phab:T219087|T219087]]
* 16:18 mateusbs17: deployment-chromium0[1-2] Update Proton on to {{Gerrit|8988283}} ([[phab:T213362|T213362]], [[phab:T216191|T216191]], [[phab:T212322|T212322]])
== 2019-04-10 ==
* 23:32 James_F: Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. [[phab:T220653|T220653]]
* 22:36 James_F: Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches.
* 20:27 paladox: create operations/software/gerrit/plugins/MassBranchCreation repository
* 14:16 hashar: contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 14:13 hashar: contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin  # [[phab:T220606|T220606]]
* 13:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/502790
* 12:46 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502786
* 12:40 hashar: contint2001: stopped puppet and zuul-merger for debugging
== 2019-04-09 ==
* 23:08 Krinkle: Reloading Zuul to deploy https://phabricator.wikimedia.org/T220561
* 17:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502560
* 16:38 Krenair: deleting deployment-db03, unused since [[phab:T216635|T216635]] and shut down since 24th March. [[phab:T219087|T219087]]
* 16:21 bearND: (beta): Update mobileapps to {{Gerrit|3edfcad}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]]) - 3rd time is the charm
== 2019-04-08 ==
* 22:46 hashar: cleaned docker images on  integration-slave-docker-1030 and  integration-slave-docker-1043 :)
* 20:06 bearND: (beta): Update mobileapps to {{Gerrit|cdb9928}} ([[phab:T220045|T220045]] [[phab:T219411|T219411]] [[phab:T219667|T219667]])
* 14:47 hashar: hard rebooting integration-slave-docker-1053 OOM / deadlocked
* 13:39 hashar: Deleting integration-slave-docker-1045 again. It uses Stretch instead of Jessie
* 13:21 hashar: integration: fix cumin profiles that got renamed by {{Gerrit|9e0aa8264659799e74c3e815ea39c640e5f05393}}
* 10:15 hashar: Building Docker image  releng/quibble-stretch-php73:0.0.31-2 # [[phab:T220237|T220237]]
== 2019-04-07 ==
* 00:30 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501793
== 2019-04-06 ==
* 02:17 legoktm: rebuilding npm-php image again https://gerrit.wikimedia.org/r/501848
* 01:52 legoktm: rebuilding npm-php image https://gerrit.wikimedia.org/r/501847
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/501786 https://gerrit.wikimedia.org/r/501714 https://gerrit.wikimedia.org/r/501707 https://gerrit.wikimedia.org/r/501782 https://gerrit.wikimedia.org/r/501709 https://gerrit.wikimedia.org/r/500111 https://gerrit.wikimedia.org/r/500106 https://gerrit.wikimedia.org/r/500127 https://gerrit.wikimedia.org/r/500119
* 00:53 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/501813
== 2019-04-05 ==
* 23:53 Krinkle: Beta cluster puppetmaster is stalled behind origin/production as of 24 hours ago (57 patches behind) due to a local merge conflict
* 22:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501789
* 22:02 legoktm: rebuilding mediawiki-phan docker image https://gerrit.wikimedia.org/r/501794
* 20:11 thcipriani: updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/501465
* 08:56 hashar: Reloaded Zuul for operations/software/gerrit/plugins/barricade https://gerrit.wikimedia.org/r/#/c/integration/config/+/501507/
* 08:21 legoktm: deploying https://gerrit.wikimedia.org/r/501416
== 2019-04-04 ==
* 23:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501442
* 22:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/500360
* 21:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501428
* 20:47 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/501407
* 17:17 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/500719 (
== 2019-04-03 ==
* 06:13 hashar: gerrit: renamed group "scholarships" to "wikimedia-wikimania-scholarships". Made it owned by "Gerrit Managers" # [[phab:T218864|T218864]]
== 2019-04-02 ==
* 22:24 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/PageAssessments/maintenance/purgeUnusedProjects.php --wiki=enwikivoyage {{!}} [[phab:T219935|T219935]]
* 09:46 hashar: Upgrading CI Quibble jobs to 0.0.31
== 2019-04-01 ==
* 21:42 hauskatze: Imported tool-ldap from Diffusion to Gerrit with full history {{!}} [[phab:T219703|T219703]]
* 21:33 hauskatze: Created https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/ldap {{!}} [[phab:T219703|T219703]]
* 20:53 hashar: ssh contint1001.wikimedia.org sudo rm /tmp/docker-pkg-build.log
* 20:44 hashar: Building Quibble 0.0.31 containers again # [[phab:T219647|T219647]]  [[phab:T219786|T219786]]
* 20:10 hashar: gerrit: flush-caches --cache git_tags  # some tag got stalled when querying over https -  [[phab:T219786|T219786]]
* 18:50 hauskatze: Created mediawiki/extensions/ContributionCredits.git per request on mediawiki.org
* 18:00 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/#/c/integration/config/+/500504/
* 17:25 Krinkle: fresnel-node10-browser-docker failing with ENOMEM. Depooled integration-slave-docker-1049 as precaution.
* 17:04 hashar: Building CI docker images for Quibble 0.0.31 (yes it is a long day...)
* 13:50 hashar: Reverted CI Jenkins jobs to Quibble 0.0.28 # [[phab:T219647|T219647]]
* 13:11 hashar: Upgraded CI Jenkins jobs to Quibble 0.0.30 # [[phab:T219647|T219647]]
* 10:50 hashar: Manually triggering postmerge step of citoid due to [[phab:T219017|T219017]] for mvolz. On contint1001: zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/citoid --change 497315,1
* 08:08 hashar: Rebuilding Quibble Jessie containers that failed to build last week due to wikimedia-jessie container. # [[phab:T219647|T219647]]
* 08:07 hashar: Rebuilding container docker-registry.wikimedia.org/wikimedia-jessie # [[phab:T219683|T219683]]
== 2019-03-29 ==
* 19:12 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/497975/ + https://gerrit.wikimedia.org/r/498887/
* 17:30 hashar: some quibble 0.0.30 images fail to build for an unknown reason . Left to figure out after the week-end has passed..
* 17:22 hashar: Building Ci docker images for quibble 0.0.30 [[phab:T219647|T219647]]  [[phab:T219645|T219645]]
* 12:03 Krenair: added alaasarhan to deployment-prep [[phab:T219621|T219621]]
== 2019-03-28 ==
* 18:49 Krenair: shut off deployment-db04 instance per [[phab:T219087|T219087]]
* 18:20 thcipriani: reload zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/
* 17:36 paladox: forking plugins/quota from upstream
* 17:24 Krenair: deployment-prep [[phab:T219087|T219087]] beginning master switch
* 10:45 hashar: Tagged Quibble 0.0.30 {{Gerrit|6ddc6d508cb554e6443ff72648da3ea8a3253fff}}
* 08:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/499539 (no-op) and https://gerrit.wikimedia.org/r/499717 (SecurePoll phan)
== 2019-03-27 ==
* 22:27 James_F: Altered Wikimedia GitHub settings to require 2FA; see [[phab:T198810|T198810]]
* 14:16 hashar: contint: refreshed all git caches manually from cumin:  cumin --force 'name:docker' 'find /srv/git -name '*.git' -type d -print -exec git -C {} fetch --prune \;'
* 14:16 hashar: contint: added repositories to the git caches https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499482/
== 2019-03-26 ==
* 23:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499344 (part 1, 2 and 3)
* 20:36 paladox: create gerrit repo operations/software/gerrit/plugins/WikimediaBlocks [[phab:T219300|T219300]]
* 20:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499312
* 17:43 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/499262
* 17:42 Krinkle: Reloading Zuul to deploy merged-but-not-deployed patches https://gerrit.wikimedia.org/r/#/c/integration/config/+/497800/ and https://gerrit.wikimedia.org/r/#/c/498684/
== 2019-03-25 ==
* 14:46 mateusbs17: sunset deployment-maps03
== 2019-03-24 ==
* 16:06 Krenair: shut off old deployment-db03 instance per [[phab:T219087|T219087]]
* 05:39 Krenair: cleaned up old puppet certs/nodes -certcentral-testclient03 -certcentral-testdns -certcentral03 -zotero01 -eventgate-analytics -t153468-test -rd3-cptest-master01 -maps05
* 04:12 Krenair: removed php7.0-fpm package (conflicting with php7.2-fpm) and removed /etc/nginx/sites-enabled/default (conflicting with apache, puppet will remove the available copy too) from -deploy02, -jobrunner03, -mwmaint01, and -mediawiki-07 hosts to try to get puppet there happy again
== 2019-03-23 ==
* 23:44 legoktm: deploying https://gerrit.wikimedia.org/r/498681 https://gerrit.wikimedia.org/r/498682 https://gerrit.wikimedia.org/r/498683
* 23:33 legoktm: deploying https://gerrit.wikimedia.org/r/498247 https://gerrit.wikimedia.org/r/498249 https://gerrit.wikimedia.org/r/498680
* 20:52 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/498664 / [[phab:T215562|T215562]]
== 2019-03-22 ==
* 19:59 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/498466/ # [[phab:T219017|T219017]]
* 19:34 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/494548/
* 19:24 thcipriani: clean old docker-pkg image from contint1001
* 03:08 legoktm: rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T218963|T218963]]
* 00:36 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/498276  / [[phab:T215562|T215562]])
== 2019-03-21 ==
* 23:34 legoktm: rebuilding mediawiki-phan docker image for https://gerrit.wikimedia.org/r/498266
* 22:34 legoktm: onlined integration-slave-jessie-1002
* 22:33 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 21:03 legoktm: deploying https://gerrit.wikimedia.org/r/498178 https://gerrit.wikimedia.org/r/498185 https://gerrit.wikimedia.org/r/498187 https://gerrit.wikimedia.org/r/498189 https://gerrit.wikimedia.org/r/498182
* 15:22 <hashar>: pruning images/containers on integration-slave-docker-1021
* 10:13 <hashar>: deployment-deploy01: sudo rm -fR /tmp/mw-cache-master | files were from Mar 15 10:21
* 00:33 <legoktm>: deploying https://gerrit.wikimedia.org/r/497600
== 2019-03-20 ==
* 13:20 <hashar>: Scheduled update of Diffusion repository wikibase-termbox via https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ on request of Pablo_WMDE
== 2019-03-19 ==
* 22:47 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497322 / T218553
* 21:00 <paladox>: hiding operations/software/gerrit/plugins/WikimediaWebSessions from normal users
* 20:59 <paladox>: create operations/software/gerrit/plugins/WikimediaWebSessions project T218739
* 20:23 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/preferences.d/jessie_mitaka_pinning_*' # T218559
* 19:56 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list' # T218559
* 16:56 <Reedy>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/494802
* 15:48 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497437
* 14:01 <hashar>: Removed Zuul "check" pipeline | https://gerrit.wikimedia.org/r/#/c/integration/config/+/493188/ | T192217
* 02:21 <Krenair>: sudo aptly publish --architectures="all,amd64" --skip-signing repo buster-deployment-prep
* 02:09 <Krenair>: sudo aptly repo create -component="main" -distribution="buster-deployment-prep" buster-deployment-prep
== 2019-03-18 ==
* 16:39 Krenair: yet*
* 16:38 Krenair: created deployment-acme-chief01 and a client instance for further acme-chief testing + dev. used stretch, would be buster like prod but not sure that's easily available outside testlabs yes
* 10:09 hashar: contint1001: rm -fR /srv/doc1001.eqiad.wmnet
* 10:06 hashar: github: deleting https://github.com/wikimedia/wikidata-gremlin # archived [[phab:T155829|T155829]]
* 09:58 hashar: arming keyholder on integration-cumin
* 09:55 hashar: deleting shutdowned instance integration-publisher02 , we do not use it anymore since doc publishing got overhauled ( [[phab:T137890|T137890]] )  # [[phab:T218146|T218146]]
* 09:12 hashar: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
* 07:37 legoktm: deployed https://gerrit.wikimedia.org/r/496610
== 2019-03-17 ==
* 16:29 Krenair: deactivated and cleaned certs for deployment-redis3-changeprop02 and deployment-prometheus01 (which no longer appear to exist but were causing cumin to be upset)
* 16:27 Krenair: armed deployment-cumin keyholder
* 00:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496887
== 2019-03-16 ==
* 21:56 Krinkle: krinkle@contint1001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/OOJsUIAjaxLogin --change 490979,1
* 21:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497077
* 21:41 legoktm: deploying https://gerrit.wikimedia.org/r/496507 https://gerrit.wikimedia.org/r/496524 https://gerrit.wikimedia.org/r/496520 https://gerrit.wikimedia.org/r/496516 https://gerrit.wikimedia.org/r/496510
* 21:21 Krinkle: krinkle@contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/EventLogging --change 264494,4
* 21:20 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/497076
* 21:20 Krinkle: Removing doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-extensions-EventLogging (created by accident)
== 2019-03-15 ==
* 21:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496978
* 21:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496882
* 20:10 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/496688
* 18:21 thcipriani: clean old docker images from contint1001
* 17:40 thcipriani: rearm beta keyholder
* 16:54 thcipriani: reenable beta-scap-eqiad
* 16:08 thcipriani: disable beta-scap-eqiad to test new php, back shortly
== 2019-03-14 ==
* 23:48 Krinkle: Abort job quibble-vendor-mysql-hhvm-docker/39874/ for mwext-CentralAuth (stuck after 59 minutes)
* 21:25 hashar: Manually triggered tests for 12 ContentTranslation changes that had label:verified=-1 # [[phab:T216689|T216689]]
* 21:21 hashar: Updated quibble-vendor-mysql-hhvm-docker with latest libc6 hopefully fixing HHVM segfault within libpthread # [[phab:T216689|T216689]]
* 19:14 ebernhardson: restart logstash on deployment-logstash2 to re-read and re-create apifeatureusage template
* 16:34 hashar: rollback quibble-vendor-mysql-hhvm-docker job to no more capture core files, we have enough and a good lead ( reverting https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ ) # [[phab:T216689|T216689]]
* 12:31 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/4 # [[phab:T216689|T216689]]
* 12:12 hashar: Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/3 # [[phab:T216689|T216689]]
* 10:49 hashar: triggering tests for all ContentTranslation pending changes # [[phab:T216689|T216689]]
* 09:54 hashar: ci: live hacked job https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/ in attempt to capture 'core' files from hhvm {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ {{!}} [[phab:T216689|T216689]]
== 2019-03-13 ==
* 21:26 thcipriani: pool new bigram CI instance integration-slave-docker-1054
* 20:58 thcipriani: deleting bigram CI instance integration-slave-docker-1046 due to corrupt disk cf: [[phab:T218245|T218245]]
* 20:44 thcipriani: marking integration-slave-docker-1046 offline
* 20:38 hashar: Added integration-slave-docker-1045 to Jenkins. The instance existed in WMCS but was not in Jenkins
* 20:13 bearND: (beta): Update mobileapps to {{Gerrit|5865552}} ({{Gerrit|7074964}} {{Gerrit|d6dc3cd}} {{Gerrit|fbc6262}})
* 19:13 hashar: integration-slave-docker-1046 is back online # [[phab:T218245|T218245]]
* 19:04 hashar: hard rebooting integration-slave-docker-1046 , not reachable over ssh # [[phab:T218245|T218245]]
* 18:48 hashar: Building containers releng/quibble-jessie-hhvm and releng/quibble-stretch-hhvm with HHVM core_dump_report enabled # [[phab:T216689|T216689]]
* 18:09 ebernhardson: restart elasticsearch on deployment-elastic* to deploy apifeature usage fix ([[phab:T183156|T183156]])
* 10:39 hashar: Triggered tests for ContentTranslation changes that had label:verified=-1 # [[phab:T217654|T217654]]
* 10:03 hashar: Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # [[phab:T217654|T217654]]
== 2019-03-12 ==
* 18:38 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/479558  / [[phab:T203506|T203506]]
* 18:23 thcipriani: bring integration-slave-jessie-1001 back online, /srv disk space now at 20% (not sure if someone cleared disk and forgot to repool)
* 16:45 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/495931
* 13:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/493139/
== 2019-03-11 ==
* 15:59 Krenair: previous !log was for deployment-prep
* 15:56 Krenair: added MSantos to projectadmin, per chat in -infrastructure he'll use it to work on maps. user is a new foundation software engineer
* 01:20 legoktm: deploying https://gerrit.wikimedia.org/r/495593
== 2019-03-10 ==
* 02:50 legoktm: deploying https://gerrit.wikimedia.org/r/495443 https://gerrit.wikimedia.org/r/493741
* 02:45 legoktm: deploying https://gerrit.wikimedia.org/r/495332 https://gerrit.wikimedia.org/r/495470
== 2019-03-08 ==
* 22:18 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490950/
* 20:20 legoktm: deploying https://gerrit.wikimedia.org/r/495302
* 20:16 legoktm: deploying https://gerrit.wikimedia.org/r/495300 https://gerrit.wikimedia.org/r/495298 https://gerrit.wikimedia.org/r/493502
* 11:36 hasharLunch: integration: deleting old java8 docker images:  sudo cumin --force 'name:docker' 'docker images{{!}}grep java8{{!}}awk "{ print $3 }"{{!}}xargs docker rmi'
* 10:04 hashar: Updating maven based jobs to latest java8 containers {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/495022/ {{!}} [[phab:T208938|T208938]]
== 2019-03-06 ==
* 18:25 hasharAway: contint1001: restart Jenkins for plugins upgrade
* 18:24 hasharAway: reloading zuul for {{Gerrit|I21d7fa2939f507441f7d130f8207541e28d762ad}}
* 18:02 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org
* 17:37 hashar: Reloading Zuul for  mediawiki/libs/Zest https://gerrit.wikimedia.org/r/#/c/integration/config/+/494609/2/zuul/layout.yaml
* 16:27 Krinkle: doc1001 had bad permissions set on /org/wikimedia/doc (chmod 755 instead of 775). Making it impossible to git pull in the way that post-merge Jenkins job on integration/docroot recommends. Fixed with `sudo -u doc-uploader chmod 775 /srv/wikimedia/org/wikimedia/doc`.
* 15:58 andrewbogott: deleting deployment-prometheus01 on Filippo's advice
* 13:31 gehel: upgrading logstash to 5.6.14 on deployment-logstash2
* 13:15 gehel: upgrading elasticsearch to 5.6.14 on deployment-logstash2
* 11:33 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mediawiki-09:~$ sudo systemctl restart php7.2-fpm # [[phab:T217323|T217323]]
== 2019-03-05 ==
* 20:10 thcipriani: reenable beta-scap-eqiad
* 19:17 thcipriani: disable beta-scap-eqiad due to [[phab:T217587|T217587]]
* 10:13 hashar: integration: fixed erroneous ssh key restriction for cumin {{!}} [[phab:T217642|T217642]]
== 2019-03-04 ==
* 17:36 hashar: Build Docker containers for chromium=v71 pin ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/494243/ )
* 15:14 herron: cherry picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/492390/ to deployment-puppetmaster03
* 13:49 hauskatze: GitHub: deleted wikimedia/mediawiki-extensions-UploadLocal {{!}} [[phab:T213011|T213011]]
* 09:59 hashar: cleaned docker images on integration-slave-docker-1037
* 07:38 legoktm: deploying https://gerrit.wikimedia.org/r/494167
== 2019-03-03 ==
* 02:26 Krinkle: tried rebooting or shutting down integration-slave-docker-1021, no response on horizon. Did pause/resume instead, which did work, after which shutdown/start worked. Jenkins agent has been relaunched and seems online again.
* 02:20 Krinkle: integration-slave-docker-1021 (ci1.medium) has jobs failing on it due to ENOMEM. Horizon shows in log: integration-slave-docker-1021 login: [4961938.696837] Out of memory: Kill process 21770 (chromium) score 841 or sacrifice child; [4961938.699176] Killed process 21770 (chromium) total-vm:3171496kB, anon-rss:1379288kB, file-rss:0kB, shmem-rss:1636kB
== 2019-03-02 ==
* 22:12 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/493959
* 20:44 hauskatze: Renamed https://github.com/wikimedia/wikimedia-github-community-health-defaults to https://github.com/wikimedia/.github
* 20:42 hauskatze: ssh -p 29418 gerrit.wikimedia.org replication start wikimedia/github-community-health-defaults --wait
* 20:40 hauskatze: github created https://github.com/wikimedia/wikimedia-github-community-health-defaults
* 20:31 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/493881
* 20:30 Krinkle: Failure on integration-slave-docker-1021 (ENOMEM) https://integration.wikimedia.org/ci/job/fresnel-node10-browser-docker/61/console
* 19:51 legoktm: deploying https://gerrit.wikimedia.org/r/493872
* 19:37 legoktm: deploying https://gerrit.wikimedia.org/r/493862
* 18:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493808
* 18:21 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493837
== 2019-03-01 ==
* 19:17 thcipriani: integration-slave-docker-1021:/# docker rmi $(docker images {{!}} grep " months " {{!}}grep -v " [1-2] months " {{!}} awk '{print $3}')
* 17:02 thcipriani: integration-slave-jessie-1004 back online
* 16:58 thcipriani: integration-slave-jessie-1002 back online (disk space looked fine); rebooting integration-slave-jessie-1004 -- can't ssh to machine
* 16:11 Lucas_WMDE: delete refs/master and refs/gerrit/master on WikibaseQualityConstraints repository [[phab:T217408|T217408]]
* 15:49 hashar: wikidata/query/blazegraph  change Gerrit config to require a change-id # [[phab:T216855|T216855]]
* 14:28 hashar: Upgrading integration/jenkins-job-builder to version 2.0.2  + one custom hack 11aa5de4...a06d173e  # [[phab:T143731|T143731]]
* 14:18 hashar: integration/jenkins-job-builder : importing upstream code to new branch "upstream".  Push all upstream tags to our repository
== 2019-02-28 ==
* 18:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/493261
* 03:42 duh: deploying https://gerrit.wikimedia.org/r/493357
* 03:42 duh: deploying
* 03:30 duh: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/493355
== 2019-02-27 ==
* 22:02 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/
* 19:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/492758
* 19:08 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758 (take II)
* 18:27 thcipriani: updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758
== 2019-02-26 ==
* 22:52 ebernhardson: delete logstash logs in /var/log/logstash generated prior to 2019
* 22:51 ebernhardson: restart logstash on deployment-logstash2 while hacking around to see why apifeatureusage doesn't work
* 18:01 dcausse: deployement-prep: installing elastic 6.5.4 to deployment-elastic* machines
* 16:46 addshore: added Cparle to deployment-prep
* 16:11 hashar: Generating 1.33.0-wmf.19 deploy notes https://integration.wikimedia.org/ci/job/train-deploy-notes/9/console {{!}} [[phab:T206673|T206673]]
* 09:32 godog: remove now-merged node-exporter timer disable, cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/492632
== 2019-02-25 ==
* 23:32 twentyafterfour: root@deployment-db05# mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata # [[phab:T216067|T216067]]
* 22:14 thcipriani: docker rmi images without "latest" tag on contint1001 to free space -- should have kept all current docker-pkg images as well as images with children -- [[phab:T217094|T217094]]
* 13:39 hashar: Rebuilding some CI Docker images using PHP sury.org to switch the sury.org component from jessie to stretch ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/492666/ )
== 2019-02-24 ==
* 21:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/492561, [[phab:T216964|T216964]]
* 04:54 legoktm: rebuilding docker image for https://gerrit.wikimedia.org/r/485241
* 04:41 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/tools$ sudo -u zuul rm -rf phan
== 2019-02-23 ==
* 22:25 legoktm: deploying https://gerrit.wikimedia.org/r/492497
* 02:32 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/492427
* 01:59 Krinkle: Reloading Zuul to deploy ttps://gerrit.wikimedia.org/r/492425
== 2019-02-22 ==
* 20:59 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/492377
* 16:48 paladox: created "block-users" group for https://bugs.chromium.org/p/gerrit/issues/detail?id=10507 (reported by a wmf user)
* 13:38 Reedy: restarting zuul because nothing is being run
== 2019-02-21 ==
* 21:02 paladox: branch deploy/wmf/stable-2.16 from {{Gerrit|0e0ea0ff735da0a494347884917fc48881d7e545}} in operations/software/gerrit
* 19:49 Amir1: ores:5d937b1 is going beta
* 18:22 Amir1: ores:2d84709 going beta
* 17:45 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/483225/
* 16:35 hashar: Adjusting IP of https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ and adding it back # [[phab:T216513|T216513]]
* 14:10 Amir1: deploying ores:5d50713 to beta
== 2019-02-20 ==
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status  # [[phab:T216635|T216635]]
* 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz {{!}} sudo tee /var/lib/dpkg/status
* 17:26 hashar: For beta cluster the MySQL master database has some innodb issue [[phab:T216635|T216635]] , the MySQL slave has an issue as well [[phab:T216067|T216067]]
* 17:09 hashar: reloading zuul for {{Gerrit|Id1e3afd0afba9b388778066b9b6e8e564a25826b}}
* 17:09 hashar: contint1001: fix broken root ownership on zuul git deploy repo: sudo find /etc/zuul/wikimedia/.git -not -user zuul -exec chown zuul:zuul {} +
* 17:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490606/
* 16:57 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490559
* 16:36 hauskatze: Ran replication start mediawiki/extensions/PageViewInfo --wait on gerrit.wikimedia to populate GitHub mirror (success messages afterwards) {{!}} [[phab:T180864|T180864]]
* 00:25 greg-g: disabled beta-update-databases-eqiad in the jenkins UI - [[phab:T216067|T216067]]
== 2019-02-19 ==
* 21:23 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/491577, also deploys  https://gerrit.wikimedia.org/r/490492  / [[phab:T198495|T198495]]
* 20:28 hashar: jenkins: disable Android Emulator plugin {{!}} [[phab:T198495|T198495]]
* 20:27 hashar: gerrit: archived repository integration/jenkinsci/android-emulator-plugin {{!}} [[phab:T198495|T198495]]
* 20:23 hashar: Deleting Jenkins job apps-android-wikipedia-periodic-test {{!}} [[phab:T198495|T198495]]
* 15:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491513
* 15:46 hashar: Creating integration-slave-jessie-android m1.large it got deleted last week {{!}} [[phab:T216517|T216517]]
* 08:55 hashar: Cleaning contint1001 / partition
* 04:55 bd808: Removed stale cherry-pick of {{Gerrit|Ic7e726768701fefdee68622b08e3f2995779fe5a}} from deployment-puppetmaster03 that was blocking rebase on origin/production
* 04:50 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491405
* 04:14 Krinkle: Disabled spam account on Phab - https://phabricator.wikimedia.org/people/manage/18915/
* 03:55 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491403
* 00:54 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491381  / {{Gerrit|bc8e4198961cb73}} /  [[phab:T133646|T133646]]
== 2019-02-18 ==
* 22:32 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491379  / [[phab:T211784|T211784]]
* 21:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/491378 / [[phab:T133646|T133646]])
* 21:37 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/459268 / [[phab:T133646|T133646]]
* 15:38 elukey: kill/spawn deployment-aqs0[2,3] in deployment-prep with Debian Stretch
* 14:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490590
* 12:00 Krenair: [[phab:T216067|T216067]] Stopping mysql on -db04 to begin copy to -db05. Note crashed tables centralauth.globaluser and centralauth.localuser
* 11:57 elukey: kill/spawn deployment-aqs01 with Debian Stretch in deployment-prep
* 11:45 arturo: manually start deployment-db03 per Krenair request
* 11:29 hasharAway: beta: tried to start instance deployment-db03 172.16.5.23 --> ERROR  {{!}} [[phab:T216067|T216067]]
== 2019-02-17 ==
* 07:21 legoktm: deploying https://gerrit.wikimedia.org/r/491029
* 07:10 legoktm: Building image docker-registry.discovery.wmnet/releng/tox-acme-chief:0.3.4
* 06:28 legoktm: building new tox-acme-chief docker image https://gerrit.wikimedia.org/r/489725
* 01:12 Krinkle: beta-scap-eqiad (cron) failing with "sudo: a password is required"
== 2019-02-16 ==
* 19:44 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490937 / [[phab:T216275|T216275]])
* 17:23 thcipriani: installed php7.0-curl on deployment-deploy01 (why was that suddenly necessary?)
== 2019-02-15 ==
* 17:28 thcipriani: integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
== 2019-02-14 ==
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T216102|T216102]]
* 20:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528  / [[phab:T133646|T133646]]
* 13:24 thcipriani: rearm keyholder on deployment-deploy01: sudo keyholder arm, passwords on https://wikitech.wikimedia.org/wiki/Keyholder
== 2019-02-13 ==
* 21:32 marxarelli: dduvall@integration-slave-jessie-1001:/mnt/home/jenkins-deploy$ `rm -rf .gradle/ .m2/` due to full disk
* 21:21 marxarelli: bringing integration-slave-docker-1046 and integration-slave-jessie-1001 back online
* 21:20 marxarelli: dduvall@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
* 21:15 marxarelli: removing old docker images on integration-slave-docker-1046
* 21:10 marxarelli: starting migrated integration-slave-docker-1046 instance
* 21:01 marxarelli: pooling new jenkins node for integration-slave-docker-1052
* 20:46 marxarelli: pooling jenkins node for integration-slave-docker-1051
* 20:45 marxarelli: launching replacement instance integration-slave-docker-1052
* 20:35 marxarelli: launching replacement instance integration-slave-docker-1051
* 20:32 marxarelli: pooling jenkins node for integration-slave-docker-1050
* {{safesubst:SAL entry|1=20:15 marxarelli: integration-slave-docker-{1044,1046,1047} unresponsiveness due to cloudvirt failure. 1046 is being moved already by CS. deleting 1044 and 1047}}
* {{safesubst:SAL entry|1=19:57 marxarelli: seeing jenkins agent connection failures for integration-slave-docker-{1044,1046,1047}}}
* 19:48 marxarelli: pooling replacement jenkins node integration-slave-docker-1049
* 19:34 marxarelli: deleting integration-slave-jessie-android jenkins node and instance
* 19:33 marxarelli: deleting integration-slave-jessie-1003 jenkins node and instance
* 19:32 marxarelli: deleting integration-slave-docker-1033 jenkins node and instance
* 19:25 marxarelli: deleting integration-slave-docker-1017 jenkins node and instance
* 18:45 Krinkle: integration-slave-jessie-1003 seems to be consitently unable to start jobs, marking as offline manually
* 18:32 thcipriani: bringing up new integration-castor03, re-enabling castor-save* jobs
* 18:15 marxarelli: adding new jenkins node integration-slave-docker-1048
* 18:02 marxarelli: launching new integration-slave-docker-1048 instance
* 17:59 marxarelli: deleting integration-slave-docker-1038 node and deleting instance
* 17:50 marxarelli: bringing integration-slave-docker-1033 back online after clearing out old docker images
* 17:33 thcipriani: rebuilding integration-castor03
* 17:21 thcipriani: stopping rsync server on castor03
* 17:21 twentyafterfour: stopped rsync on castor03
* 17:16 twentyafterfour: disconnected castor03 from jenkins
* 16:48 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/487880/
* 14:34 thcipriani: modified castor-save-workspace-cache to exit 0 and run on blubber nodes while integration-castor03 is down
* 14:26 dcausse: deployement-prep: upgrading to elastic 5.6.14
== 2019-02-12 ==
* 23:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490130/1
== 2019-02-11 ==
* 21:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/489689
* 20:39 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/489688/
* 19:46 marxarelli: installing/enabling HTTP Request jenkins plugin on integration.wikimedia.org/ci to support https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/480689 changes
* 16:04 addshore: bring integration-slave-docker-1040 back online
* 16:04 addshore: addshore@integration-slave-docker-1040:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.52GB
* 10:47 godog: shut deployment-prometheus01, unused now
* 03:06 Reedy: graceful restart of zuul as no jobs were running
== 2019-02-10 ==
* 03:22 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489434  (Create quibble-stretch-hhvm, replacing jessie)
* 02:06 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/489430
== 2019-02-08 ==
* 20:20 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config, ref [[phab:T91410|T91410]]
* 15:59 addshore: this reload also included "Switch npm-audit job to node10"? [[phab:T211784|T211784]], which did touch the zuul file
* 15:58 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/489241/
* 03:10 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config
* 00:28 Krinkle: krinkle@doc1001: sudo -u doc-uploader chmod 775 /srv/docroot/org/wikimedia/doc/
* 00:12 marxarelli: removed old docker images on contint1001 to free up space
== 2019-02-07 ==
* 23:17 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
* 23:15 thcipriani: integration-slave-docker-1033:sudo docker image prune and bring back online
* 22:28 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/467550 (
* 19:09 paladox: created integration/zuul/build gerrit repo for [[phab:T215458|T215458]]
* 19:05 paladox: created integration/zuul/wheels gerrit repo for [[phab:T215458|T215458]]
* 15:48 addshore: brought integration-slave-docker-1043 back online
* 15:48 addshore: addshore@integration-slave-docker-1043:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 14.86GB
* 08:49 hashar: cleaning docker images on integration-slave-docker-1021
== 2019-02-06 ==
* 22:34 shdubsh: Deploy node-exporter 0.17 [[phab:T213708|T213708]]
* 14:12 godog: shut off deployment-prometheus01 - [[phab:T215272|T215272]]
* 14:00 godog: switch beta-prometheus to deployment-prometheus02 - [[phab:T215272|T215272]]
== 2019-02-05 ==
* 20:07 ebernhardson: jobrunner port 9006 is firewalled, revert to 9005 and created [[phab:T215339|T215339]] to fix job queue in beta cluste
* 19:36 ebernhardson: Update profile::cpjobqueue::{jobrunner,videoscaler}_host in horizon hiera from port 9005 to 9006 to match new restrictions in gerrit.wikimedia.org/r/481866
* 16:29 addshore: [[phab:T215288|T215288]] added mirrys to deployment-prep as a user
* 15:32 addshore: [[phab:T215278|T215278]] addshore@integration-slave-docker-1037:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.59GB
== 2019-02-04 ==
* 23:13 thcipriani: integration-slave-docker-1040:sudo docker image prune and bring back online
* 23:12 thcipriani: integration-slave-docker-1038:sudo docker image prune and bring back online
* 21:48 ebernhardson: restart logstash on deployment-logstash2
* 15:25 hashar: removed Jenkins user "nodepoolmanager" as well as related authorizations {{!}} [[phab:T209361|T209361]]
== 2019-02-03 ==
* 06:15 legoktm: deployed https://gerrit.wikimedia.org/r/487627
* 05:44 legoktm: deployed https://gerrit.wikimedia.org/r/485967
* 04:36 legoktm: deploying https://gerrit.wikimedia.org/r/487534
== 2019-02-02 ==
* 22:17 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2019-01-31 ==
* 15:03 thcipriani: rearm keyholder on deployment-deploy01
* 12:05 arturo: VM instances deployment-deploy01,deployment-deploy02,deployment-fluorine02,deployment-kafka-jumbo-2,deployment-kafka-main-1,deployment-maps04,deployment-mcs01,deployment-mediawiki-09,deployment-memc04,deployment-ms-be03,deployment-ms-fe02,deployment-parsoid09,deployment-sca04,deployment-webperf12, were stopped briefly due to issue in hypervisor ([[phab:T215012|T215012]])
== 2019-01-30 ==
* 08:35 legoktm: deploying https://gerrit.wikimedia.org/r/486439 https://gerrit.wikimedia.org/r/481570 https://gerrit.wikimedia.org/r/481571
== 2019-01-29 ==
* 07:41 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * b/c full disk
== 2019-01-28 ==
* 16:33 hashar: contint1001: cleaning up disk space on /
* 13:07 addshore: bringing integration-slave-docker-1041 back online
* 13:07 addshore: addshore@integration-slave-docker-1041:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.12GB
* 09:37 Amir1: ores:ad160b0 is going beta
== 2019-01-27 ==
* 19:57 addshore: bringing integration-slave-docker-1034 back online
* 19:50 addshore: addshore@integration-slave-docker-1034:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.12GB
== 2019-01-26 ==
* 22:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486791
* 21:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/486737
== 2019-01-25 ==
* 19:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486503/
* 19:07 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/486501/
* 14:56 hashar: contint1001: systemctl stop zuul-merger && find /srv/zuul/git -name .git -type d -print -execdir git gc --prune=now \;
* 13:35 hashar: flake8 broken under python2.7 due to configparser==3.5.2  https://github.com/jaraco/configparser/issues/27 https://github.com/jaraco/configparser/issues/27
* 00:58 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* clean gradle cache, bring back online
== 2019-01-23 ==
* 22:15 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/486170/
== 2019-01-22 ==
* 23:46 thcipriani: Updating (more) tox docker image on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485971/
* 23:24 thcipriani: integration-slave-docker-1017 clean docker images and repool
* 21:06 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485895/
* 20:18 legoktm: rebuilding php, composer, phan images for apt security update https://gerrit.wikimedia.org/r/485883
* 19:59 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/485882/
* 19:28 legoktm: rebuilding ci support images for apt security upgrade https://gerrit.wikimedia.org/r/485874
* 19:09 legoktm: rebuilding ci-stretch/ci-jessie images https://gerrit.wikimedia.org/r/485873
* 19:08 legoktm: manually docker pulled docker-registry.wikimedia.org/wikimedia-jessie and docker-registry.wikimedia.org/wikimedia-stretch on contint1001
* 18:56 bearND: (beta) Update mobileapps to {{Gerrit|0aac268}} (fix pronunciation detection in mobile-sections [[phab:T214338|T214338]])
* 18:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/485870/ and https://gerrit.wikimedia.org/r/#/c/integration/config/+/485331/
* 14:31 addshore: reloaded zuul for REVERT: Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:26 addshore: reloaded zuul for Add ArticlePlaceholder to gated extensions [integration/config] - https://gerrit.wikimedia.org/r/485754
* 13:03 addshore: reload zuul for Add ArticlePlaceholder to Wikibase Tests [integration/config] - https://gerrit.wikimedia.org/r/485753
== 2019-01-21 ==
* 19:49 hashar: integration: update sudo rule for debian-glue to keep env variable EXTRAPACKAGES. Would let us get eatmydata included {{!}} [[phab:T214328|T214328]]
* 15:40 hashar: contint1001: removing all generated doc/cover from /srv/org/wikimedia/doc {{!}} [[phab:T137890|T137890]]
== 2019-01-18 ==
* 23:22 hashar: contint1001: sudo docker image prune  # Total reclaimed space: 3.592GB
* 23:00 Krinkle: Some docker builds on integration-slave-docker-1021 failing with ENOMEM
* 23:00 mutante: contint1001 - gzipping more files in /var/log/zuul/
* 22:57 mutante: contint1001 - moved zuul logs from 2018 and gzipped zuul logs from /var/log/zuul to /srv/logs/zuul to free disk space on /
* 22:39 mutante: contint1001 - apt-get clean - disk space low
* 22:31 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482527  / [[phab:T212602|T212602]]
== 2019-01-17 ==
* 19:34 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 08:39 legoktm: deploying composer docker image - https://gerrit.wikimedia.org/r/484853
== 2019-01-16 ==
* 21:11 bearND: (beta): Update mobileapps to {{Gerrit|258d76b}} page summary changes
== 2019-01-15 ==
* 09:00 hashar: Deleting Docker images on integration-slave-docker-1021
== 2019-01-14 ==
* 22:02 bearND: (beta): Update mobileapps to {{Gerrit|f2658de}}
* 21:47 mutante: deployment-mcs01 - sudo su deploy-service; cd /srv/deployment/mobileapps/deploy-cache/revs/1182b3b8f288df0221257b929ca43fb86862c2f8/scap ; touch log  (for debugging permission problem reported by bearND)
* 14:31 hashar: Nuked Castor cache for all *tox* jobs. Some might have cached binary wheels compiled against a lib that is no more existing (eg libmysqlclient.so.18 for mysql-python). Follow up the jessie -> stretch upgrade # [[phab:T191764|T191764]]
* 14:28 hashar: Deleted Castor cache for wikimedia-cz/tracker  mysql-python got cached as  a wheel but compiled against libmysqlclient.so.18. That fails with the new tox...:0.3.0 containers which uses mariadb / libmysqlclient.so compat symlink
== 2019-01-11 ==
* 20:48 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
* 19:29 legoktm: deploying https://gerrit.wikimedia.org/r/483832
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/483824
* 04:37 Krinkle: Updating docker-pkg files on contint1001 for  https://gerrit.wikimedia.org/r/483674
* 03:48 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483671
* 03:15 twentyafterfour: mediawiki-1.32.0 uploaded to releases.wikimedia.org (archives should be downloadable in a few minutes after puppet runs and the proxy's cache expires)
* 03:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483667 Switch unicodejs jobs to Node 10  - [[phab:T211784|T211784]]
* 02:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483666
* 02:01 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/483656
* 01:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/483644
* 00:12 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482755 - [[phab:T211784|T211784]]
== 2019-01-09 ==
* 19:59 legoktm: deploying https://gerrit.wikimedia.org/r/482766
* 18:05 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/483189 https://gerrit.wikimedia.org/r/#/c/integration/config/+/481947/
* 13:27 hasharAway: shutting down instance integration-publishing02 [172.16.4.5]  No more used {{!}} [[phab:T137890|T137890]]
* 13:26 hasharAway: deleting https://integration.wikimedia.org/ci/job/publish-on-contint1001/ no more used {{!}} [[phab:T137890|T137890]]
== 2019-01-08 ==
* 21:52 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/476600/
* 21:31 Hauskatze: github: @niedzielski updated @jdlrobson permission on Wikimedia from `read` to `admin`
* 21:30 Hauskatze: github:
* 20:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/482855/
* 19:53 mutante: deployment-prep adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 19:53 mutante: adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
* 14:25 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org/
* 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T 101131
* 05:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482752
* 02:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482751
== 2019-01-07 ==
* 20:29 Krinkle: Reloading Zuul to apply https://gerrit.wikimedia.org/r/482695
* 20:12 Krinkle: Delete job 'fresnel-npm6-browser-node-6-docker' and 'fresnel-node-10-docker'
* 20:08 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/482572
* 19:32 awight: [[phab:T212530|T212530]]: ORES revscoring 2.3.0
* 19:22 legoktm: building node10 docker images w/ https://gerrit.wikimedia.org/r/482573 this time
* 00:30 legoktm: deploying https://gerrit.wikimedia.org/r/482392 / create node10 mage
== 2019-01-06 ==
* 00:39 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482392
== 2019-01-03 ==
* 20:15 Hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/SparkPost
* 20:03 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-SparkPost
* 18:46 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests, part 2
* 18:32 bearND: (beta): Update mobileapps to {{Gerrit|f6ad0e5}} Set timeout for backend /page/html requests
* 16:44 thcipriani: remove old docker images and bring integration-slave-docker-1021 back online
== 2019-01-02 ==
* 10:19 hashar: updating all debian-glue jobs and creating new ones with hardcoded distributions (trusty, jessie, stretch, unstable) [[phab:T210780|T210780]]
== 2019-01-01 ==
* 15:33 hashar: contint1001: deleting some extensions documentation for wmf branches: rm -fR /srv/org/wikimedia/doc/{Kartographer,MinervaNeue,MobileFrontend,Wikibase}/wmf  # [[phab:T118599|T118599]]


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Latest revision as of 23:09, 5 June 2023

2023-06-05

  • 23:09 mutante: contint* - PHP upgraded to version 7.4
  • 23:02 James_F: Zuul: [integration/docroot] Drop PHP 7.3 CI, box now runs 7.4
  • 22:22 James_F: Dockerfiles: [maven-java8] Update gerrit.wikimedia.org IP for T326368
  • 12:41 hashar: Reloaded Zuul to enable SonarQube codehealth on WikimediaMessages, WikimediaIncubator, WikimediaEvents and WikimediaMaintenance # T321837
  • 09:15 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/927122 # T337749

2023-06-02

  • 21:44 mutante: based on T336427#8890714 pending a response, everything already copied to gerrit1003, and extra paths being added to Bacule... shell access to gerrit1001 will be removed next week
  • 18:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/926593

2023-06-01

  • 13:48 James_F: doc: `sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/GWToolset/` for T337062
  • 13:34 James_F: Zuul: [wikidata/query/flink-rdf-streaming-updater] Archive
  • 13:18 Daimona: Dropping campaign_events.event_tracking_tool_id and campaign_events.event_tracking_tool_event_id on beta wikishared # T336362

2023-05-26

2023-05-25

  • 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # T334521
  • 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` # T334521
  • 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait # T334521
  • 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd

2023-05-24

  • 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, T319477
  • 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # T336362

2023-05-23

  • 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
  • 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` T337172

2023-05-21

2023-05-17

  • 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # T336840
  • 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)

2023-05-16

2023-05-12

2023-05-11

2023-05-10

  • 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot (T327424)

2023-05-09

2023-05-05

2023-05-04

2023-05-03

2023-05-02

2023-05-01

  • 17:20 brennen: experimenting with ldap user sync on gitlab.devtools

2023-04-28

  • 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for T335602

2023-04-27

  • 17:10 James_F: Zuul: [wvui] Mark repo as archived for T310244

2023-04-26

  • 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list

2023-04-25

  • 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (manually it works, but that's the same command that puppet is supposed to run !?) - T257317 T326368

2023-04-24

  • 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - T327853

2023-04-23

  • 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%{facts.fqdn}' instead of performance.discovery.wmnet as the latter doesn't resolve / would be an invalid cert for https://deployment-webperf21, ref T291015

2023-04-22

2023-04-21

  • 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # T335181

2023-04-20

2023-04-19

  • 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for T334998
  • 10:56 joe: Updating docker-pkg files on contint primary for helm-linter

2023-04-18

  • 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
  • 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for T334397

2023-04-17

  • 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for T281500

2023-04-12

2023-04-10

  • 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
  • 18:44 thcipriani: restart zuul-merger

2023-04-09

  • 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2

2023-04-06

  • 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list

2023-04-05

2023-04-04

2023-04-03

  • 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
  • 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 | https://gerrit.wikimedia.org/r/905190
  • 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
  • 10:20 hashar: Tag Quibble 1.5.4 @ 0a37ce9 # T332930 T331280 T333412
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T333586
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T33586

2023-03-31

2023-03-29

2023-03-23

  • 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.

2023-03-22

  • 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # T242031
  • 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql | sql centralauth --write # T242031

2023-03-20

  • 21:43 brennen: phabricator: setting sbassett to administrator per T331928

2023-03-19

  • 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in T332507
  • 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` T322469
  • 14:43 hashar: deployment-prep: databases update hourly job times out, filed T332507

2023-03-17

  • 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 (T332455)
  • 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 (T332455)
  • 10:36 mutante: disabled phab user 'Cloudduck1233'

2023-03-16

2023-03-15

  • 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T327708
  • 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
  • 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 | https://gerrit.wikimedia.org/r/c/integration/config/+/898807 | T331061, T331621
  • 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034

2023-03-14

  • 16:54 hashar: Building Docker images for Quibble 1.5.3
  • 16:41 hashar: Tag Quibble 1.5.3 @ df49e8a04f5 # T331621 T331061
  • 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
  • 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for T326163
  • 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for T326163
  • 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 02:44 James_F: MediaWiki being branched to REL1_40 from fd3ce76 for T329079

2023-03-13

  • 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging T326192
  • 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging T326192
  • 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior

2023-03-12

  • 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
  • 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
  • 22:57 James_F: Zuul: Add Want to the CI whitelist
  • 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 T328240

2023-03-11

  • 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-10

  • 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
  • 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`
  • 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233
  • 14:37 TheresNoTime: (deployment-prep) clear lock
  • 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-09

2023-03-08

2023-03-07

  • 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, T326163
  • 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/893765

2023-03-06

  • 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~` T327853
  • 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small | T327853
  • 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
  • 12:24 hashar: Building Docker images for Quibble 1.5.2 | https://gerrit.wikimedia.org/r/894617
  • 12:15 hashar: Tag Quibble 1.5.2 @ 932b48cab06c # T331236 T331061 T323750

2023-03-05

  • 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons (T331214#8666432)

2023-03-03

2023-03-02

  • 22:01 zabe: enable read-only mode and create dump of all databases # T331019
  • 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T331019
  • 16:35 zabe: create volume db13 and attach to deployment-db13 # T331019
  • 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # T331019
  • 16:24 zabe: shutoff deployment-db09 # T331019
  • 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # T331019
  • 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
  • 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was ed287d5 ) # T330752
  • 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # T330752
  • 12:13 zabe: delete deploymentwiki # T198673

2023-03-01

2023-02-28

2023-02-27

2023-02-25

  • 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
  • 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)

2023-02-24

  • 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- T289883
  • 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)

2023-02-23

  • 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
  • 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
  • 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
  • 14:00 taavi: update quibble* jenkins jobs for 891303
  • 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config

2023-02-22

2023-02-21

  • 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
  • 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs T329034
  • 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
  • 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
  • 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation | T321837

2023-02-20

2023-02-16

  • 10:31 jnuche: restarted Zuul to clear stale events

2023-02-15

2023-02-14

  • 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er T329000
  • 15:02 zabe: deployment-db12: start replicating from deployment-db09 # T329577
  • 14:04 zabe: delete deployment-db10 and volume db10 # T329577
  • 11:38 zabe: deployment-db11: start replicating from deployment-db09 # T329577
  • 02:19 zabe: deployment-db12: import dump into mariadb # T329577
  • 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # T329577
  • 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T329577
  • 01:50 zabe: create volume db12 and attach to deployment-db12 # T329577
  • 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # T329577
  • 01:14 zabe: deployment-db11: import dump into mariadb # T329577
  • 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # T329577
  • 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # T329577

2023-02-13

  • 23:48 zabe: create volume db11 and attach to deployment-db11 # T329577
  • 23:44 zabe: shutoff deployment-db10 # T329577
  • 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # T329577
  • 23:20 zabe: created backup of all databases on deployment-db09 # T329577
  • 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
  • 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm

2023-02-10

  • 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for T277440
  • 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for T324732

2023-02-08

2023-02-07

  • 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for T329077

2023-02-06

2023-02-03

  • 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old

2023-02-02

  • 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for T321309
  • 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete "' research/navigation-vectors'"`

2023-01-31

  • 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
  • 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
  • 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
  • 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for T326372

2023-01-30

2023-01-27

2023-01-25

  • 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner. T327949
  • 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for T327949

2023-01-24

  • 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now T326372
  • 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for T326372
  • 17:19 thcipriani: restarting ci jenkins for updates
  • 17:15 thcipriani: restarted releases-jenkins for plugin updates

2023-01-23

  • 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
  • 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for T292818

2023-01-20

2023-01-19

  • 20:01 James_F: Zuul: Add Izno to the CI allowlist
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # T299954
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
  • 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
  • 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
  • 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907

2023-01-18

2023-01-17

2023-01-16

2023-01-13

2023-01-12

2023-01-10

2023-01-09

  • 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
  • 16:30 wm-bot2: Increased quotas by 6 cores (T326568) - cookbook ran by arturo@nostromo
  • 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests

2023-01-08

  • 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
  • 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix

2023-01-07

2023-01-06

2023-01-05

2023-01-04

2023-01-03


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=601533"