You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Release Engineering/SAL"

From Wikitech
Jump to navigation Jump to search
imported>Stashbot
(Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - T143162)
imported>Stashbot
(James_F: Zuul: Define pipelines for REL1_34 T232025)
Line 1: Line 1:
 +
== 2019-09-12 ==
 +
* 23:37 James_F: Zuul: Define pipelines for REL1_34 [[phab:T232025|T232025]]
 +
* 22:19 hashar: integration-slave-jessie-1002: rm /srv/pbuilder/build/* /srv/pbuilder/aptcache/*/*
 +
* 22:16 hashar: Pooling back integration-slave-jessie-1002 in Jenkins ( /srv was 95% full )
 +
* 22:15 hashar: integration-slave-jessie-1002: apt-get clean
 +
* 21:45 James_F: Temporarily taking deployment-deploy01 offline as CI is stuck
 +
* 20:54 James_F: Zuul: Drop operations-mw-config-hhvm-composer-test-docker, the PHP72 one suffices
 +
* 20:31 James_F: Zuul: Add seccheck to PageNotice
 +
* 16:36 halfak: deploying ores {{Gerrit|7d45b80}}
 +
* 15:53 hauskatze: Phabricator: Switched articlequality, drafttopic, draftquality and editquality Diffusion (ORES project) to mirror to Gerrit via SSH/K19 refs. [[phab:T232661|T232661]]
 +
* 08:14 hashar: Marking integration-slave-docker-1043 offline , triggering mwcore-phpunit-coverage-master again # [[phab:T232706|T232706]]
 +
 
== 2019-09-11 ==
 
== 2019-09-11 ==
 
* 22:31 Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - [[phab:T143162|T143162]]
 
* 22:31 Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - [[phab:T143162|T143162]]

Revision as of 23:37, 12 September 2019

2019-09-12

  • 23:37 James_F: Zuul: Define pipelines for REL1_34 T232025
  • 22:19 hashar: integration-slave-jessie-1002: rm /srv/pbuilder/build/* /srv/pbuilder/aptcache/*/*
  • 22:16 hashar: Pooling back integration-slave-jessie-1002 in Jenkins ( /srv was 95% full )
  • 22:15 hashar: integration-slave-jessie-1002: apt-get clean
  • 21:45 James_F: Temporarily taking deployment-deploy01 offline as CI is stuck
  • 20:54 James_F: Zuul: Drop operations-mw-config-hhvm-composer-test-docker, the PHP72 one suffices
  • 20:31 James_F: Zuul: Add seccheck to PageNotice
  • 16:36 halfak: deploying ores 7d45b80
  • 15:53 hauskatze: Phabricator: Switched articlequality, drafttopic, draftquality and editquality Diffusion (ORES project) to mirror to Gerrit via SSH/K19 refs. T232661
  • 08:14 hashar: Marking integration-slave-docker-1043 offline , triggering mwcore-phpunit-coverage-master again # T232706

2019-09-11

  • 22:31 Krinkle: Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - T143162
  • 21:06 James_F: Zuul: Reload to add operations-mw-config-php72-composer-test-docker and operations-mw-config-hhvm-composer-lint-docker
  • 14:27 hashar: puppet cert clean integration-agent-docker-1001.integration.eqiad.wmflabs # T232619 | might unbreak puppet

2019-09-10

2019-09-09

2019-09-06

2019-09-05

  • 19:36 James_F: Zuul: Drop debian-glue-non-voting from cassandra-table-properties

2019-09-04

2019-09-03

  • 21:59 James_F: Zuul: Move ORES and CirrusSearch to codehealth T224990
  • 20:47 James_F: Zuul: Suspend trigger-mediawiki-pipeline-dev for now T218360
  • 15:57 James_F: Zuul: Enable the publish pipeline for mediawiki/services/kartotherian T223275
  • 15:36 James_F: Zuul: Add phan dependencies for MachineVision
  • 14:34 James_F: Zuul: And wmf/ branches for VisualEditor core.
  • 14:21 James_F: Zuul: Drop EducationProgram from dependencies
  • 09:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/533916 "Apply REL based pipelines"
  • 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
  • 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533905/ "Remove wmf pipelines from non mediawiki repos"
  • 08:32 hashar: err
  • 08:32 hashar: reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533909/ "zuul: add pipelines for MediaWiki releases"

2019-09-02

2019-08-31

2019-08-30

  • 18:28 James_F: Zuul: Move Popups and MobileFrontend to codehealth
  • 15:40 Krinkle: Updated "Prod Error" task form to have Title at the top and Trace below Description.
  • 10:53 hashar: deployment-mediawiki-07: nuke /mnt/mediawiki , last touched in April 2018 and using space on /

2019-08-29

2019-08-28

  • 23:00 James_F: Zuul: add tests for the labs/tools/maintain-kubeusers repo T228499
  • 23:00 James_F: Zuul: [TimedMediaHandler] Run phan job T224766 (merged but not deployed?)
  • 21:08 thcipriani: restarting zuul server since it's out-of-sync with reality
  • 16:11 hashar: Upgraded Jenkins for security update (2.176.3)
  • 16:08 hashar: integration: docker container prune -f ; docker image prune -f
  • 00:07 James_F: JJB: Delete mwext-npm-doc-publish, no longer used
  • 00:07 James_F: Zuul: Migrate MobileFrontend to extension-javascript-documentation T230831

2019-08-27

  • 20:25 James_F: Zuul: TwnMainPage: switch to extension-quibble-composer-nohhvm T231289

2019-08-26

  • 22:31 James_F: Zuul: Move Popups to extension-javascript-documentation
  • 22:12 James_F: Zuul: Move MinervaNeue to extension-javascript-documentation
  • 18:23 James_F: Zuul: [MinervaNeue, MobileFrontend, Popups] Add extension-javascript-documentation template
  • 15:35 hashar: deployment-mwmaint01: /var/cache/hhvm/cli.hhbc.sq3 (1.3GB) - T161598
  • 13:51 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/532379

2019-08-25

  • 21:27 Reedy: centralauth.globalblocks `ALTER TABLE /*_*/globalblocks MODIFY gb_reason varbinary(767) NOT NULL;
  • 16:43 Urbanecm: Revert 'Revert 45831e6 locally on beta cluster to test possible root cause of T231162'
  • 16:33 Urbanecm: Revert 45831e6 locally on beta cluster to test possible root cause of T231162

2019-08-23

  • 11:12 Urbanecm: deployment-prep: run scap sync to ensure everything is deployed (T231058)
  • 07:12 rxy: Applied SQL queries per phab:T231058#5433197

2019-08-21

  • 15:49 James_F: Zuul: Add phan for LiquidThreads
  • 08:41 hashar: Build docker-registry.discovery.wmnet/releng/quibble-coverage:0.0.34-4 # T230423

2019-08-20

  • 15:55 James_F: Zuul: Update Shreyas Minocha's whitelist e-mail
  • 15:45 urandom: creating Parsoid/PHP storage schema in deployment-prep -- T230792
  • 09:39 James_F: Zuul: [kartotherian] Disable pipeline on gate as it doesn't pass T223275

2019-08-19

  • 18:43 thcipriani: git -C /srv/mediawiki-staging/php-master/extensions/ConfirmEdit checkout maintenance/GenerateFancyCaptchas.php

2019-08-15

  • 12:13 James_F: Zuul: Make GrowthExperiments depend on Echo for non-Phan too.
  • 10:50 James_F: Zuul: Make GrowthExperiments depend on Echo
  • 09:05 James_F: Zuul: Kartotherian pipeline enablement T223275

2019-08-14

2019-08-13

2019-08-08

2019-08-07

  • 16:00 thcipriani: restarting jenkins for updates

2019-08-06

  • 21:24 James_F: Zuul: [TheWikipediaLibrary] Enable plan
  • 21:05 James_F: Zuul: [TheWikipediaLibrary] Enable phan-seccheck
  • 21:01 James_F: Zuul: [TheWikipediaLibrary] Add phan dependencies
  • 19:26 paladox: ssh -p 29418 gerrit.wikimedia.org replication start --url github mediawiki/core --wait --now
  • 17:09 accraze: deploying ores d08fa62
  • 11:42 paladox: ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --wait

2019-08-05

  • 22:24 accraze: deploying ores 4270244
  • 19:22 James_F: Zuul: [mediawiki/services/chromium-render] Enable pipeline testing and publishing T217114

2019-08-04

  • 12:11 hauskatze: gerrit: ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/OATHAuth --wait # T229756

2019-08-02

  • 17:30 James_F: Zuul: [operations/mediawiki-config] Stop running php70 jobs"

2019-08-01

  • 20:19 Krinkle: Resolve Jenkins admin warning about legacy token by revoking it from the only user id "<unknown>" that seems to have one
  • 19:00 James_F: Zuul: [BlueSpiceExtendedSearch] Disable HHVM tests, this doesn't work there

2019-07-31

  • 19:20 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki (T225056)
  • 18:27 James_F: Zuul: Move three repos to generic tox-docker job
  • 17:50 James_F: Zuul: Move cassandra-table-properties to use cassandra-table-properties-tox-docker
  • 16:27 James_F: Zuul: Archive SemanticTitle T229422
  • 15:55 James_F: Docker: Pushing operations-puppet:0.5.4
  • 15:46 James_F: Docker: Pushing quibble-coverage:0.0.34-3
  • 14:21 andrewbogott: moving deployment-sca02 to cloudvirt1030
  • 12:59 andrewbogott: moving deployment-elastic05, deployment-kafka-main-2, deployment-mx02, deployment-webperf11 to new cloudvirts

2019-07-30

  • 21:17 James_F: Docker: Push operations-puppet:0.5.3
  • 16:17 James_F: Docker: Pushing quibble-coverage:0.0.34-2
  • 15:39 cdanis: manually set dbctl etcd key to dummy value: cdanis@deployment-etcd-01:~$ etcdctl -u root -C http://127.0.0.1:2378 set /conftool/v1/mediawiki-config/eqiad/dbconfig
  • 14:44 James_F: Docker: Pushing quibble-coverage:0.0.34-1

2019-07-29

  • 18:23 James_F: Zuul: [translatewiki] T228376 Make PHP72 job voting, drop HHVM one
  • 18:13 James_F: Zuul: [translatewiki] T228376 Provide experimental PHP72 job to replace HHVM one

2019-07-27

2019-07-26

  • 21:10 legoktm: deploying https://gerrit.wikimedia.org/r/525871
  • 18:38 James_F: Zuul: operations/software/cassandra-table-properties initial jobs
  • 16:02 hauskatze: Deleted https://github.com/wikimedia/mediawiki-extensions-Copyvio | refs. T228087
  • 16:00 James_F: Zuul: [Copyvio] Mark as archived for CI T228087
  • 15:49 Krinkle: Fix Jenkins Console output section "Selenium" to account for Quibble 0.34 changes (multiple Selenium sections, one implicitly closing the other)
  • 15:45 James_F: doc1001: sudo -u doc-uploader rm -rf /srv/docroot/org/wikimedia/doc/cover-extensions/Copyvio/ # T228087
  • 04:43 James_F: Zuul: [mediawiki/libs/LangConv] Make this PHP7.2+ only
  • 04:36 James_F: Zuul: Add testing for mediawiki/libs/LangConv

2019-07-25

2019-07-24

  • 20:00 James_F: Zuul: Mark VIKI, VikiSemanticTitle, and VikiTitleIcon as broken.
  • 10:55 hauskatze: Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. T228655
  • 10:54 hauskatze: deployment-prep Dry-running extensions/AbuseFilter/maintenance/fixOldLogEntries.php refs. T228655

2019-07-23

2019-07-22

  • 22:43 James_F: Zuul: [wikimedia/iegreview] Adding mediawiki-i18n-check-docker template
  • 10:49 Amir1: ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master/extensions$ sudo rm -rf Wikidata

2019-07-19

2019-07-18

  • 22:18 James_F: Zuul: Archive maps/kartotherian and maps/tilerator
  • 21:17 mutante: rebooting gerrit server
  • 19:18 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.1
  • 19:03 James_F: Zuul: [analytics-mediawiki-storage] Drop node 6, switch to node10-browser-docker
  • 18:25 mutante: contint2001 - puppet run reinstalled python-dateutil and ruby-safe-yeaml, stopped zuul service but no more failures now
  • 18:25 mutante: contint2001 /etc/init.d/zuul-merger stop; apt install zuul (this time it worked to configure zuul after also stopping zuul-merge) ; apt-get autoremove (removed unused python packages); puppet agent -tv
  • 18:22 mutante: contint2001 /usr/sbin/dpkg-reconfigure: zuul is broken or not fully installed
  • 18:22 mutante: contint2001 systemctl stop zuul ; apt install zuul ... still fails
  • 18:10 James_F: Zuul: [wikimedia/services/kartotherian] Switch to custom job T228170
  • 18:06 James_F: Docker: Creating docker-registry.discovery.wmnet/releng/node10-kartotherian:0.1.0
  • 13:51 hashar: integration: in hiera project config, remove classes: [ contint::packages::apt ] | T225735
  • 13:48 hashar: integration: cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523150/ for apt unattended_upgrades | T225735
  • 13:37 hashar: integration: rm /etc/apt/sources.list.d/sury-php.list and remove php packages | https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523148/ | T225735

2019-07-17

  • 16:49 James_F: Zuul: [PageForms] Add phan dependencies on AdminLinks, Cargo, PageSchemas T228155
  • 16:48 James_F: Zuul: Register initial tests for eventgate-wikimedia T226668
  • 16:08 hashar: Jenkins: updating wikidata-query-gui-build to get rid of slave-scripts/bin/npm-setup.sh which requires an obsolete/non existing tmpfs # T227818
  • 09:48 hashar: Reloaded Zuul [Convert2Wiki] Archive extension
  • 00:42 James_F: Docker: Deploy ci-buster and lintr:0.2.0
  • 00:04 James_F: Docker: Complete rebuild finished and published; only took 2.5 hours. But now T228196 is fixed elsewise anyway, oh well.

2019-07-16

  • 21:58 hauskatze: GitHub: deleted `wikimedia/mediawiki-extensions-Convert2Wiki` refs. T228198
  • 21:29 James_F: Docker: Publishing a whole new set of RelEng images for T228196
  • 20:25 James_F: Docker: Running a general rebuild for all missing RelEng images T228196
  • 16:49 James_F: Zuul: [ContentTranslation] Adding CentralAuth to phan dependencies
  • 16:48 James_F: Zuul: Adding first tasks for mediawiki/services/kartotherian T228170
  • 15:19 James_F: Ran manual scap pull on deployment-deploy01
  • 15:10 James_F: beta-scap-eqiad failing because mw-config repo is out of date?
  • 14:57 James_F: Taking deployment-deploy01 offline
  • 14:50 James_F: Jenkins deadlock? Beta jenkins jobs haven't successfully run for ~20 hours.

2019-07-15

  • 22:40 James_F: Zuul: [CentralAuth] Add phan T189371
  • 18:05 James_F: Zuul: [MinervaNeue] Make phan voting T133664
  • 17:00 James_F: Zuul: [GrowthExperiments] Add MobileFrontend to phan dependencies
  • 16:49 James_F: Zuul: [MinervaNeue] Add phan as non-voting T133664
  • 16:04 Urbanecm: Run urbanecm@deployment-deploy01:~$ mwscript migrateUserGroup.php --wiki=commonswiki Image-reviewer image-reviewer (T216406)
  • 15:59 James_F: Zuul: Adding Echo and MobileFrontend dependencies to skins/MinervaNeue T133664
  • 15:38 James_F: Zuul: [SecurePoll] Make seccheck voting T205972
  • 15:37 James_F: Zuul: [CentralAuth] Add seccheck T189227
  • 13:30 hashar: Deleting compiler1003.puppet-diffs.eqiad.wmflabs .. insanely slow for some reason (was on cloudvirt1002) # T221969 )
  • 12:55 hashar: Creating compiler1003.puppet-diffs.eqiad.wmflabs [172.16.2.46] m1.large / 4vCPUs # T221969

2019-07-14

2019-07-12

  • 23:07 James_F: Zuul: Instead of hacking around for WebAuthn, define bespoke quibble php72+ template
  • 22:10 James_F: Zuul: Make seccheck voting on WebAuthn T226737
  • 19:28 James_F: Zuul: Add MobileFrontend and MinervaNeue to Wikibase's dependencies
  • 16:56 James_F: Zuul: Add Flow as phan dependency to GrowthExperiments
  • 00:32 James_F: Zuul: Migrate from php70-phan-seccheck to php72-phan-seccheck T226420

2019-07-11

  • 23:33 James_F: Pushing docker mediawiki-phan-seccheck:0.4.0
  • 21:34 James_F: Pushing new php-based docker images with php-bcmath extension for T227043
  • 20:30 mutante: add project member cdanis
  • 18:57 Krinkle: Fix "setup quibble prepare" Console Section pattern in Jenkins to account for changes in 'mediawiki-quibble-composertest-php72-docker' job
  • 18:22 James_F: Zuul: Archive ZeroBanner and ZeroPortal; drop from dependencies
  • 17:50 James_F: jforrester@doc1001:/srv/docroot/org/wikimedia/doc/cover-extensions$ rm -rf WikibaseQuality/ # T205065
  • 16:15 James_F: Zuul: Archiving WikibaseQuality
  • 09:48 hashar: jenkins: add more log details to hudson.model.Queue (FINER > FINEST) https://integration.wikimedia.org/ci/log/Jenkins%20Queue/configure # T72597
  • 09:06 hashar: beta cluster jobs are dead locked. Taking a thread dump in case it helps figure out what is going on. T72597

2019-07-10

  • 21:33 James_F: Zuul: Fix gerrit-docker pointer
  • 18:27 James_F: Zuul: Adding phan to StructuredDiscussions T224759
  • 17:55 James_F: Zuul: Disabling mwgate-rake for mediawiki/extensions/CentralNotice
  • 13:30 hashar: on zuul-merger: rm /srv/zuul/git/operations/software/gerrit due to T157818 # T189549

2019-07-09

  • 18:47 James_F: Zuul: Move mediawiki-quibble-composertest from php70 to php72
  • 18:18 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move back to php72 as primary
  • 18:08 James_F: Push docker composer-package-php72:0.2.2-s2
  • 17:21 James_F: Push docker mediawiki-phan-seccheck:0.3.6-2
  • 16:18 James_F: Push docker image +mediawiki-phan-seccheck:0.3.6-1
  • 15:53 James_F: Pushing docker images php-ast:1.0.1-2 and mediawiki-phan-seccheck:0.3.6
  • 15:29 thcipriani: contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to T157818
  • 15:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/520437
  • 14:08 James_F: Pushing mediawiki-phan-seccheck:0.3.6 docker image.
  • 09:48 hashar: contint1001: removing /var/lib/docker , no more needed since we now use /mnt/docker # T207707
  • 00:04 James_F: Pushing php-ast:1.0.1-1 and related composer-package* docker image updates

2019-07-08

  • 23:54 James_F: Zuul: Move wikimania-scholarships from php56 to php72
  • 23:35 James_F: Dropping mwext-MobileFrontend-npm-run-lint-modules-docker from Zuul T224997
  • 21:39 thcipriani: reenable jenkins jobs service-pipeline-test and service-pipeline-test-and-publish following docker mount move
  • 21:27 thcipriani: docker-pkg failed building docker-registry.discovery.wmnet/releng/composer-php56:0.1.5
  • 21:18 thcipriani: fab deploy_docker to pull down ci docker-pkg images (I hope)
  • 21:15 thcipriani: restart docker on contint1001 to pickup config change for https://gerrit.wikimedia.org/r/520738/
  • 21:14 thcipriani: disabled jenkins jobs service-pipeline-test and service-pipeline-test-and-publish pre-contint1001 docker restart
  • 13:38 hashar: Build docker-registry.wikimedia.org/releng/bazel:0.1.0 # T189549

2019-07-06

  • 11:20 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --wiki=deploymentwiki | T209565
  • 11:09 hauskatze: beta cluster: maurelio@deployment-deploy01 foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --dry-run | T209565

2019-07-05

  • 21:17 James_F: Adding node10-docker for react.ooui
  • 19:08 James_F: Pushing java8-sonar-scanner:0.4.11

2019-07-03

  • 21:01 accraze: deploying ores 676f7ba
  • 13:12 Urbanecm: deployment-prep Manually ran sudo -u jenkins-deploy scap sync-file wmf-config/CommonSettings-labs.php 'Not synced yet?' to sync file that's mysteriously not synced, but present on deployment host

2019-07-02

  • 20:34 James_F: Zuul: Adding quibble to the PopupAlerts, UnblockMe and Scribe extensions

2019-07-01

  • 23:24 James_F: Pushing quibble-coverage:0.0.32-1 image
  • 22:30 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
  • 20:40 James_F: Rebuilt codehealth jjbs using node10-test with https://gerrit.wikimedia.org/r/519652
  • 18:49 James_F: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental
  • 18:43 James_F: Reloading Zuul for mediawiki/tools/phan/SecurityCheckPlugin php72 tests

2019-06-30

2019-06-28

2019-06-27

2019-06-26

  • 17:57 James_F: Adding the experimental helm-lint job to deployment-charts T216049
  • 17:55 James_F: Pushing initial helm-linter docker image for T216049
  • 01:29 James_F: Moving labs/tools/stewardbots's composer test to php72+ T226585

2019-06-25

  • 22:00 hauskatze: Ran `replication start operations/debs/coredns --wait`
  • 21:07 hauskatze: gerrit: Created new repo: operations/debs/coredns - requested by fsero on mediawiki
  • 19:11 Krinkle: Update Jenkins collapsible section patterns to account for "quibble.cmd" => "quibble.commands" changing in the output
  • 15:58 James_F: Adding CentralAuth to FileImporter required extensions in Zuul
  • 15:51 awight: restart php7.2-fpm for wikidiff2 upgrade (T223391)
  • 15:50 awight: deployment-prep restart php7.2-fpm for wikidiff2 upgrade (T223391)
  • 14:39 hauskatze: Ran `replication start operations/debs/file-read-backwards --wait` on gerrit.wikimedia.org refs. T226449
  • 14:34 hauskatze: github: created https://github.com/wikimedia/operations-debs-file-read-backwards mirror refs. T226449
  • 14:14 hauskatze: gerrit: created `operations/debs/file-read-backwards.git` refs. T226449
  • 11:42 hashar: Deleting Jenkins jobs puppet-cdh-rake-docker and puppet-cdh4-rake-docker # T226474

2019-06-24

  • 23:31 thcipriani: cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604
  • 22:08 James_F: Zuul: Restore graphoid deploy repo tests, and comment on change-propagation ones.
  • 22:02 James_F: Pushed graphoid-deploy-npm-node-6-docker into jjb
  • 22:01 James_F: Re-publishing npm-test-graphoid, whoops.
  • 15:17 James_F: Reloading Zuul to drop wikiba.se-composer-php56-docker T224905
  • 13:04 hashar: Updating git mirrored repos: integration-cumin:~$ sudo cumin --trace --force 'name:docker' 'find /srv/git -name "*.git" -exec git -C {} fetch --prune \;'
  • 12:00 hashar: Updating Quibble based jobs to use version 0.0.32
  • 10:09 hashar: Tag Quibble 0.0.32 @ c0fe6eb # T211701 T218357 T220199 T223752
  • 09:50 hashar: Building container releng/zuul-cloner:0.2.0 and switching jobs # T224908
  • 08:25 hashar: Removing mysql-server from the CI slaves # https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517095/ | T225735

2019-06-22

2019-06-21

2019-06-20

  • 22:03 James_F: Pushing new java8-sonar-scanner:0.4.9 image.
  • 21:29 James_F: Pushing new java8-sonar-scanner:0.4.8 image
  • 20:06 James_F: Add new mwcore-codehealth-* config to Zuul
  • 20:04 James_F: Pushed new jjb mwcore-codehealth-* jobs and updated mwext-codehealth-* ones
  • 19:54 James_F: Pushing new java8-sonar-scanner:0.4.7 image
  • 17:45 James_F: Updated JJB definitions for mwext-php7\d-phan(-seccheck)?-docker for new docker-setup-mwext-for-phan macro.
  • 17:35 James_F: Updated JJB definitions for wikibase-{client|repo}-docker to use php72 not php70.
  • 17:12 James_F: Updated JJB definitions for all six coverage jobs
  • 15:09 hashar: Nuking castor caches for Quibble jobs integration-castor03 : rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/*quibble*/{composer,npm}
  • 14:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/518033/
  • 14:16 hashar: building some container for codehealth pipeline https://gerrit.wikimedia.org/r/518034

2019-06-19

2019-06-18

2019-06-15

  • 00:17 Krinkle: Disable "Publish/Notify" for mediawiki/libs/ObjectFactory.git in Diffusion – T222409, T143162

2019-06-14

  • 22:15 James_F: Dropped jjb jobs trending-edits?(-deploy)-npm-node-6-docker
  • 22:14 James_F: Reloading Zuul to archive the trending-edits repos
  • 21:50 James_F: Dropped docker-registry.discovery.wmnet/releng/hhvm-jessie:latest, unused
  • 19:57 James_F: Ran `docker rmi docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-4` on contint1001; unused.
  • 19:41 James_F: Zuul: Deploying Codehealth: Add Translate extension to pipeline
  • 19:39 James_F: Zuul: Deploying Remove coverage ci step for extensions with codehealth
  • 19:35 James_F: Zuul: Deploying Add sonar CodeHealth checks to Wikibase repos
  • 18:44 James_F: Publishing image docker-registry.discovery.wmnet/releng/quibble-fresnel:0.0.31-5
  • 16:58 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/517087
  • 16:08 James_F: Updating docker-pkg to provide quibble-coverage on php7.2
  • 16:04 James_F: Dropped docker-registry.discovery.wmnet/releng/quibble-stretch-php70:0.0.31-1 from contint1001, unused
  • 15:50 James_F: Bumped jjb mediawiki-phpunit-coverage-patch-docker

2019-06-13

  • 22:30 James_F: Deploying quibble-stretch-php70 update.
  • 22:17 James_F: Deploying quibble-coverage fix with chmod 0755 files.
  • 21:49 James_F: Reverted JJB jobs mediawiki-core-code-coverage-docker and mediawiki-phpunit-coverage-patch-docker back to quibble-stretch-php70
  • 21:43 James_F: Updated JJB job mediawiki-phpunit-coverage-patch-docker to point to quibble-coverage
  • 21:42 James_F: Update JJB job mediawiki-core-code-coverage-docker to point to quibble-coverage
  • 21:10 brennen: restarted jenkins agent on integration-trigger-01 to pick up new root configuration
  • 20:54 hashar: ¡Learning to log one step at a time!
  • 20:54 James_F: Updating docker-pkg files to create quibble-coverage image
  • 20:40 hashar: deleting docker-registry.wikimedia.org/releng/hhvm-jessie-compile from contint1001 # T224908
  • 18:17 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/514016
  • 17:05 brennen: changing jenkins remote root for integration-trigger-01 to /mnt/home/jenkins-deploy per T224069
  • 16:07 James_F: Dropped docker-registry.discovery.wmnet/releng/npm-test-mathoid:latest, unused
  • 16:06 James_F: Dropped mathoid?(-deploy)-npm-node-6-docker from jjb
  • 16:01 James_F: Dropped docker-registry.discovery.wmnet/releng/npm6:latest from contint1001, unused
  • 15:33 Krinkle: Disable "Publish/Notify" for quibble.git in diffusion, T143162, T196347
  • 00:13 James_F: Dropped jjb jobs for (change-propagation|citoid|cxserver|graphoid)?(-deploy)-npm-node-6-docker

2019-06-12

  • 23:34 brennen: deleting and rebuilding integration-trigger-01 (again) to use jessie instead of stretch
  • 23:20 James_F: Zuul: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now
  • 22:44 James_F: Enabling service-pipeline-test on four services: 3d2png, chromium-render, recommendation-api, trending-edits
  • 22:40 James_F: Dropped jjb jobs wikimedia-cz-tracker-npm-node-6-docker fundraising-dash-npm-node-6-docker wikimedia-portals-npm-browser-node-6-docker wikimedia-portals-deploy-npm-browser-node-6-docker
  • 22:38 brennen: deleting and re-creating integration-trigger-01
  • 22:22 James_F: Dropped mapdata-npm-node-6-docker and marvin-npm-node-6-docker from jjb
  • 22:21 James_F: Reloading Zuul to make mpadata use generic-node10-browser-docker
  • 21:54 James_F: Reloading Zuul to deploy, umm, 12 patches. 🤷🏽‍♂️
  • 21:07 James_F: Updating Zuul with https://gerrit.wikimedia.org/r/516674 https://gerrit.wikimedia.org/r/516675 https://gerrit.wikimedia.org/r/516676
  • 18:35 James_F: Reloaded Zuul to use node10 things for OOUI experimentally
  • 18:32 James_F: Deployed new jjb job ooui-npm-special-node10-plus-composer-docker
  • 16:28 Krinkle: Fix Jenkins Console section "setup quibble mw-install" pattern to not slurp the whole output on 'mediawiki-quibble-composertest' jobs
  • 16:27 James_F: Reloading Zuul to add service-pipeline-test?(-and-publish) to mediawiki/services/mobileapps
  • 00:49 James_F: Updated Zuul phan dependencies for ContentTranslation, adding GlobalPreferences

2019-06-11

  • 23:51 James_F: Updated Zuul phan dependencies for WikibaseLexeme, adding Wikibase T225524
  • 23:37 James_F: Dropped unused jjb jobs mwext-jsduck-publish mwext-VisualEditor-jsduck-docker, and mwext-VisualEditor-docker-publish
  • 23:34 James_F: Updating Zuul to drop extension-jsduck, no longer used (yay)
  • 23:14 James_F: Reloading Zuul to have Wikibase use extension-javascript-documentation
  • 22:39 James_F: PHP55 docker images deleted; needed to specify :0.… and also :latest because we're not just dropping one tag, we're dropping the double pair. All now done. T224907
  • 22:34 James_F: PHP55 docker images not deleted because they're still referenced (by what?).
  • 22:25 James_F: Running `docker rmi` for quibble-jessie-php55 (938a4b4f6abb) composer-test-php55 (9731d99e7a6f) composer-package-php55 (906edee88f4f) php55-compile (01f3fa5ef542) composer-php55 (a14836c5fc62) and php55 (f8b5dc437239)
  • 22:22 James_F: Updating docker-pkg files on contint1001 to drop all php55 images T224907
  • 22:18 James_F: Dropped from jjb: mwgate-composer-php55-docker, composer-php55-docker, composer-package-php55-docker, php-compile-php55-docker
  • 22:06 James_F: Drop unused jjb job integration-composer-check-php55-docker
  • 22:06 James_F: Reloading Zuul to drop integration-composer-check-php55-docker from integration/composer
  • 22:01 James_F: Reloading Zuul to add integration-composer-check-php72-docker to integration/composer
  • 21:59 James_F: Push new jjb job integration-composer-check-php72-docker
  • 21:56 James_F: Reloading Zuul to drop all *php55* jobs except integration-composer
  • 21:49 James_F: Reloading Zuul to add composer-php56-docker to integration/docroot
  • 21:45 James_F: Reloading Zuul to drop quibble*php55*
  • 21:42 James_F: Reloading Zuul to add composer-php72-docker to labs/tools/heritage
  • 21:07 brennen: creating integration-trigger-01 for T224069
  • 19:34 hashar: building docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.4.5 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/516511/
  • 19:04 halfak: deployment-prep deploying ores 04fbd58
  • 19:03 mdholloway: deployment-prep created deployment-wikifeeds01, added role::beta::docker_services and service config
  • 18:29 halfak: deployment-prep deploying ores 86809a4
  • 13:17 hashar: Building container docker-registry.wikimedia.org/releng/civicrm:0.1.0 # T210287

2019-06-10

  • 19:50 James_F: Restarting zuul to drop quibble experimental php70-phan-docker jobs; php72 suffices
  • 19:39 thcipriani: restarting jenkins
  • 19:37 James_F: Restarting jenkins with `sudo service restart jenkins` on contint1001 due to out-of-space issue earlier.
  • 19:10 Krinkle: Fix "setup quibble mw-install" collapsible section pattern in Jenkins administration, so that it doesn't match the "PHP 7.1+" warning from Phan. The pattern is meant to detect the first line of mw/maint/install.php output
  • 19:01 thcipriani: cleaning contint1001 docker images using https://phabricator.wikimedia.org/P8604
  • 17:48 James_F: Reloading zuul to set zuul parameters for PHP 7.2 variants of phan jobs too
  • 17:41 James_F: Reloading zuul to Add experimental mwext-php72-phan-docker jobs to quibble template
  • 17:38 James_F: Pushed new mwext-php72-phan-docker with jjb
  • 16:59 James_F: Deployed https://gerrit.wikimedia.org/r/516300
  • 16:31 legoktm: deployed https://gerrit.wikimedia.org/r/516298
  • 04:38 legoktm: deploying https://gerrit.wikimedia.org/r/516065 https://gerrit.wikimedia.org/r/516066

2019-06-08

  • 00:49 James_F: Updating docker-pkg files on contint1001 to build mediawiki-phan:0.2.0 based on php72 T223847

2019-06-07

2019-06-06

  • 18:06 James_F: Reloading Zuul to drop EventBus dependency on CentralNotice for now T225195
  • 15:12 hashar: reloading zuul for I044bed T222802
  • 15:12 hashar: reloading Zuul for I8d6ea5 T224301
  • 08:13 hashar: Reloading Zuul for I764972 which drops mwselenium-quibble-docker from Wikibase

2019-06-05

  • 19:34 andrewbogott: moving deployment-imagescaler03 to cloudvirt1029
  • 17:35 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547
  • 16:01 James_F: Reloading Zuul to deploy 172a4e7
  • 09:07 hashar: Pooled in integration-slave-docker-1058 and integration-slave-docker-1059

2019-06-04

  • 20:39 James_F: Updating jjb PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 T220917
  • 20:31 James_F: Updating docker-pkg files on contint1001 for quibble-stretch-php70 T220917
  • 17:44 James_F: Updating jjb PHP code coverage jobs to node10 quibble T224983
  • 17:31 James_F: Reloading Zuul to make mwext-MobileFrontend-npm-run-lint-modules-docker non-voting T224997
  • 16:10 hashar: Deleting integration-slave-docker-1021 and integration-slave-docker-1049 / too small disk (20G partition) and not enough ram (2G) # T221872
  • 13:24 hashar: Update all selenium-daily* jobs to use NodeJS 10 instead of NodeJS 6. T217545
  • 13:01 hashar: Building docker-registry.discovery.wmnet/releng/node10-test-browser:0.6.0 # T217545
  • 11:32 hashar: Upgrading Jenkins Pipeline plugins
  • 11:31 hashar: Upgrading Jenkins Warnings Next Generation Plugin # T224745
  • 11:30 hashar: Upgrading Jenkins BlueOcean and all its dependencies

2019-06-03

  • 19:32 James_F: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514089
  • 19:29 James_F: Generated mwgate-node10-docker and deployed via jjb.
  • 18:34 hashar: hswitch most Quibble jobs to node 10 T222406 - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ T222406
  • 17:40 James_F: Reloading Zuul to switch most Quibble jobs to node 10 T222406
  • 15:58 hashar: Deleting integration-slave-docker-1055 and integration-slave-docker-1056 . CPU is way too slow T223971
  • 15:46 hashar: reduce number of executors on all integration slave docker, they are somehow starving on CPU and/or IO when lot of mediawiki builds are running in parallel
  • 14:38 James_F: hashar and I are temporarily disabling running selenium tests in CI. See T211784 T222406 for more details.

2019-06-01

  • 22:54 James_F: Reloading Zuul to re-add Kartographer dependency for JsonConfig T224785
  • 21:14 James_F: Reloading Zuul to add phan for CentralNotice and add DannyS712 to whitelist

2019-05-31

  • 23:34 James_F: Reloading Zuul to remove Kartographer dependency for JsonConfig T224785
  • 22:12 James_F: Reloading Zuul to add phan for LdapAuthentication
  • 21:18 James_F: Reloading Zuul to add dependencies for TimedMediaHandler, ReadingLists, JsonConfig, and FlaggedRevs.
  • 21:13 James_F: Reloading Zuul to add phan for Collection, ContentTranslation, and Jade
  • 19:43 James_F: Reloading Zuul to add phan for Math(s)
  • 19:31 James_F: Reloading Zuul to add phan for MobileFrontend
  • 19:20 James_F: Reloading Zuul to add phan for Sentry
  • 18:35 James_F: Reloading Zuul to deploy CentralNotice dependency to EventBus
  • 18:17 James_F: Reloading Zuul to deploy extra dependencies for MobileFrontend
  • 18:10 James_F: Reloading Zuul to deploy phan for cldr and GlobalPreferences
  • 18:02 James_F: Reloading Zuul to deploy I0ee1c8166
  • 17:56 James_F: Reloading Zuul to deploy I9c115b1f6 and I0d98f01d

2019-05-30

2019-05-29

2019-05-28

2019-05-26

2019-05-22

2019-05-21

2019-05-20

2019-05-19

  • 15:43 hashar: Purging webperformance.integration.eqiad.wmflabs /tmp directory (inodes full)
  • 15:43 hashar: Purging all images on Docker CI slaves
  • 08:03 legoktm: deployed https://gerrit.wikimedia.org/r/511122

2019-05-18

2019-05-17

  • 22:55 paladox: created mediawiki/extensions/Scribe gerrit repo - T223662
  • 00:35 brennen: Updating dev-images docker-pkg files on contint1001

2019-05-16

  • 09:36 hashar: Successfully tagged docker-registry.discovery.wmnet/releng/composer-test-php72:0.1.0 # T223428
  • 07:35 hashar: integration-slave-jessie-1002: purging all php Debian packages and rerunning puppet. There is some package conflict somewhere :-(

2019-05-15

  • 11:55 hashar: Bring back https://integration.wikimedia.org/ci/computer/integration-castor03/ to restore the central cache behavior. In turn unblocking a wide range of builds
  • 11:32 hashar: bringing back castor, integration-castor03 is back
  • 09:35 hashar: Regenerating all CI jobs to disable castor saving entirely
  • 02:35 Krenair: Logged into deployment-sca0[12] as root and given them the correct nameservers to try to unbreak things. T221654
  • 01:00 thcipriani: /usr/local/sbin/keyholder arm

2019-05-14

  • 09:43 hashar: Upgraded all tox jobs to tox 3.10.0
  • 09:19 hashar: Building docker containers releng/tox-*:0.4.0

2019-05-13

2019-05-10

  • 18:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/509483
  • 18:08 Reedy: Reloading Zuul to deploy dependancies, new tests, phan
  • 17:53 Reedy: Reloading Zuul to deploy 5 more phan patches
  • 17:37 Reedy: Reloading Zuul to deploy 5 phan enabling patches
  • 17:23 Reedy: Reloading Zuul to deploy various phan additions and one dependancy
  • 17:08 Reedy: Reloading Zuul to deploy patches adding phab
  • 16:46 Reedy: Reloading Zuul to deploy various dependancy patches

2019-05-09

  • 08:17 elukey: remove mediawiki memcached nutcracker config from deployment-prep (should be unused) - T214275
  • 00:19 thcipriani: updating docker images on contint1001 for https://gerrit.wikimedia.org/r/508929
  • 00:19 thcipriani: clean docker images on contint1001

2019-05-07

2019-05-06

2019-05-03

2019-05-01

  • 17:53 halfak: deploying ores:52e9759

2019-04-30

  • 18:00 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/494778
  • 17:01 bearND: (beta): Update mobileapps to 142ba30
  • 02:01 hashar: Polled in integration-slave-docker-1055 and ntegration-slave-docker-1056
  • 01:35 hashar: Deleting integration-slave-docker-1037 (bigram) it is too slow for some reason # T222023

2019-04-29

  • 14:04 godog: add dsharpe user

2019-04-26

  • 13:33 Krenair: shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. T218729

2019-04-25

  • 16:35 Krenair: shutting down deployment-ms-fe02 and deployment-poolcounter04 T218729

2019-04-23

2019-04-19

  • 19:56 mutante: phab1003 - editing /srv/deployment/phabricator/deployment-cache/.config manually to replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet to fix git cloning issue on first puppet run on new host where somehow tin.eqiad still shows up. fixes puppet run on T221389

2019-04-18

2019-04-17

2019-04-16

  • 17:36 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf | tee T221107.php

2019-04-15

  • 10:16 Amir1: ores:8f01d40 going beta
  • 08:51 hashar: castor: nuked /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwselenium-quibble-docker # T220948

2019-04-13

  • 21:05 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. T91410
  • 21:00 Krinkle: Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul.
  • 21:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503669
  • 18:52 Krinkle: "Your JENKINS_HOME (/var/lib/jenkins) is almost full. "
  • 18:16 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503664
  • 17:44 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502807 (postgres php72)
  • 00:06 Krenair: transferred /home from deployment-cache-upload04 to deployment-cache-upload05 and shut down old one
  • 00:06 Krenair: transferred /home from deployment-cumin to deployment-cumin02 and shut down old one

2019-04-12

  • 15:09 Krenair: upload traffic now through cache-upload05

2019-04-11

2019-04-10

  • 23:32 James_F: Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. T220653
  • 22:36 James_F: Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches.
  • 20:27 paladox: create operations/software/gerrit/plugins/MassBranchCreation repository
  • 14:16 hashar: contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606
  • 14:13 hashar: contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606
  • 13:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/502790
  • 12:46 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502786
  • 12:40 hashar: contint2001: stopped puppet and zuul-merger for debugging

2019-04-09

2019-04-08

  • 22:46 hashar: cleaned docker images on integration-slave-docker-1030 and integration-slave-docker-1043 :)
  • 20:06 bearND: (beta): Update mobileapps to cdb9928 (T220045 T219411 T219667)
  • 14:47 hashar: hard rebooting integration-slave-docker-1053 OOM / deadlocked
  • 13:39 hashar: Deleting integration-slave-docker-1045 again. It uses Stretch instead of Jessie
  • 13:21 hashar: integration: fix cumin profiles that got renamed by 9e0aa82
  • 10:15 hashar: Building Docker image releng/quibble-stretch-php73:0.0.31-2 # T220237

2019-04-07

2019-04-06

2019-04-05

2019-04-04

2019-04-03

  • 06:13 hashar: gerrit: renamed group "scholarships" to "wikimedia-wikimania-scholarships". Made it owned by "Gerrit Managers" # T218864

2019-04-02

  • 22:24 hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/PageAssessments/maintenance/purgeUnusedProjects.php --wiki=enwikivoyage | T219935
  • 09:46 hashar: Upgrading CI Quibble jobs to 0.0.31

2019-04-01

  • 21:42 hauskatze: Imported tool-ldap from Diffusion to Gerrit with full history | T219703
  • 21:33 hauskatze: Created https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/ldap | T219703
  • 20:53 hashar: ssh contint1001.wikimedia.org sudo rm /tmp/docker-pkg-build.log
  • 20:44 hashar: Building Quibble 0.0.31 containers again # T219647 T219786
  • 20:10 hashar: gerrit: flush-caches --cache git_tags # some tag got stalled when querying over https - T219786
  • 18:50 hauskatze: Created mediawiki/extensions/ContributionCredits.git per request on mediawiki.org
  • 18:00 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/500504/
  • 17:25 Krinkle: fresnel-node10-browser-docker failing with ENOMEM. Depooled integration-slave-docker-1049 as precaution.
  • 17:04 hashar: Building CI docker images for Quibble 0.0.31 (yes it is a long day...)
  • 13:50 hashar: Reverted CI Jenkins jobs to Quibble 0.0.28 # T219647
  • 13:11 hashar: Upgraded CI Jenkins jobs to Quibble 0.0.30 # T219647
  • 10:50 hashar: Manually triggering postmerge step of citoid due to T219017 for mvolz. On contint1001: zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/citoid --change 497315,1
  • 08:08 hashar: Rebuilding Quibble Jessie containers that failed to build last week due to wikimedia-jessie container. # T219647
  • 08:07 hashar: Rebuilding container docker-registry.wikimedia.org/wikimedia-jessie # T219683

2019-03-29

2019-03-28

2019-03-27

  • 22:27 James_F: Altered Wikimedia GitHub settings to require 2FA; see T198810
  • 14:16 hashar: contint: refreshed all git caches manually from cumin: cumin --force 'name:docker' 'find /srv/git -name '*.git' -type d -print -exec git -C {} fetch --prune \;'
  • 14:16 hashar: contint: added repositories to the git caches https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499482/

2019-03-26

2019-03-25

  • 14:46 mateusbs17: sunset deployment-maps03

2019-03-24

  • 16:06 Krenair: shut off old deployment-db03 instance per T219087
  • 05:39 Krenair: cleaned up old puppet certs/nodes -certcentral-testclient03 -certcentral-testdns -certcentral03 -zotero01 -eventgate-analytics -t153468-test -rd3-cptest-master01 -maps05
  • 04:12 Krenair: removed php7.0-fpm package (conflicting with php7.2-fpm) and removed /etc/nginx/sites-enabled/default (conflicting with apache, puppet will remove the available copy too) from -deploy02, -jobrunner03, -mwmaint01, and -mediawiki-07 hosts to try to get puppet there happy again

2019-03-23

2019-03-22

2019-03-21

2019-03-20

2019-03-19

  • 22:47 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497322 / T218553
  • 21:00 <paladox>: hiding operations/software/gerrit/plugins/WikimediaWebSessions from normal users
  • 20:59 <paladox>: create operations/software/gerrit/plugins/WikimediaWebSessions project T218739
  • 20:23 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/preferences.d/jessie_mitaka_pinning_*' # T218559
  • 19:56 <hashar>: integration: sudo cumin --force '*' 'rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list' # T218559
  • 16:56 <Reedy>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/494802
  • 15:48 <Krinkle>: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497437
  • 14:01 <hashar>: Removed Zuul "check" pipeline | https://gerrit.wikimedia.org/r/#/c/integration/config/+/493188/ | T192217
  • 02:21 <Krenair>: sudo aptly publish --architectures="all,amd64" --skip-signing repo buster-deployment-prep
  • 02:09 <Krenair>: sudo aptly repo create -component="main" -distribution="buster-deployment-prep" buster-deployment-prep

2019-03-18

  • 16:39 Krenair: yet*
  • 16:38 Krenair: created deployment-acme-chief01 and a client instance for further acme-chief testing + dev. used stretch, would be buster like prod but not sure that's easily available outside testlabs yes
  • 10:09 hashar: contint1001: rm -fR /srv/doc1001.eqiad.wmnet
  • 10:06 hashar: github: deleting https://github.com/wikimedia/wikidata-gremlin # archived T155829
  • 09:58 hashar: arming keyholder on integration-cumin
  • 09:55 hashar: deleting shutdowned instance integration-publisher02 , we do not use it anymore since doc publishing got overhauled ( T137890 ) # T218146
  • 09:12 hashar: deployment-deploy01: cleaning disk: rm /var/cache/hhvm/cli.hhbc.sq3
  • 07:37 legoktm: deployed https://gerrit.wikimedia.org/r/496610

2019-03-17

  • 16:29 Krenair: deactivated and cleaned certs for deployment-redis3-changeprop02 and deployment-prometheus01 (which no longer appear to exist but were causing cumin to be upset)
  • 16:27 Krenair: armed deployment-cumin keyholder
  • 00:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/496887

2019-03-16

2019-03-15

2019-03-14

2019-03-13

  • 21:26 thcipriani: pool new bigram CI instance integration-slave-docker-1054
  • 20:58 thcipriani: deleting bigram CI instance integration-slave-docker-1046 due to corrupt disk cf: T218245
  • 20:44 thcipriani: marking integration-slave-docker-1046 offline
  • 20:38 hashar: Added integration-slave-docker-1045 to Jenkins. The instance existed in WMCS but was not in Jenkins
  • 20:13 bearND: (beta): Update mobileapps to 5865552 (7074964 d6dc3cd fbc6262)
  • 19:13 hashar: integration-slave-docker-1046 is back online # T218245
  • 19:04 hashar: hard rebooting integration-slave-docker-1046 , not reachable over ssh # T218245
  • 18:48 hashar: Building containers releng/quibble-jessie-hhvm and releng/quibble-stretch-hhvm with HHVM core_dump_report enabled # T216689
  • 18:09 ebernhardson: restart elasticsearch on deployment-elastic* to deploy apifeature usage fix (T183156)
  • 10:39 hashar: Triggered tests for ContentTranslation changes that had label:verified=-1 # T217654
  • 10:03 hashar: Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # T217654

2019-03-12

2019-03-11

  • 15:59 Krenair: previous !log was for deployment-prep
  • 15:56 Krenair: added MSantos to projectadmin, per chat in -infrastructure he'll use it to work on maps. user is a new foundation software engineer
  • 01:20 legoktm: deploying https://gerrit.wikimedia.org/r/495593

2019-03-10

2019-03-08

2019-03-06

  • 18:25 hasharAway: contint1001: restart Jenkins for plugins upgrade
  • 18:24 hasharAway: reloading zuul for I21d7fa
  • 18:02 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org
  • 17:37 hashar: Reloading Zuul for mediawiki/libs/Zest https://gerrit.wikimedia.org/r/#/c/integration/config/+/494609/2/zuul/layout.yaml
  • 16:27 Krinkle: doc1001 had bad permissions set on /org/wikimedia/doc (chmod 755 instead of 775). Making it impossible to git pull in the way that post-merge Jenkins job on integration/docroot recommends. Fixed with `sudo -u doc-uploader chmod 775 /srv/wikimedia/org/wikimedia/doc`.
  • 15:58 andrewbogott: deleting deployment-prometheus01 on Filippo's advice
  • 13:31 gehel: upgrading logstash to 5.6.14 on deployment-logstash2
  • 13:15 gehel: upgrading elasticsearch to 5.6.14 on deployment-logstash2
  • 11:33 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mediawiki-09:~$ sudo systemctl restart php7.2-fpm # T217323

2019-03-05

  • 20:10 thcipriani: reenable beta-scap-eqiad
  • 19:17 thcipriani: disable beta-scap-eqiad due to T217587
  • 10:13 hashar: integration: fixed erroneous ssh key restriction for cumin | T217642

2019-03-04

2019-03-03

  • 02:26 Krinkle: tried rebooting or shutting down integration-slave-docker-1021, no response on horizon. Did pause/resume instead, which did work, after which shutdown/start worked. Jenkins agent has been relaunched and seems online again.
  • 02:20 Krinkle: integration-slave-docker-1021 (ci1.medium) has jobs failing on it due to ENOMEM. Horizon shows in log: integration-slave-docker-1021 login: [4961938.696837] Out of memory: Kill process 21770 (chromium) score 841 or sacrifice child; [4961938.699176] Killed process 21770 (chromium) total-vm:3171496kB, anon-rss:1379288kB, file-rss:0kB, shmem-rss:1636kB

2019-03-02

2019-03-01

  • 19:17 thcipriani: integration-slave-docker-1021:/# docker rmi $(docker images | grep " months " |grep -v " [1-2] months " | awk '{print $3}')
  • 17:02 thcipriani: integration-slave-jessie-1004 back online
  • 16:58 thcipriani: integration-slave-jessie-1002 back online (disk space looked fine); rebooting integration-slave-jessie-1004 -- can't ssh to machine
  • 16:11 Lucas_WMDE: delete refs/master and refs/gerrit/master on WikibaseQualityConstraints repository T217408
  • 15:49 hashar: wikidata/query/blazegraph change Gerrit config to require a change-id # T216855
  • 14:28 hashar: Upgrading integration/jenkins-job-builder to version 2.0.2 + one custom hack 11aa5de4...a06d173e # T143731
  • 14:18 hashar: integration/jenkins-job-builder : importing upstream code to new branch "upstream". Push all upstream tags to our repository

2019-02-28

2019-02-27

2019-02-26

2019-02-25

  • 23:32 twentyafterfour: root@deployment-db05# mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata # T216067
  • 22:14 thcipriani: docker rmi images without "latest" tag on contint1001 to free space -- should have kept all current docker-pkg images as well as images with children -- T217094
  • 13:39 hashar: Rebuilding some CI Docker images using PHP sury.org to switch the sury.org component from jessie to stretch ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/492666/ )

2019-02-24

2019-02-23

2019-02-22

2019-02-21

2019-02-20

  • 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz | sudo tee /var/lib/dpkg/status # T216635
  • 19:45 hashar: deployment-db03: restored some old /var/lib/dpkg/status file : sudo zcat /var/backups/dpkg.status.2.gz | sudo tee /var/lib/dpkg/status
  • 17:26 hashar: For beta cluster the MySQL master database has some innodb issue T216635 , the MySQL slave has an issue as well T216067
  • 17:09 hashar: reloading zuul for Id1e3af
  • 17:09 hashar: contint1001: fix broken root ownership on zuul git deploy repo: sudo find /etc/zuul/wikimedia/.git -not -user zuul -exec chown zuul:zuul {} +
  • 17:05 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490606/
  • 16:57 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/490559
  • 16:36 hauskatze: Ran replication start mediawiki/extensions/PageViewInfo --wait on gerrit.wikimedia to populate GitHub mirror (success messages afterwards) | T180864
  • 00:25 greg-g: disabled beta-update-databases-eqiad in the jenkins UI - T216067

2019-02-19

2019-02-18

2019-02-17

2019-02-16

2019-02-15

  • 17:28 thcipriani: integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk

2019-02-14

2019-02-13

  • 21:32 marxarelli: dduvall@integration-slave-jessie-1001:/mnt/home/jenkins-deploy$ `rm -rf .gradle/ .m2/` due to full disk
  • 21:21 marxarelli: bringing integration-slave-docker-1046 and integration-slave-jessie-1001 back online
  • 21:20 marxarelli: dduvall@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ `sudo rm -rf *` due to full disk
  • 21:15 marxarelli: removing old docker images on integration-slave-docker-1046
  • 21:10 marxarelli: starting migrated integration-slave-docker-1046 instance
  • 21:01 marxarelli: pooling new jenkins node for integration-slave-docker-1052
  • 20:46 marxarelli: pooling jenkins node for integration-slave-docker-1051
  • 20:45 marxarelli: launching replacement instance integration-slave-docker-1052
  • 20:35 marxarelli: launching replacement instance integration-slave-docker-1051
  • 20:32 marxarelli: pooling jenkins node for integration-slave-docker-1050
  • {{safesubst:SAL entry|1=20:15 marxarelli: integration-slave-docker-{1044,1046,1047} unresponsiveness due to cloudvirt failure. 1046 is being moved already by CS. deleting 1044 and 1047}}
  • {{safesubst:SAL entry|1=19:57 marxarelli: seeing jenkins agent connection failures for integration-slave-docker-{1044,1046,1047}}}
  • 19:48 marxarelli: pooling replacement jenkins node integration-slave-docker-1049
  • 19:34 marxarelli: deleting integration-slave-jessie-android jenkins node and instance
  • 19:33 marxarelli: deleting integration-slave-jessie-1003 jenkins node and instance
  • 19:32 marxarelli: deleting integration-slave-docker-1033 jenkins node and instance
  • 19:25 marxarelli: deleting integration-slave-docker-1017 jenkins node and instance
  • 18:45 Krinkle: integration-slave-jessie-1003 seems to be consitently unable to start jobs, marking as offline manually
  • 18:32 thcipriani: bringing up new integration-castor03, re-enabling castor-save* jobs
  • 18:15 marxarelli: adding new jenkins node integration-slave-docker-1048
  • 18:02 marxarelli: launching new integration-slave-docker-1048 instance
  • 17:59 marxarelli: deleting integration-slave-docker-1038 node and deleting instance
  • 17:50 marxarelli: bringing integration-slave-docker-1033 back online after clearing out old docker images
  • 17:33 thcipriani: rebuilding integration-castor03
  • 17:21 thcipriani: stopping rsync server on castor03
  • 17:21 twentyafterfour: stopped rsync on castor03
  • 17:16 twentyafterfour: disconnected castor03 from jenkins
  • 16:48 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/487880/
  • 14:34 thcipriani: modified castor-save-workspace-cache to exit 0 and run on blubber nodes while integration-castor03 is down
  • 14:26 dcausse: deployement-prep: upgrading to elastic 5.6.14

2019-02-12

2019-02-11

2019-02-10

2019-02-08

  • 20:20 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config, ref T91410
  • 15:59 addshore: this reload also included "Switch npm-audit job to node10"? T211784, which did touch the zuul file
  • 15:58 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/489241/
  • 03:10 Krinkle: Delete various jobs on Jenkins that no longer exist in JJB config
  • 00:28 Krinkle: krinkle@doc1001: sudo -u doc-uploader chmod 775 /srv/docroot/org/wikimedia/doc/
  • 00:12 marxarelli: removed old docker images on contint1001 to free up space

2019-02-07

  • 23:17 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
  • 23:15 thcipriani: integration-slave-docker-1033:sudo docker image prune and bring back online
  • 22:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/467550 (
  • 19:09 paladox: created integration/zuul/build gerrit repo for T215458
  • 19:05 paladox: created integration/zuul/wheels gerrit repo for T215458
  • 15:48 addshore: brought integration-slave-docker-1043 back online
  • 15:48 addshore: addshore@integration-slave-docker-1043:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 14.86GB
  • 08:49 hashar: cleaning docker images on integration-slave-docker-1021

2019-02-06

  • 22:34 shdubsh: Deploy node-exporter 0.17 T213708
  • 14:12 godog: shut off deployment-prometheus01 - T215272
  • 14:00 godog: switch beta-prometheus to deployment-prometheus02 - T215272

2019-02-05

  • 20:07 ebernhardson: jobrunner port 9006 is firewalled, revert to 9005 and created T215339 to fix job queue in beta cluste
  • 19:36 ebernhardson: Update profile::cpjobqueue::{jobrunner,videoscaler}_host in horizon hiera from port 9005 to 9006 to match new restrictions in gerrit.wikimedia.org/r/481866
  • 16:29 addshore: T215288 added mirrys to deployment-prep as a user
  • 15:32 addshore: T215278 addshore@integration-slave-docker-1037:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.59GB

2019-02-04

  • 23:13 thcipriani: integration-slave-docker-1040:sudo docker image prune and bring back online
  • 23:12 thcipriani: integration-slave-docker-1038:sudo docker image prune and bring back online
  • 21:48 ebernhardson: restart logstash on deployment-logstash2
  • 15:25 hashar: removed Jenkins user "nodepoolmanager" as well as related authorizations | T209361

2019-02-03

2019-02-02

  • 22:17 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2019-01-31

  • 15:03 thcipriani: rearm keyholder on deployment-deploy01
  • 12:05 arturo: VM instances deployment-deploy01,deployment-deploy02,deployment-fluorine02,deployment-kafka-jumbo-2,deployment-kafka-main-1,deployment-maps04,deployment-mcs01,deployment-mediawiki-09,deployment-memc04,deployment-ms-be03,deployment-ms-fe02,deployment-parsoid09,deployment-sca04,deployment-webperf12, were stopped briefly due to issue in hypervisor (T215012)

2019-01-30

2019-01-29

  • 07:41 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * b/c full disk

2019-01-28

  • 16:33 hashar: contint1001: cleaning up disk space on /
  • 13:07 addshore: bringing integration-slave-docker-1041 back online
  • 13:07 addshore: addshore@integration-slave-docker-1041:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 16.12GB
  • 09:37 Amir1: ores:ad160b0 is going beta

2019-01-27

  • 19:57 addshore: bringing integration-slave-docker-1034 back online
  • 19:50 addshore: addshore@integration-slave-docker-1034:~$ sudo docker image prune -a --force --filter "until=2191h" // (3 months?) Total reclaimed space: 17.12GB

2019-01-26

2019-01-25

2019-01-23

2019-01-22

2019-01-21

  • 19:49 hashar: integration: update sudo rule for debian-glue to keep env variable EXTRAPACKAGES. Would let us get eatmydata included | T214328
  • 15:40 hashar: contint1001: removing all generated doc/cover from /srv/org/wikimedia/doc | T137890

2019-01-18

  • 23:22 hashar: contint1001: sudo docker image prune # Total reclaimed space: 3.592GB
  • 23:00 Krinkle: Some docker builds on integration-slave-docker-1021 failing with ENOMEM
  • 23:00 mutante: contint1001 - gzipping more files in /var/log/zuul/
  • 22:57 mutante: contint1001 - moved zuul logs from 2018 and gzipped zuul logs from /var/log/zuul to /srv/logs/zuul to free disk space on /
  • 22:39 mutante: contint1001 - apt-get clean - disk space low
  • 22:31 Krinkle: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482527 / T212602

2019-01-17

  • 19:34 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
  • 08:39 legoktm: deploying composer docker image - https://gerrit.wikimedia.org/r/484853

2019-01-16

  • 21:11 bearND: (beta): Update mobileapps to 258d76b page summary changes

2019-01-15

  • 09:00 hashar: Deleting Docker images on integration-slave-docker-1021

2019-01-14

  • 22:02 bearND: (beta): Update mobileapps to f2658de
  • 21:47 mutante: deployment-mcs01 - sudo su deploy-service; cd /srv/deployment/mobileapps/deploy-cache/revs/1182b3b8f288df0221257b929ca43fb86862c2f8/scap ; touch log (for debugging permission problem reported by bearND)
  • 14:31 hashar: Nuked Castor cache for all *tox* jobs. Some might have cached binary wheels compiled against a lib that is no more existing (eg libmysqlclient.so.18 for mysql-python). Follow up the jessie -> stretch upgrade # T191764
  • 14:28 hashar: Deleted Castor cache for wikimedia-cz/tracker mysql-python got cached as a wheel but compiled against libmysqlclient.so.18. That fails with the new tox...:0.3.0 containers which uses mariadb / libmysqlclient.so compat symlink

2019-01-11

2019-01-09

2019-01-08

  • 21:52 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/476600/
  • 21:31 Hauskatze: github: @niedzielski updated @jdlrobson permission on Wikimedia from `read` to `admin`
  • 21:30 Hauskatze: github:
  • 20:46 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/482855/
  • 19:53 mutante: deployment-prep adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
  • 19:53 mutante: adjusting puppet config on deployment-mwmaint01. remove "mediawiki_maintenance" role from "other classes" section and apply "mediawiki::maintenance" instead after role rename in gerrit:479131 for consistency with other mediawiki:: roles
  • 14:25 hashar: Upgrading plugins on https://releases-jenkins.wikimedia.org/
  • 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T 101131
  • 05:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482752
  • 02:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/482751

2019-01-07

2019-01-06

2019-01-03

2019-01-02

  • 10:19 hashar: updating all debian-glue jobs and creating new ones with hardcoded distributions (trusty, jessie, stretch, unstable) T210780

2019-01-01

  • 15:33 hashar: contint1001: deleting some extensions documentation for wmf branches: rm -fR /srv/org/wikimedia/doc/{Kartographer,MinervaNeue,MobileFrontend,Wikibase}/wmf # T118599

Archives