You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(hashar: Updating Quibble Jenkins jobs to 0.0.26)
imported>Stashbot
(TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye)
(865 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== 2018-09-18 ==
== 2022-06-28 ==
* 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 09:46 hashar: updating mwselenium-quibble-docker to Quibble 0.0.26
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]
* 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins  # untouched since 2016
* 07:31 hashar: cleaning disk on deployment-sca01
* 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors


== 2018-09-17 ==
== 2022-06-27 ==
* 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 22:00 Reedy: update.php in screen is done
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
* 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
* 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
* 20:17 bearND: (beta): Update mobileapps to {{Gerrit|d56e4cf}}
* 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy {{Gerrit|I5df2b72fa411adf05027c8f6745284e7e1dedf7a}}
* 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
* 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest ([[phab:T109776|T109776]])
* 13:24 hashar: Building docker images for Quibble 0.0.25 {{!}} https://gerrit.wikimedia.org/r/460536
* 12:58 hashar: mwselenium-quibble-docker to Chromium 69 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ {{!}} [[phab:T204214|T204214]]
* 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
* 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
* 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
* 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775


== 2018-09-16 ==
== 2022-06-24 ==
* 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
* 20:52 taavi: added `denisse` as a member
* 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism {{!}} [[phab:T200856|T200856]]
* 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> {{!}} [[phab:T204372|T204372]]


== 2018-09-14 ==
== 2022-06-23 ==
* 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
* 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
* 14:16 hashar: deployment-deploy01:  sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
* 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
* 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
* 13:48 hashar: Tagged Quibble 0.0.25 {{Gerrit|01663f5d9bcc9e77473732a9fd230271bbd43db4}}
* 11:49 hashar: Cleaned /tmp on deployment-deploy01
* 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ {{!}} [[phab:T204340|T204340]]
* 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
* 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473


== 2018-09-13 ==
== 2022-06-22 ==
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor  mediawiki/vendor fails composer --ansi validate --no-check-publish
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 {{!}} [[phab:T203902|T203902]]
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 08:42 hashar: aborted maintenance-disconnect-full-disks job {{!}} [[phab:T204077|T204077]]
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180
* 05:13 legoktm: onlined integration-slave-docker-1021
* 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
* 05:08 legoktm: deployed
* 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184


== 2018-09-12 ==
== 2022-06-21 ==
* 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
* 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
* 20:51 marxarelli: deleting integration-slave-docker-1002 instance
* 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
* 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
* 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
* 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
* 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
* 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
* 18:21 marxarelli: deleting integration-slave-docker-1027 instance
* 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
* 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
* 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
* 17:59 marxarelli: deleting integration-slave-docker-1001 instance
* 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
* 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
* 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
* 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
* 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
* 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
* 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker  for Chromium 69 and Firefox 60 - [[phab:T203902|T203902]]
* 00:14 marxarelli: deleting instance integration-slave-docker-1029


== 2018-09-11 ==
== 2022-06-20 ==
* 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
* 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
* 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*'  # [[phab:T188742|T188742]]


== 2018-09-10 ==
== 2022-06-17 ==
* 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 08:34 Amir1: ores {{Gerrit|ae96071}} is going to beta
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack


== 2018-09-09 ==
== 2022-06-16 ==
* 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]


== 2018-09-08 ==
== 2022-06-15 ==
* 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
* 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
* 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
* 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882


== 2018-09-07 ==
== 2022-06-14 ==
* 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 18:30 legoktm: started gear_client on contint1001
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production
* 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
* 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4.  8 CPUS can not sustain 5 concurrent Quibble builds {{!}} [[phab:T201972|T201972]]


== 2018-09-06 ==
== 2022-06-13 ==
* 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
* 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill {{Gerrit|eae9ba3a1459}} -> stopped a container that had been running for 5 hours
* 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
* 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026


== 2018-09-05 ==
== 2022-06-12 ==
* 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
* 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
* 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095


== 2018-09-04 ==
== 2022-06-10 ==
* 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for [[phab:T203431|T203431]]
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check


== 2018-09-03 ==
== 2022-06-09 ==
* 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration
* 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
* 07:54 hashar: deployment-deploy01 scap sync-l10n master
* 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
* 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
* 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
* 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
* 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278


== 2018-09-02 ==
== 2022-06-08 ==
* 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv {{!}} attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ {{!}} more permissions fixes for beta-scap-eqiad
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134
* 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap {{!}} fixing beta-scap-update-equiad failures


== 2018-09-01 ==
== 2022-06-06 ==
* 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
* 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
* 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
* 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
* 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
* 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase


== 2018-08-31 ==
== 2022-06-02 ==
* 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
* 11:26 hashar: Restarting Jenkins on contint2001
* 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
* 11:19 hashar: Restarting Jenkins on releases1002
* 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
* 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
* 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
* 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528


== 2018-08-30 ==
== 2022-05-31 ==
* 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for [[phab:T199417|T199417]]
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for [[phab:T199422|T199422]]
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster
* 19:38 thcipriani: bring integration-slave-docker-1025 back online
* 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per [[phab:T197918|T197918]]
* 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
* 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
* 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
* 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
* 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
* 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073


== 2018-08-29 ==
== 2022-05-30 ==
* 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ ([[phab:T203093|T203093]])
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189  ([[phab:T203110|T203110]])
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]
* 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
* 12:35 hashar: fixed keyholder on deployment-deploy01
* 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002


== 2018-08-28 ==
== 2022-05-28 ==
* 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ {{!}} [[phab:T203009|T203009]]
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]
* 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 {{!}} [[phab:T203009|T203009]]
* 17:30 legoktm: reverted back to previous quibble version ([[phab:T203009|T203009]])
* 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 {{!}} https://gerrit.wikimedia.org/r/455861
* 16:13 hashar: building docker containers for quibble 0.0.22
* 16:06 hashar: tagged quibble 0.0.22 {{Gerrit|5093492821693918da4fecdcb15a8ca831ce560f}}


== 2018-08-27 ==
== 2022-05-27 ==
* 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 18:49 Amir1: ores:0e8cc73 is going to beta cluster
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ {{!}} [[phab:T179963|T179963]]
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian {{!}} [[phab:T174338|T174338]]
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2018-08-25 ==
== 2022-05-26 ==
* 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
* 18:33 dancy: Updated Jenkins beta-* job configs
* 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch
* 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107


== 2018-08-24 ==
== 2022-05-25 ==
* 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
* 15:46 dancy: Restarted apache2 on gerrit1001
* 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO


== 2018-08-23 ==
== 2022-05-24 ==
* 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
* 11:06 _joe_: reverting my cherry-pick
* 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
* 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge


== 2018-08-22 ==
== 2022-05-23 ==
* 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 20:05 bearND: (beta): Update mobileapps to {{Gerrit|141ff20}} ([[phab:T202105|T202105]] [[phab:T202237|T202237]])
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport [[phab:T202220|T202220]]
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 05:37 TimStarling: on deployment-db03 installing pt-heartbeat
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2018-08-21 ==
== 2022-05-22 ==
* 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms ([[phab:T202260|T202260]])
* 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used)  rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* {{!}} [[phab:T202341|T202341]]
* 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
* 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
* 01:13 legoktm: triggering seccheck jobs manually on contint1001


== 2018-08-20 ==
== 2022-05-21 ==
* 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]
* 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
* 20:16 mateusbs17: (beta): Update mobileapps to {{Gerrit|95e976d}} ([[phab:T202105|T202105]])
* 19:53 marxarelli: Deleting instance integration-slave-docker-1016
* 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
* 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
* 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
* 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
* 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced {{!}} [[phab:T192561|T192561]]
* 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
* 01:42 legoktm: running jobs manually on contint1001 with gear_client
* 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
* 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
* 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884


== 2018-08-18 ==
== 2022-05-20 ==
* 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809


== 2018-08-17 ==
== 2022-05-19 ==
* 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already
* 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
* 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
* 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
* 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
* 08:46 addshore: reloaded zuul
* 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618


== 2018-08-16 ==
== 2022-05-18 ==
* 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to {{Gerrit|a808c9d}} ([[phab:T201979|T201979]])
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]


== 2018-08-15 ==
== 2022-05-17 ==
* 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for [[phab:T201782|T201782]].
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins


== 2018-08-14 ==
== 2022-05-16 ==
* 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809
* 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
* 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
* 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2


== 2018-08-13 ==
== 2022-05-14 ==
* {{safesubst:SAL entry|1=23:05 Krenair: deactivated and cleaned puppet node entries for deployment-{tin,mira} T192561}}
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
* 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......


== 2018-08-12 ==
== 2022-05-13 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2018-08-10 ==
== 2022-05-12 ==
* 17:58 marxarelli: Reloading zuul to deploy {{Gerrit|I32007b56d05c80aadefa686c505e1da23c343dc1}}
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2018-08-09 ==
== 2022-05-11 ==
* 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
* 17:13 bearND: (beta): Update mobileapps to {{Gerrit|616ffef}}
* 01:42 awight: [[phab:T201518|T201518]]: ORES, fawiki wp10, misc updates


== 2018-08-08 ==
== 2022-05-10 ==
* 20:39 James_F: Updated mobileapps to {{Gerrit|162ebcf}} in Beta Cluster
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055  / [[phab:T199850|T199850]]
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377


== 2018-08-07 ==
== 2022-05-09 ==
* 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / [[phab:T201107|T201107]])
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 19:52 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/450644
* 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005


== 2018-08-06 ==
== 2022-05-08 ==
* 16:11 addshore: reload zuul for  https://gerrit.wikimedia.org/r/450604
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
* 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514


== 2018-08-04 ==
== 2022-05-06 ==
* 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]


== 2018-08-02 ==
== 2022-05-05 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it ([[phab:T201077|T201077]])
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 14:07 gehel: re-imaging depoyment-elastic* to stretch
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins  builds from https://integration.wikimedia.org/ci/view/Beta/


== 2018-08-01 ==
== 2022-05-04 ==
* 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones {{!}} [[phab:T197469|T197469]]
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 ([[phab:T199801|T199801]])
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 10:08 dcausse: deployment-prep [[phab:T199270|T199270]]: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2018-07-31 ==
== 2022-05-03 ==
* 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor . We use 'es6' branch instead. (made for dcausse )
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2018-07-30 ==
== 2022-05-02 ==
* 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]
* 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:03 Krinkle: Delete and recreate deployment-webperf13 ([[phab:T195312|T195312]] / [[phab:T180761|T180761]])
* 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
* 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value ([[phab:T200634|T200634]])
* 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/  {{!}} [[phab:T195918|T195918]]
* 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, [[phab:T195709|T195709]]
* 17:04 hashar: github: granted @thcipriani owner rights
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087


== 2018-07-29 ==
== 2022-04-29 ==
* 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]


== 2018-07-28 ==
== 2022-04-28 ==
* 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # [[phab:T197698|T197698]]
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC


== 2018-07-27 ==
== 2022-04-27 ==
* 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container {{!}} [[phab:T195918|T195918]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029
* 14:43 hashar: Building tox-certcentral container for vgutierrez !
* 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
* 12:05 hashar: Building releng/quibble images 0.0.21
* 11:40 hashar: Tagged quibble 0.0.21  {{Gerrit|e681dce021e}}
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ [[phab:T200508|T200508]]
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
* 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337


== 2018-07-26 ==
== 2022-04-26 ==
* 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 15:03 zeljkof: Reloading Zuul to deploy {{Gerrit|208a505b5d2d864e57924325ca7665cf1231c8dc}}
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 04:00 kart_: Updated cxserver to {{Gerrit|e98c81b}}
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2018-07-24 ==
== 2022-04-25 ==
* 05:22 kart_: Updated cxserver to {{Gerrit|d3c9d15}}
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI


== 2018-07-23 ==
== 2022-04-20 ==
* 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to {{Gerrit|254cef5}}
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service
* 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
* 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, [[phab:T195312|T195312]]
* 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 [[phab:T195312|T195312]]
* 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / [[phab:T195312|T195312]]
* 15:34 Krinkle: Creating deployment-webperf13 - [[phab:T195312|T195312]]
* 15:34 Krinkle: Deleting deployment-webperf12 - [[phab:T195312|T195312]]
* 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
* 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB. The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable


== 2018-07-19 ==
== 2022-04-18 ==
* 21:47 hashar: CI:  Force npm loglevel to 'info'  via a Zuul parameter {{!}} [[phab:T198348|T198348]]
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/#/c/446863 {{!}} [[phab:T198348|T198348]]
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/442989 {{!}} [[phab:T198348|T198348]]
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.


== 2018-07-17 ==
== 2022-04-14 ==
* 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - [[phab:T195312|T195312]], [[phab:T180761|T180761]].
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh


== 2018-07-16 ==
== 2022-04-12 ==
* 21:10 awight: ran namespaceDupes.php on beta enwiki
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 19:56 bearND: (beta): Update mobileapps to {{Gerrit|bed7b29}}
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141


== 2018-07-14 ==
== 2022-04-08 ==
* 21:20 Krenair: redirected security_audit traffic (see [[phab:T72181|T72181]]) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in [[phab:T192996|T192996]])
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds -  [[phab:T179963|T179963]]


== 2018-07-13 ==
== 2022-04-07 ==
* 17:31 Reedy: Reloading Zuul to deploy stuff
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]


== 2018-07-12 ==
== 2022-04-06 ==
* 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
* 20:03 thcipriani: rebooting phabricator
* 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]


== 2018-07-11 ==
== 2022-04-04 ==
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|b5e152d}}
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul


== 2018-07-10 ==
== 2022-04-02 ==
* 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002  ( apt-get clean , apt-get autoremove --purge ,  nuked /srv/jenkins-workspace/workspace/*lint
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
* 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator {{!}} [[phab:T199103|T199103]]
* 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884  as well)
* 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in [[phab:T72792|T72792]])
* 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 {{!}} [[phab:T198171|T198171]]


== 2018-07-09 ==
== 2022-03-31 ==
* 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs ([[phab:T142152|T142152]])
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
* 22:09 Krinkle: Deleting instance deployment-apertium02 - [[phab:T142152|T142152]]
* 19:10 Krenair: deleted deployment-mediawiki06 [[phab:T192996|T192996]]
* 17:35 zeljkof: Reloading Zuul to deploy {{Gerrit|7967dad02843d3692abaf3acfd548047f192c9f6}}
* 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation {{!}} [[phab:T198952|T198952]]
* 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport {{!}} [[phab:T198957|T198957]]
* 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat {{!}} [[phab:T198955|T198955]]
* 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon {{!}} [[phab:T198864|T198864]]


== 2018-07-08 ==
== 2022-03-29 ==
* 20:17 Krinkle: Shutting off deployment-apertium02 ([[phab:T142152|T142152]])
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 18:31 Krenair: deleted deployment-mx [[phab:T184244|T184244]]
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 16:55 Krenair: deleted deployment-redis01 [[phab:T179371|T179371]]
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
* 16:54 Krenair: deleted deployment-redis02 [[phab:T179371|T179371]]
* 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/


== 2018-07-07 ==
== 2022-03-28 ==
* 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]


== 2018-07-06 ==
== 2022-03-27 ==
* 03:53 kart_: Updated cxserver to {{Gerrit|bfc9c84}}
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]


== 2018-07-05 ==
== 2022-03-26 ==
* 06:27 kart_: Updated cxserver to {{Gerrit|f8c71a1}}
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly


== 2018-07-04 ==
== 2022-03-25 ==
* 20:15 Reedy: beta unbroke beta code autodeploy [[phab:T72597|T72597]]
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ [[phab:T192996|T192996]]
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 05:31 Krinkle: beta-scap-eqiad  failing due to mediawiki06 still being a scap target  (despite being shutdown per [[phab:T192996|T192996]])
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord


== 2018-07-03 ==
== 2022-03-24 ==
* 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev {{!}} [[phab:T198076|T198076]]
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 ([[phab:T195312|T195312]])
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
* 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod  ([[phab:T195312|T195312]], [[phab:T194390|T194390]])
* 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - [[phab:T192996|T192996]]


== 2018-07-02 ==
== 2022-03-23 ==
* 20:28 hashar: building docker image releng/tox:0.1.1  , that adds libssl-dev {{!}} [[phab:T198076|T198076]]
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub {{!}} [[phab:T198322|T198322]]
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023


== 2018-06-30 ==
== 2022-03-22 ==
* 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945


== 2018-06-28 ==
== 2022-03-21 ==
* 21:17 hasharAway: castor02: nuking cache of npm/node jobs via  rm -fR /srv/jenkins-workspace/caches/*/*/*node*  (note: other jobs might still have a npm cache) {{!}} [[phab:T198348|T198348]]
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm  and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm {{!}} [[phab:T198348|T198348]]
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]
* 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker {{!}} [[phab:T198348|T198348]]
* 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.


== 2018-06-27 ==
== 2022-03-18 ==
* 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 {{!}} [[phab:T196346|T196346]] [[phab:T198336|T198336]]
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to {{Gerrit|d7221ba}}
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)


== 2018-06-26 ==
== 2022-03-17 ==
* 22:17 Krenair: arming keyholder on deployment-deploy01
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 20:16 Krenair: done the same on -sca02 and -apertium02
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015  (converted to m1.medium instances)
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]
* 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
* 13:18 hashar: cleaned containers on integration-slave-docker-1006


== 2018-06-25 ==
== 2022-03-16 ==
* 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to {{Gerrit|8c76d52}}
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 17:45 Krenair: fixed puppet repo rebasing
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 16:20 hashar: deployment-prep: git gc  on a few repositories under /srv/mediawiki-staging/php-master
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 12:19 kart_: "Beta: Update cxserver to ece5e7a"
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep
* 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice {{!}} [[phab:T196618|T196618]]


== 2018-06-23 ==
== 2022-03-15 ==
* 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up
* 18:26 brennen: gitlab: removed most existing /people groups
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2018-06-22 ==
== 2022-03-14 ==
* 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'    # some instances are causing disk warnings
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 14:46 zeljkof: Reloading Zuul to deploy {{Gerrit|94c38c16c14c59bf6ff2de6c9b693b9a903da1c2}}
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 12:34 hashar: Building docker images for Quibble 0.0.19 {{!}} [[phab:T197687|T197687]]
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium {{!}} [[phab:T197687|T197687]]
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2018-06-21 ==
== 2022-03-11 ==
* 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor {{!}} [[phab:T197698|T197698]]
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
* 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2  https://gerrit.wikimedia.org/r/c/integration/config/+/441349/


== 2018-06-20 ==
== 2022-03-10 ==
* 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck {{!}} [[phab:T197806|T197806]]
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image {{!}} [[phab:T197806|T197806]]
* 20:03 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2018-06-19 ==
== 2022-03-09 ==
* 15:01 hashar: CI docker slaves: docker container prune --force ; docker image prune --force
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php


== 2018-06-18 ==
== 2022-03-08 ==
* 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 20:31 brennen: requiring 2fa for all users under /repos
* 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:07 Amir1: change_tag_def is back now
* 16:05 Amir1: making enwiki in beta clsuter readonly


== 2018-06-15 ==
== 2022-03-07 ==
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|529965bb8c8cfa2bb7539c7c79c9dd55de6c11ac}}
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]
* 08:30 hashar: github: deleted lot of archived/read-only extensions and skins {{!}} [[phab:T180864|T180864]]
* 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
* 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
* 07:51 hashar: cleaning disk space on integration-slave-jessie-1001


== 2018-06-14 ==
== 2022-03-04 ==
* 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 14:26 zeljkof: Reloading Zuul to deploy {{Gerrit|04b1c864f3b9d49e9b3882a1fc4c8fbf272c1bd7}}
* 19:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768068
* 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
* 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
* 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01{{!}} [[phab:T197207|T197207]]


== 2018-06-13 ==
== 2022-03-03 ==
* 19:47 Krenair: powered off deployment-mx [[phab:T184244|T184244]]
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera {{!}} TT186096
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2018-06-12 ==
== 2022-03-02 ==
* 17:40 halfak: deploying ores {{Gerrit|36037b6}}
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 15:27 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/767493
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04


== 2018-06-11 ==
== 2022-02-28 ==
* 21:47 halfak: deploying ores {{Gerrit|6ee8775}}
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean {{!}} [[phab:T196430|T196430]]
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin
* 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *


== 2018-06-10 ==
== 2022-02-26 ==
* 17:03 Krenair: also fixed scap install on -zotero01
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases


== 2018-06-09 ==
== 2022-02-24 ==
* 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from [[phab:T182927|T182927]] to test
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances [[phab:T179371|T179371]]
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2018-06-08 ==
== 2022-02-23 ==
* 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn {{!}} [[phab:T196433|T196433]]
* 20:10 dancy: Updating scap in beta
* 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration {{!}} [[phab:T196442|T196442]]
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/  # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2018-06-07 ==
== 2022-02-22 ==
* 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])
* 17:24 Reedy: unstuck jenkins deployments to beta
* 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
* 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
* 14:16 _joe_: killed "puppet master" process in depoloyment-prep
* 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex {{!}} [[phab:T193844|T193844]]
* 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
* 10:41 hashar: Building Docker images for Quibble 0.0.18
* 09:15 addshore: added wmde-leszek to deployment-prep


== 2018-06-06 ==
== 2022-02-21 ==
* 21:40 bearND: BC: Update mobileapps to {{Gerrit|5ea008c}}
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 20:36 Amir1: deploying {{Gerrit|65ce165}} on ores
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 20:25 Amir1: deploying {{Gerrit|65e979f}} on ores
* 07:27 hashar: Refreshing all Jenkins jobs
* 19:59 bearND: Update mobileapps to {{Gerrit|3bf9be5}} on BC
* 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) {{!}} [[phab:T196108|T196108]]
* 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore  (archived repo) {{!}} [[phab:T196513|T196513]]
* 18:58 thcipriani: @deployment-tin: sudo keyholder arm
* 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow {{!}} [[phab:T186969|T186969]]
* 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ {{!}} [[phab:T196509|T196509]]
* 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS  {{!}} [[phab:T196509|T196509]]


== 2018-06-05 ==
== 2022-02-20 ==
* 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to {{Gerrit|66727b7}}
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
* 12:07 kart_: Update cxserver to {{Gerrit|7fb7671}}


== 2018-06-04 ==
== 2022-02-19 ==
* 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to {{Gerrit|f579f0d}}
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force [[phab:T196401|T196401]]
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 15:44 awight: ORES: Fix [[phab:T194322|T194322]]
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]


== 2018-06-02 ==
== 2022-02-18 ==
* 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}
* 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky


== 2018-06-01 ==
== 2022-02-17 ==
* 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01)  - [[phab:T195314|T195314]]
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, [[phab:T195314|T195314]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 20:51 legoktm: deleting old versions of docker images
* 11:12 hashar: Bringing deployment-deploy03 back
* 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome
* 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
* 20:26 mutante: contint1001 - apt-get clean  got a little bit more disk space
* 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
* 08:45 hashar: Bumping Quibble jobs to 0.0.16
* 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716


== 2018-05-31 ==
== 2022-02-16 ==
* 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ {{!}} [[phab:T187622|T187622]]
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - [[phab:T195314|T195314]]
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs [[phab:T196062|T196062]]
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}
* 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
* 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
* 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
* 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
* 07:16 legoktm: repooled integration-slave-docker-1010
* 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
* 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - [[phab:T195314|T195314]]
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
* 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, [[phab:T195314|T195314]]
* 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722


== 2018-05-30 ==
== 2022-02-15 ==
* 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large ([[phab:T192561|T192561]])
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}


== 2018-05-29 ==
== 2022-02-14 ==
* 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
* 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - [[phab:T193734|T193734]]
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 12:42 zeljkof: Reloading Zuul to deploy {{Gerrit|bcd36d9a2cd2b2ebb1ec77bdf0fcdcc612aeae8d}}
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon
* 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
* 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
* 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
* 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
* 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
* 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
* 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
* 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
* 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
* 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938


== 2018-05-28 ==
== 2022-02-12 ==
* 11:09 hashar: Building Quibble Docker images 0.0.13 {{!}} [[phab:T195634|T195634]]
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type {{!}} [[phab:T195634|T195634]]
* 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems


== 2018-05-27 ==
== 2022-02-10 ==
* 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673


== 2018-05-26 ==
== 2022-02-09 ==
* 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
* 15:22 taavi: deleted shutoff deployment-mx02
* 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours


== 2018-05-25 ==
== 2022-02-08 ==
* 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 21:41 hashar: cleaned up disk space on deployment-tin  . Reenabled jenkins slave.
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03


== 2018-05-24 ==
== 2022-02-07 ==
* 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
* 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
* 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
* 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
* 15:56 zeljkof: Reloading Zuul to deploy {{Gerrit|2e02d97a03e9aaf6a01e287f381934b8a7b03c9a}}
* 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
* 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
* 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846


== 2018-05-23 ==
== 2022-02-04 ==
* 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
* 20:07 bearND: Beta Cluster: Update mobileapps to {{Gerrit|29ebe0f}}
* 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
* 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
* 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363


== 2018-05-22 ==
== 2022-02-03 ==
* 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for [[phab:T194926|T194926]], finished without errors on any of the hosts)
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done [[phab:T194926|T194926]]
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref [[phab:T195312|T195312]]


== 2018-05-21 ==
== 2022-02-02 ==
* 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta


== 2018-05-20 ==
== 2022-02-01 ==
* 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]


== 2018-05-19 ==
== 2022-01-31 ==
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|9f3a185cb7d020757830e546e30e5d5e4ed67978}}
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]
* 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
* 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
* 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845


== 2018-05-18 ==
== 2022-01-28 ==
* 18:31 zeljkof: Reloading Zuul to deploy {{Gerrit|f8e9625622aab96f0718f5fc9655117c8ebc3f55}}
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs [[phab:T194900|T194900]] [[phab:T191050|T191050]]
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`


== 2018-05-17 ==
== 2022-01-27 ==
* 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # [[phab:T299431|T299431]]
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins


== 2018-05-16 ==
== 2022-01-26 ==
* 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs [[phab:T177896|T177896]]
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2018-05-15 ==
== 2022-01-25 ==
* 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 23:22 legoktm: live hacking zuul config on contint1001
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2018-05-14 ==
== 2022-01-24 ==
* 19:41 bearND: BC: Update mobileapps to {{Gerrit|39c16e4}}
* 21:44 Reedy: unstick beta ci jobs
* 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2018-05-11 ==
== 2022-01-22 ==
* 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts


== 2018-05-10 ==
== 2022-01-21 ==
* 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
* 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
* 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
* 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
* 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
* 19:36 legoktm: kicked postmerge queue in jenkins
* 19:10 twentyafterfour: reset jenkins slave on deployment-tin
* 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' {{!}} [[phab:T194376|T194376]]
* 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'


== 2018-05-09 ==
== 2022-01-20 ==
* 22:53 awight: ORES: wheels fixups
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 21:31 awight: Bump ORES wheels
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 21:04 awight: ORES: drafttopic in beta
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2018-05-08 ==
== 2022-01-19 ==
* 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
* 18:20 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 ([[phab:T190717|T190717]])
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]


== 2018-05-07 ==
== 2022-01-18 ==
* 20:19 bearND: Update mobileapps to {{Gerrit|c1f4de6}} in BC
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0  with proper PHP version in the images # [[phab:T299389|T299389]]
* 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2018-05-06 ==
== 2022-01-17 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro


== 2018-05-05 ==
== 2022-01-14 ==
* afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta


== 2018-05-04 ==
== 2022-01-13 ==
* 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2018-05-03 ==
== 2022-01-12 ==
* 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 {{!}} [[phab:T193767|T193767]]
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
* 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159


== 2018-05-02 ==
== 2022-01-11 ==
* 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
* 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
* 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for [[phab:T193611|T193611]]
* 10:43 hashar: Building quibble docker iamges 0.0.11-2  [[phab:T118371|T118371]] [[phab:T193222|T193222]]
* 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
* 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
* 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291


== 2018-05-01 ==
== 2022-01-10 ==
* 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]


== 2018-04-30 ==
== 2022-01-05 ==
* 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble(thanks zeljkof )
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697 to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2018-04-27 ==
== 2022-01-04 ==
* 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins.  Triggered with a custom gear_client.py script from contint1001 . PID 13134
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383


== 2018-04-26 ==
== 2022-01-03 ==
* 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001.  Restarted CI Jenkins.
* 19:00 hashar: Building releng/quibble 0.0.11 docker images
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
* 17:41 mobrovac: stopped cpjobqueue and purging logs
* 16:51 thcipriani: mark integration-slave-jessie-1001 online
* 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline


== 2018-04-25 ==
* 21:27 hasharAway: building releng/quibble docker images @0.0.10
* 21:13 hasharAway: tagged quibble 0.0.10 {{Gerrit|a4e88ae}}
* 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
* 18:55 awight: ORES: Revscoring 2.2.2
* 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769


== 2018-04-24 ==
{{SAL-archives/Release Engineering}}
* 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859


== 2018-04-23 ==
* 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
* 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
* 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to {{Gerrit|b011b2a}}
* 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
* 07:27 hashar: Upgrading Blue Ocean on CI Jenkins  1.4.2 -> 1.5.0
== 2018-04-20 ==
* 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
* 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
* 08:38 hashar: Cleaned integration-slave-docker-1005  disk (deleting workspace and all docker images)
* 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
* 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
* 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
* 00:46 awight: roll back ORES beta to master
* 00:08 awight: Push ORES git-lfs to look at stuff
== 2018-04-19 ==
* 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
* 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
* 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 {{!}} [[phab:T192576|T192576]]
* 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
* 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
* 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
* 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
* 09:41 hashar: building releng/quibble-stretch:0.0.8-2
* 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done [[phab:T192476|T192476]]
== 2018-04-18 ==
* 22:53 eddiegp: started nutcracker on deployment-jobrunner03
* 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref [[phab:T178457|T178457]]
* 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
* 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
* 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
* 20:29 Reedy: reloaded zuul for 427469
* 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to {{Gerrit|fb161d7}}
* 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - [[phab:T192471|T192471]]
* 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
* 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
* 13:18 addshore: added WMDE-Fisch to deployment-prep
* 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
* 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/  [[phab:T192432|T192432]]
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
* 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
* 04:26 legoktm: try once more to refresh nodepool snapshot
* 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
== 2018-04-17 ==
* 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
* 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
* 20:28 hasharAway: building releng/quibble.*:0.0.8 images
* 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
* 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009  (m4executor label)
* 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only  use of that role in WMCS
* 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only  use of that role in WMCS
* 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)
== 2018-04-16 ==
* 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
* 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
* 22:25 legoktm: kick stuch beta-update jenkins jobs
* 20:35 mdholloway: restarted restbase and mobileapps services for testing ([[phab:T192287|T192287]])
* 18:58 awight: Update ORES editquality; [[phab:T185903|T185903]]
* 13:13 _joe_: also shutting down deployment-jobrunner02 [[phab:T192071|T192071]]
* 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
* 12:48 _joe_: turning off deployment-mediawiki{04,05} [[phab:T192071|T192071]], will be deleted by EOW
* 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837
== 2018-04-13 ==
* 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
* 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
* 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
* 08:48 eddiegp: debugging [[phab:T189493|T189493]] on beta
* 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state [[phab:T192071|T192071]]
* 07:22 legoktm: restarting jenkins
* 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
* 07:01 hashar: rebuilding quibble containers to use 0.0.7
* 00:53 awight: ORES: Test large file in LFS
== 2018-04-12 ==
* 23:01 awight: Try gerrit-based submodules for ORES, [[phab:T180627|T180627]]
* 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
* 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
* 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep ([[phab:T192071|T192071]])
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
* 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
* 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver [[phab:T192071|T192071]]
* 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/
== 2018-04-11 ==
* 21:04 awight: ORES experiment with git-lfs, [[phab:T180627|T180627]]
* 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446
== 2018-04-10 ==
* 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module ([[phab:T184854|T184854]])
* 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package {{!}} [[phab:T191771|T191771]]
== 2018-04-09 ==
* 19:50 awight: Redundant virtualenv for ORES
* 18:06 awight: Restore to ORES master branch
* 17:17 awight: Test git-lfs in ORES
* 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix [[phab:T191786|T191786]] (success: Notice: Applied catalog in 27.11 seconds)
== 2018-04-06 ==
* 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for [[phab:T173887|T173887]]
* 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 ([[phab:T190717|T190717]])
* 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep ([[phab:T191578|T191578]])
== 2018-04-05 ==
* 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
* 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt {{!}} mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
* 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak [[phab:T173887|T173887]]
* 17:08 bearND: Update mobileapps to {{Gerrit|dbc0687}}
* 17:05 hashar: docker: building releng/quibble:0.0.5
* 17:00 bearND: Update mobileapps to {{Gerrit|2d5ab5b}} on beta
* 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
* 07:52 moritzm: removed  unused/defunct  deployment-videoscaler01 from deployment-prep ([[phab:T191293|T191293]])
* 07:52 moritzm: removed  unused/defunct  deployment-tmh01 from deployment-prep ([[phab:T191293|T191293]])
* 07:47 hashar: deployment-prep: made Muehlenhoff an admin
== 2018-04-04 ==
* 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
* 21:24 awight: Roll back beta ORES
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to {{Gerrit|2d5ab5b}}
* 20:12 awight: Try dsh scap config for ORES
* 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
* 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
* 15:30 addshore: reload zuul for  https://gerrit.wikimedia.org/r/423652
* 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
* 11:58 hashar: Building releng/quibble-stretch:0.0.4
* 11:58 hashar: Building releng/quibble:0.0.4
* 07:11 hashar: deployment-prep: adding EddieGP as a member
== 2018-04-03 ==
* 09:49 hashar: building releng/quibble:0.0.3
* 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
== 2018-04-02 ==
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to {{Gerrit|58a0a88}}
== 2018-03-31 ==
* 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
== 2018-03-30 ==
* 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis ([[phab:T189694|T189694]])
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} [[phab:T191077|T191077]]
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} TT191077
== 2018-03-29 ==
* 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript  updateCollation.php --wiki=fawiki ([[phab:T190965|T190965]])
* 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
* 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
* 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1  https://gerrit.wikimedia.org/r/#/c/422959/
* 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
* 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
* 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
* 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host ([[phab:T191029|T191029]])
* 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug [[phab:T184234|T184234]]" due to git sync failures
== 2018-03-28 ==
* 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
* 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
* 20:45 bearND: Update mobileapps to {{Gerrit|a5833a0}} on BC
* 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
* 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
* 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)
== 2018-03-27 ==
* 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 {{!}} https://gerrit.wikimedia.org/r/#/c/422168/
* 08:30 kart_: Update cxserver to {{Gerrit|9e8ebda}}
* 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs
== 2018-03-26 ==
* 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to {{Gerrit|534f95d}}
* 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 {{!}} [[phab:T190393|T190393]]
* 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)
== 2018-03-23 ==
* 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin
== 2018-03-22 ==
* 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)
== 2018-03-21 ==
* 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
* 23:22 bd808: Raised security group quota from 20 to 40
* 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
* 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to {{Gerrit|e6b50a0}}
* 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
* 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
* 17:52 hashar: deployment-prep:  scap sync-file  docroot/wwwportal/portal '[[phab:T173887|T173887]]'
* 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins
== 2018-03-20 ==
* 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
* 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to {{Gerrit|634a15f}}
* 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
* 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
* 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
* 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
== 2018-03-19 ==
* 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
* 16:57 awight: ORES beta service is restored.
* 16:45 awight: Put ORES-beta back onto master branch
* 16:43 awight: ORES-beta has been down since January.
* 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
* 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
* 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
* 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1  {{!}} https://gerrit.wikimedia.org/r/#/c/420311/ {{!}} [[phab:T190032|T190032]]
* 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
* 07:43 kart_: Add cx_version to cx_translations table ([[phab:T190009|T190009]])
== 2018-03-18 ==
* 22:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/420230
* 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344
== 2018-03-16 ==
* 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
* 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
* 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
* 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/*  # some pack of 6GB belonged to root
* 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
* 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock [[phab:T189859|T189859]]
* 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance
== 2018-03-15 ==
* 17:15 bearND: Update mobileapps to {{Gerrit|c5e1522}}
* 16:43 Amir1: created deployment-ores01 to use stretch
* 16:36 Amir1: deleting deployment-ores01 to reimage
* 15:47 awight: ORES with git-lfs, scap config
* 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
* 00:21 awight: ORES with git-lfs
== 2018-03-14 ==
* 23:30 awight: new ORES submodule, pre-git-lfs
* 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to {{Gerrit|9f4a80c}}
* 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
* 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
* 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
* 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
* 12:16 kart_: update cxserver to {{Gerrit|c355d0c}}
* 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
* 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
* 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch  to Nodepool (they use too much disk space) {{!}} [[phab:T189365|T189365]]
== 2018-03-13 ==
* 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:08 awight: Enable Extension:JADE, [[phab:T176333|T176333]]
* 11:26 kart_: Update cxserver to {{Gerrit|bd2ccfc}}
* 10:39 kart_: Update cxserver to {{Gerrit|fd2c4be}}
* 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 {{!}} [[phab:T188112|T188112]]
* 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ {{!}} [[phab:T188112|T188112]]
* 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3  (HACK) {{!}} [[phab:T188112|T188112]]
* 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 {{!}} [[phab:T188112|T188112]]
== 2018-03-12 ==
* 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to {{Gerrit|5c90db7}}
* 20:09 bd808: Forced puppet run on deployment-logstash2 for {{gerrit{{!}}418986}}
* 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
* 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
* 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
* 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ {{!}} [[phab:T155840|T155840]]
* 12:24 hashar: Jenkins: uninstalled the "cvs" plugin
== 2018-03-10 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - [[phab:T189365|T189365]]
== 2018-03-09 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0  https://gerrit.wikimedia.org/r/#/c/417960/
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container {{!}} [[phab:T188686|T188686]]
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343
== 2018-03-08 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 .  Upgrade packages and reboot.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 18:36 bearND: Update mobileapps to {{Gerrit|afb0167}}
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc  # 1.4GBytes
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 10:27 hashar: Deploy docker images for /deploy repositories {{!}} https://gerrit.wikimedia.org/r/#/c/417217/
== 2018-03-07 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" {{!}} [[phab:T188686|T188686]]
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 02:28 legoktm: manually triggering jenkins jobs
== 2018-03-06 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets  parent permission group to scoring/ores
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic  scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from  scoring/ores itself inheriting from scoring/
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to {{Gerrit|afbe9af}}
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h
== 2018-03-05 ==
* 16:38 Reedy: deleted the stack traces too
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-03-04 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 02:12 Krenair: Regenerated captcha images for [[phab:T164047|T164047]]
== 2018-03-02 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs
== 2018-03-01 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 02:11 legoktm: manually queued jenkins jobs
== 2018-02-28 ==
* 16:31 legoktm: manually queued jenkins jobs
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 07:29 legoktm: mass queuing jenkins jobs again
== 2018-02-27 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 -  https://gerrit.wikimedia.org/r/#/c/415102/
* 10:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d9ed9d4dded7d646fc9c4b54155613eef99752a9}}
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.
== 2018-02-26 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx ([[phab:T188288|T188288]])
== 2018-02-24 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
== 2018-02-23 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 11:50 hashar: deployment-mediawiki04 :  apt-get clean && apt-get autoremove --purge
* 11:50 hashar: deployment-mediawiki04 :  rm /var/cache/hhvm/*.sq3  and restarting hhvm
* 11:48 hashar: deployment-mediawiki04 is out of disk space on /  causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only {{!}} [[phab:T187995|T187995]]
== 2018-02-22 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/413374/
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to {{Gerrit|a1339a9}}
== 2018-02-21 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0  and reloading Zuul {{!}} https://gerrit.wikimedia.org/r/413227
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 16:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d90d617e0794725280f9f2660effabea74533cbe}}
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis
== 2018-02-20 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
== 2018-02-19 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions
== 2018-02-18 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
== 2018-02-16 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely {{!}} https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-02-15 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/  (no jobs touched)
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only {{!}} [[phab:T187428|T187428]]
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255
== 2018-02-14 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to {{Gerrit|f23519f}}
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB
== 2018-02-13 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 10:51 moritzm: shutdown deployment-videoscaler01 ([[phab:T187063|T187063]])
* 10:49 moritzm: shutdown deployment-tmh01 ([[phab:T187063|T187063]])
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/  # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089
== 2018-02-12 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for [[phab:T186870|T186870]]
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. [[phab:T54919|T54919]]
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - [[phab:T186870|T186870]])
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - [[phab:T186870|T186870]])
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - [[phab:T186870|T186870]]
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/
== 2018-02-10 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - [[phab:T186870|T186870]])
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
== 2018-02-09 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 [[phab:T186856|T186856]]
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. {{!}} [[phab:T179378|T179378]] [[phab:T186856|T186856]]
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for {{Gerrit|I73a3142271d76f9640005752b1a336123b921b9f}}
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - [[phab:T186856|T186856]]
* 00:59 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
* 00:58 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
== 2018-02-08 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test {{!}} [[phab:T179552|T179552]]
== 2018-02-07 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230
== 2018-02-06 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix  [[phab:T186381|T186381]]
* 21:14 legoktm: restarted zuul due to patch being stuck ([[phab:T186381|T186381]])
* 19:25 hashar: Restarted Zuul due to [[phab:T186381|T186381]]
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480
== 2018-02-05 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 16:09 mdholloway: mobileapps deployment to BC failed with error ([[phab:T186532|T186532]])
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 {{!}} [[phab:T186494|T186494]]
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 {{!}} [[phab:T186494|T186494]]
== 2018-02-04 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
== 2018-02-03 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins
== 2018-02-01 ==
* 16:14 Amir1: deleting deployment-sca03 ([[phab:T184501|T184501]])
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-31 ==
* 22:01 mdholloway: updated mobileapps to {{Gerrit|3d717fa}} on beta cluster
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-30 ==
* 20:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/404778
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
== 2018-01-29 ==
* 23:24 awight: Experiment with versioned ORES venv, [[phab:T181071|T181071]]
== 2018-01-24 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
== 2018-01-23 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from  integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase {{!}} https://wikitech.wikimedia.org/wiki/Keyholder
== 2018-01-22 ==
* afk: restarting jenkins
== 2018-01-20 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
== 2018-01-19 ==
* 17:23 zeljkof: Reloading Zuul to deploy {{Gerrit|25cab6fdb53f1e5d284192159c43f0ce28f6e604}}
* 17:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7d6b4eecc6d7854de7db6714bcfd006d860f3e28}}
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
== 2018-01-18 ==
* 18:29 bearND: (beta): Update mobileapps to {{Gerrit|2690899}}
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state
== 2018-01-17 ==
* 16:24 zeljkof: Reloading Zuul to deploy {{Gerrit|5f757310f499a6a2cdf036dde3d258046377186f}}
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for [[phab:T185024|T185024]]  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
== 2018-01-16 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images{{!}}grep wmfreleng{{!}}awk '{print \$3}'{{!}}sort{{!}}uniq{{!}}xargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images {{!}} https://gerrit.wikimedia.org/r/#/c/404422/
== 2018-01-15 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy {{Gerrit|ff0a02decf1c75b2cef4f438c622d548c45fd270}}
* 17:14 zeljkof: Reloading Zuul to deploy {{Gerrit|fff6431628437cd9b3ccd5bafe5b58480b2a3714}}
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for [[phab:T180841|T180841]]
* 13:37 kart_: Ran update.php on metawiki betacluster ([[phab:T180841|T180841]])
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git {{!}} [[phab:T174338|T174338]]
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo
== 2018-01-12 ==
* 14:11 zeljkof: Reloading Zuul to deploy {{Gerrit|3816837e6d70ee7eb276c9b00034a5d2012a8afd}}
== 2018-01-11 ==
* 17:50 tgr: added Groovier1 to project members for [[phab:T158909|T158909]]
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403654
* 13:45 hashar: Migrate  composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403647
== 2018-01-10 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers {{!}} https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826
== 2018-01-09 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' {{!}} [[phab:T158243|T158243]] [[phab:T162191|T162191]]
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git {{!}} [[phab:T183569|T183569]]
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki ([[phab:T184276|T184276]])
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores ([[phab:T184282|T184282]])
* 08:48 Amir1: stopping ores services in deployment-sca03 ([[phab:T184282|T184282]])
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 ([[phab:T184282|T184282]])
== 2018-01-08 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
== 2018-01-05 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
== 2018-01-03 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3  the version in apt.wm.o
* 19:09 hashar: apt-get upgrade on deployment-tin  . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)
== 2018-01-01 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for [[phab:T183862|T183862]]
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki ([[phab:T183862|T183862]])
* 19:56 Amir1: restarting ores services in deployment-sca03 ([[phab:T183862|T183862]])
== 2017-12-27 ==
* 19:42 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2017-12-25 ==
* 04:54 legoktm: deployed https://gerrit.wikimedia.org/r/400153 https://gerrit.wikimedia.org/r/400103
== 2017-12-24 ==
* 16:07 Amir1: ladsgroup@deployment-tin$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=wikidatawiki --load-from https://en.wikipedia.beta.wmflabs.org/w/api.php ([[phab:T183633|T183633]])
== 2017-12-23 ==
* 14:10 addshore: made ladsgroup owner of the github org
== 2017-12-22 ==
* 22:39 thcipriani: integration-slave-jessie-1004 removed mediawiki-core-jsduck, mwgate-php55lint, mediawikicore-php55lint as /srv mount was full [[phab:T179963|T179963]]
* 16:37 hashar: cancelled update of npm and npm-test images. npm is broken when used with a proxy {{!}} https://gerrit.wikimedia.org/r/#/c/399837/
* 16:12 hashar: rebuilding npm and npm-test docker images https://gerrit.wikimedia.org/r/#/c/388450/
* 09:17 hashar: Fixed mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs that used a wrong Docker image name {{!}} https://gerrit.wikimedia.org/r/399789
* 09:08 hashar: updating mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs to the new ci-src-setup docker image https://gerrit.wikimedia.org/r/399754
* 09:08 hashar: updating mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs to the new ci-src-setup docker image mediawiki-core-php70-phan-docker mwext-php70-phan-docker
== 2017-12-21 ==
* 16:54 awight: Update ORES to {{Gerrit|eb0f776bb}}
* 15:14 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399612/ "fix hhvm docker-pkg definitions"
* 14:50 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399611/ "fix php55 definitions"
* 14:20 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399609/ "fix zuul-cloner docker-pkg definition"
* 13:51 hashar: wikitech: change email of PortalsBuilder user from releng@lists.wikimedia.org to portals@lists.wikimedia.org {{!}} Credentials come from https://phabricator.wikimedia.org/D872 {{!}} [[phab:T179694|T179694]]
== 2017-12-20 ==
* 20:51 hashar: Rebuilding hhvm Docker containers https://gerrit.wikimedia.org/r/399406 {{!}} [[phab:T183324|T183324]]
* 18:00 RoanKattouw: Importing dump from deployment-db03 on deployment-db04
* 15:30 RoanKattouw: Restarting dump again, failed due to lack of disk space
* 15:07 RoanKattouw: Dropped invalid view labswiki.updates, restarting dump
* 14:59 RoanKattouw: Dumping all databases on deployment-db03 so I can restore replication on deployment-db04. This may cause MediaWiki writes to fail while the dump runs
== 2017-12-19 ==
* 20:10 RoanKattouw: (Earlier today) Depooled deployment-db04, it needs fixing after replication broke badly. It's out of sync with deployment-db03, where I manually fixed inconsistencies
* 18:11 awight: Update beta ORES service to {{Gerrit|f109792}}
* 17:00 awight: Disable ORES UI for beta wikidatawiki, [[phab:T183266|T183266]]
* 15:58 zeljkof: Reloading Zuul to deploy {{Gerrit|2f514e4d9731c1793be1b95bb601315c5966ffd6}}
* 15:37 zeljkof: Reloading Zuul to deploy {{Gerrit|fb9327ef2be7ca9ff50f74095b87fa6d449f09ef}}
* 15:34 hashar: Switched tox jobs from wmfreleng/tox to docker-registry.wikimedia.org/releng/tox {{!}} https://gerrit.wikimedia.org/r/388449
* 14:50 zeljkof: Reloading Zuul to deploy {{Gerrit|2fbfc1d90aefd0c01c83d1eed31b59af62339253}}
* 12:12 hashar: CI: switching mwgate-composer-php70 job from Nodepool to Docker {{!}} https://gerrit.wikimedia.org/r/#/c/398921/
* 11:59 hashar: CI: switching composer-php55 / composer-package-php55 jobs from Nodepool to Docker {{!}} https://gerrit.wikimedia.org/r/#/c/398920/
* 11:30 hashar: building php55 docker images on contint1001 {{!}} https://gerrit.wikimedia.org/r/#/c/397634/
== 2017-12-18 ==
* 19:43 thcipriani: checkout master on deployment-tin:/srv/mediawiki-staging/php-master to fix beta-code-update-eqiad
* 17:36 addshore: paused beta-code-update-eqiad for a while while I test something
* 09:58 hashar: gerrit: deleted unused user-metrics-2 repo. Been created 4 years and 7 months ago but otherwise unused
* 09:53 hashar: gerrit: deleted wikibase/data-model and wikibase/data-model-services . They are on Github https://github.com/wmde/WikibaseDataModel and https://github.com/wmde/WikibaseDataModelServices
== 2017-12-13 ==
* 22:08 mdholloway: deployed mobileapps@ddddebb to BC
* 18:25 thcipriani: failed Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/394551/ permissions errors with fabfile.py
* 18:18 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/394551/
* 17:24 awight: Install aspell-is for ORES
* 17:06 awight: Deploy ORES service {{Gerrit|b67bba7}}
* 00:38 mdholloway: deployed mobileapps@bfc3588 to BC
== 2017-12-12 ==
* 18:22 mdholloway: deployed mobileapps@5b8796d to BC
* 15:34 addshore: deploy zuul for parameter_functions update
* 15:27 addshore: unblocked beta scaps and files syncs on jenkins
== 2017-12-11 ==
* 23:18 mdholloway: deployed mobileapps@e290b17 to BC
* 21:06 mdholloway: deployed mobileapps@61ca333 to BC
* 19:03 andrewbogott: upgraded deployment-puppetmaster02 to puppet v4
* 17:24 hashar: Update node-6-docker jobs so the entry point recognizes setting  NPM_RUN_SCRIPT=doc {{!}}  https://gerrit.wikimedia.org/r/#/c/397576/
* 10:00 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-GitHub {{!}} [[phab:T182231|T182231]]
* 09:59 hashar: github: archiving https://github.com/wikimedia/mediawiki-extensions-SwiftCloudFiles - [[phab:T182384|T182384]]
== 2017-12-09 ==
* 03:13 legoktm: deployed https://gerrit.wikimedia.org/r/396486
== 2017-12-08 ==
* 21:20 legoktm: deployed https://gerrit.wikimedia.org/r/396480
* 19:47 legoktm: deployed https://gerrit.wikimedia.org/r/396453
* 18:09 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=enwiki --fix - [[phab:T182356|T182356]]
* 13:01 hashar: github: disable vulnerability alerts and archived https://github.com/wikimedia/wikimedia-lobbypop/ {{!}} [[phab:T180878|T180878]]
* 13:01 hashar: github: disable vulnerability alerts and archived https://github.com/wikimedia/labs-tools-Wikimedia-Emoji-Bot/ {{!}} [[phab:T180878|T180878]]
* 13:00 hashar: github: disable vulnerability alerts for the archived repo https://github.com/wikimedia/labs-tools-Wikimedia-Emoji-Bot/
* 11:06 zeljkof: Reloading Zuul to deploy {{Gerrit|1faf444d1710eecb43edf5a40c58377d31d61713}}
== 2017-12-07 ==
* 20:53 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=dewiki --fix
* 20:25 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=deploymentwiki --fix --add-prefix=Broken/
* 20:18 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.loginidol.org
* 17:56 mdholloway: deployed mobileapps@71f581c to beta cluster
* 10:09 hashar: integration: sudo cumin --force 'name:integration-slave-jessie-100*' /usr/local/sbin/run-puppet-agent  {{!}} https://gerrit.wikimedia.org/r/395961
* 10:06 hashar: integration: unbroke puppet on some permanent slaves. Add been broken since Nov 29th ~ 19:50UTC {{!}}  https://gerrit.wikimedia.org/r/#/c/395961/
* 09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase.  wmf/* and REL branches are going to be broken though {{!}} https://gerrit.wikimedia.org/r/395704 {{!}} [[phab:T181838|T181838]]
== 2017-12-06 ==
* 21:43 awight: Update ORES to {{Gerrit|42cf532}}
* 17:54 gehel: logstash upgrade on deployment-logstash2 completed, 5 minutes of logs lost during upgrade - [[phab:T178412|T178412]]
* 17:26 gehel: upgrading ELK on deployment-logstash2 - [[phab:T178412|T178412]]
* 16:48 Hauskatze: Ran cleanupSpam.php on deploymentwiki
* 10:03 hashar: docker push wmfreleng/npm:v2017.12.06.09.55 wmfreleng/npm-stretch:v2017.12.06.09.55 wmfreleng/npm-test:v2017.12.06.09.55 wmfreleng/npm-test-stretch:v2017.12.06.09.55 !!! wmfreleng/npm-browser-test:v2017.12.06.09.55 {{!}} https://gerrit.wikimedia.org/r/#/c/395555/
== 2017-12-05 ==
* 12:18 hashar: deployment-videoscaler01: rm /var/log/hhvm/*  /var/log/apache2/*  . Restarted apache2/hhvm/syslog
* 12:16 hashar: integration: sudo cumin --force '*' 'apt-get clean'
* 12:16 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'
* 12:15 hashar: deployment-videoscaler01: apt-get clean  to free up disk space
* 08:51 hashar: jenkins: adding global property FORCE_COLOR=1 to https://integration.wikimedia.org/ci/configure . That forces webdriver.io to spurts color in the Jenkins console when not using a TTY
* 06:37 kart_: Updated cxserver to {{Gerrit|1693bcf}}
== 2017-12-04 ==
* 17:44 awight: ORES: Try enwiki models on simplewiki, [[phab:T181848|T181848]] ({{Gerrit|6baed71}})
== 2017-12-03 ==
* 21:27 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * # to clear out full /srv
== 2017-12-01 ==
* 21:15 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/394655
* 13:46 godog: deployment-prep bounce elasticsearch on logstash2 to test jmx_exporter
* 11:55 hashar: updating *npm-browser-node-6-docker jobs to use a new container based on Stretch with Chromium/Firefox {{!}} https://gerrit.wikimedia.org/r/#/c/394340/ {{!}} [[phab:T179360|T179360]]
* 10:08 hashar: docker push wmfreleng/npm-browser-test-stretch:v2017.11.30.21.30 && docker push wmfreleng/npm-browser-test-stretch:latest {{!}} https://gerrit.wikimedia.org/r/#/c/394340/ {{!}} [[phab:T179360|T179360]]
* 08:40 hashar: rebased operations/puppet on deployment-prep and integration puppetmasters
* 08:40 hashar: deployment-prep: removed a hack to puppetmaster environments/future/environment.conf containing:  parser = future \n manifest = $confdir/manifests\n
* 08:38 hashar: integration: removed a hack to puppetmaster environments/future/environment.conf containing:  parser = future \n manifest = $confdir/manifests\n
== 2017-11-30 ==
* 23:08 addshore: turned beta-scap-eqiad back on
* 23:03 addshore: reload zuul to deploy Revert "Use gate-and-submit-swat for mediawiki-config" [integration/config] - https://gerrit.wikimedia.org/r/394484
* 22:58 addshore: also reloaded with hashar Switch ArticlePlaceholder to npm-browser-test & Remove mwgate-npm-node-6-jessie
* 22:57 addshore: reloaded zuul for Use gate-and-submit-swat for mediawiki-config [integration/config] - https://gerrit.wikimedia.org/r/394464
* 21:05 hashar: docker push wmfreleng/npm-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-stretch:latest && docker push wmfreleng/npm-test-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-test-stretch:latest {{!}} https://gerrit.wikimedia.org/r/#/c/394338/ {{!}} [[phab:T179360|T179360]]
* 20:50 addshore: temp disable beta-scap-eqiad so that it doesnt block me doing my own scaps
* 18:59 bd808: Testing stashbot fix for double phab logging ([[phab:T181731|T181731]])
* 17:49 anomie: Finished running cleanupUsersWithNoId.php on Beta Cluster for [[phab:T181731|T181731]]
* 16:58 anomie: Running cleanupUsersWithNoId.php on Beta Cluster, see [[phab:T181731|T181731]]
== 2017-11-29 ==
* 21:27 awight: Update ores submodule, for RevIdScorer statistics
* 21:17 awight: deployment-prep Verbose logging for ORES Celery
* 14:32 chasemp: git pull on /var/lib/git/labs/private and resolve one merge conflict.  (the root key file is too old here)
* 09:18 hashar: gerrit: forcing replication: ssh -p 29418 hashar@gerrit.wikimedia.org replication start operations/software/druid_exporter  # [[phab:T181219|T181219]]
* 09:14 hashar: github: created wikimedia/operations-debs-contenttranslation-apertium-crh-tur and wikimedia/operations-debs-prometheus-openldap-exporter
* 09:08 hashar: github: created repo operations-software-druid_exporter {{!}} [[phab:T181219|T181219]]
* 03:56 legoktm: deleted all workspaces on integration-slave-jessie-1003 /srv ran out of space
* 03:23 Krinkle: Jenkins jobs for mediawiki-core-php55lint consistently failing on integration-slave-jessie ("git: stderr: error: failed to write..")
* 00:02 halfak: deploy-prep awight enabled ORES service
* 00:01 halfak: deploy-prep awight disabled ORES service
== 2017-11-28 ==
* 17:42 awight: Remove stale ORES customizations for the beta cluster.
* 17:31 awight: Remove beta cluster customizations for ORES
== 2017-11-27 ==
* 19:06 awight: Update beta ORES to latest, {{Gerrit|e58bfbf}}
* 08:38 hashar: reactivating https://phabricator.wikimedia.org/source/iegreview/ , it still developped https://phabricator.wikimedia.org/D894
== 2017-11-24 ==
* 08:16 hashar: pooling integration-slave-docker-1003 again {{!}} [[phab:T179378|T179378]]
* 08:14 hashar: nodepool: Image snapshot-ci-jessie-1511510623 in wmflabs-eqiad is ready
* 08:13 hashar: upgrading blubber on contint2001
* 08:03 hashar: nodepool: manually rebuilding snapshot-ci-jessie
== 2017-11-23 ==
* 19:34 hashar: migrating pywikibot/core jobs to Docker https://gerrit.wikimedia.org/r/#/c/393091/
* 19:34 hasharAway: migrating pywikibot/core jobs to Docker https://gerrit.wikimedia.org/r/#/c/393091/
== 2017-11-22 ==
* 18:55 greg-g: beta update jobs are back
* 18:48 greg-g: hung beta updates, doing the monthly dance: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 16:58 halfak: deploying ores-prod-deploy:5084251 [[phab:T181168|T181168]]
== 2017-11-21 ==
* 23:29 mdholloway: deployed mobileapps@52d6a83 on the beta cluster
* 21:28 TabbyCat: deployment-prep Ran cleanupSpam.php on deploymentwiki.
* 18:38 mdholloway: deployed mobileapps@9d1602d on the beta cluster
* 17:06 hasharAway: docker push wmfreleng/tox-cergen:v2017.11.21.16.52 {{!}} https://gerrit.wikimedia.org/r/392678 {{!}} For https://integration.wikimedia.org/ci/job/cergen-tox-docker/ which pass !
* 13:24 hashar: gerrit: adding Jdrewniak to wmf-deployment group https://gerrit.wikimedia.org/r/#/admin/groups/21,members {{!}} [[phab:T180639|T180639]]
* 13:08 hashar: gerrit: created wikimedia/portals/deploy  https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/portals/deploy  for jan_drewniak {{!}} [[phab:T180777|T180777]]
* 13:02 hashar: docker push wmfreleng/ci-src-setup:v2017.11.21.12.57 && docker push wmfreleng/ci-src-setup:latest {{!}} https://gerrit.wikimedia.org/r/392632 {{!}} [[phab:T177684|T177684]]
* 02:42 Krinkle: Adding relative time to [[Deployments]] calendar (Common.js), e.g. "4 hours from now" or "soon"
== 2017-11-20 ==
* 15:57 hashar: gerrit: deleted operations/network-diagrams mostly empty and no changes. Created back in 2012.
* 15:03 hashar: integration: pass all environment variables to the docker run commands {{!}} https://gerrit.wikimedia.org/r/#/c/390432/ {{!}} [[phab:T177684|T177684]]
* 10:06 hashar: nodepool: manually deleted left over instances ci-jessie-wikimedia-894187 and ci-jessie-wikimedia-894188 . Jenkins fails to ssh to it and they were left ready for 72 hours.
* 10:05 hashar: deployment-phab : set hiera 'phabricator_cluster_search: []'  trying to unblock puppet and soft rebooted the instance {{!}} [[phab:T180935|T180935]]
* 09:39 hashar: deployment-prep added missing key between_bytes_timeout to cache::app_def_be_opts  for deployment-cache-text04 and deployment-cache-upload04 {{!}} [[phab:T180935|T180935]]
* 09:29 hashar: deployment-tin:  apt-mark hold scap  {{!}} the apt-repo on deployment-tin is out of date {{!}} [[phab:T180935|T180935]]
== 2017-11-16 ==
* 23:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/391902
* 00:10 thcipriani: removed old differential-docker-test images on integration-slave-docker-1001
== 2017-11-15 ==
* 21:05 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/391079/6
* 17:29 thcipriani: updating docker-pkg dockerfiles on contint1001 for https://gerrit.wikimedia.org/r/#/c/388448/
* 09:55 addshore: created wmf/1.31.0-wmf.8 branch of Wikidata extension repo [[phab:T180539|T180539]]
* 07:30 Krinkle: Aborting jobs in 'test' pipeline for backport REL commits that are already merged meanwhile in 'submit' pipeline
== 2017-11-13 ==
* 23:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/391136/2
== 2017-11-10 ==
* 22:17 hashar: docker push wmfreleng/npm:v2017.11.10.22.15 && docker push wmfreleng/npm:latest && docker push wmfreleng/npm-test:v2017.11.10.22.15 && docker push wmfreleng/npm-test:latest {{!}} BABEL_CACHE_PATH {{!}} [[phab:T179425|T179425]]
* 21:39 hashar: docker push wmfreleng/tox-pyspark:v2017.11.10.21.37 && docker push wmfreleng/tox-pyspark:latest {{!}} {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint  {{!}} [[phab:T176747|T176747]]
* 21:36 hasharAway: docker push wmfreleng/tox:v2017.11.10.21.35 && docker push wmfreleng/tox:latest {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint  {{!}} [[phab:T176747|T176747]]
* 21:36 hasharAway: docker push wmfreleng/tox:v2017.11.10.21.35 && docker push wmfreleng/tox:latest {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint
* 16:18 hashar: docker push wmfreleng/tox:v2017.11.10.16.17 && docker push wmfreleng/tox:latest  for https://gerrit.wikimedia.org/r/#/c/388084/
* 12:39 hashar: Updated Jenkins tox jobs {{!}} https://gerrit.wikimedia.org/r/#/c/389924/ {{!}} Does not quite fix [[phab:T176747|T176747]] yet thoughaaaaaaaaaaaa
* 11:56 hashar: docker push wmfreleng/tox:v2017.11.10.11.49 && docker push wmfreleng/tox:latest && docker push wmfreleng/tox-pyspark:v2017.11.10.11.49 && docker push wmfreleng/tox-pyspark:latest {{!}} https://gerrit.wikimedia.org/r/#/c/389924/
== 2017-11-09 ==
* 21:45 legoktm: deployed https://gerrit.wikimedia.org/r/390325
* 16:54 zeljkof: Reloading Zuul to deploy {{Gerrit|baceaeb09d5b924a323e372559de00e29fc5e698}}
* 15:20 hashar: docker push wmfreleng/npm-test:v2017.11.09.15.15 {{!}} https://gerrit.wikimedia.org/r/390261 {{!}} [[phab:T176747|T176747]]
* 12:52 hashar: Jenkins: fixed/changed the global git user.name / user.email  (now: "Wikimedia CI" and "releng@lists.wikimedia.org" )
* 10:59 hashar: Drop jsduck doc/ri from npm/npm-test images {{!}} https://gerrit.wikimedia.org/r/#/c/389943 {{!}} docker push wmfreleng/npm:v2017.11.09.10.57 && docker push wmfreleng/npm-test:v2017.11.09.10.57
* 07:53 hasharAway: Killed stuck containers wmfreleng/npm-test on integration-slave-docker-1001  - [[phab:T176747|T176747]]
== 2017-11-08 ==
* 13:43 Reedy: ran apt-get clean{{!}}autoclean on deplyoment-mediawiki04 to free up some space
== 2017-11-07 ==
* 18:45 twentyafterfour: cowboy-committed and pushed rMSCAc1f2ac2 to hopefully unbreak `scap deploy` in beta
* 17:56 legoktm: integration-slave-jessie-1003 /srv full, legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mwgate-* mediawiki-*
* 17:27 hashar: Image snapshot-ci-jessie-1510074928 in wmflabs-eqiad is ready - [[phab:T179772|T179772]]
* 17:15 hashar: Updating Nodepool snapshot to get php5.5-zip  - [[phab:T179772|T179772]]
* 16:15 hashar: Created portalsbuilder in Gerrit, generated a ssh key pair for it and stored in Jenkins credentials store - [[phab:T179694|T179694]]
* 15:15 hashar: Created VPS account "PortalsBuilder"  - [[phab:T179694|T179694]]
== 2017-11-06 ==
* 23:49 thcipriani: ssh-keyscan deployment-videoscaler01.deployment-prep.eqiad.wmflabs >> /etc/ssh/ssh_known_hosts
* 22:29 hashar: killed stuck npm Docker containers on integration-slave-docker-1002 (due to  [[phab:T176747|T176747]] ). Pooled the instance back, the slowness it experienced is probably not related to labvirt CPU usage ( [[phab:T179378|T179378]] )
* 20:35 Amir1: deploy ores:93e8846 in beta cluster
* 16:02 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/388546/ and https://gerrit.wikimedia.org/r/#/c/389463/
== 2017-11-03 ==
* 13:51 hashar: pooled integration-slave-docker-1004 and integration-slave-docker-1007
* 13:30 hashar: Unpool integration-slave-docker-1002 and integration-slave-docker-1003 . They are slow CPU wise, most probably due to the underlying labvirt being CPU starved. - [[phab:T179378|T179378]]
* 12:38 hashar: [[phab:T179593|T179593]] generate doc for cumin@v1.2.2 : contint1001$ zuul enqueue-ref --trigger gerrit --pipeline publish --project operations/software/cumin --ref refs/tags/v1.2.2 --newrev {{Gerrit|f7453873ff7c242743a22b119fc5e2171321c38b}}
* 11:20 hashar: generate doc for cumin@v1.2.2 : contint1001$ zuul enqueue-ref --trigger gerrit --pipeline publish --project operations/software/cumin --ref refs/tags/v1.2.2
* 11:17 addshore: zuul reload for zuul: add noop jobs for new analytics/wmde/WDCM-* repos [integration/config] - https://gerrit.wikimedia.org/r/388423
* 11:17 hashar: generate doc for cumin ( [[phab:T179593|T179593]] ) : contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/cumin --change 388261,2
* 02:04 legoktm: integration-slave-jessie-1004 deleted mwgate-php55lint (5.2GB) and mediawiki-core-php55lint (2.5GB) workspaces due to low disk space in /srv
== 2017-11-02 ==
* 22:30 halfak: deploying ores-deploy {{Gerrit|82a13ae}}
* 16:58 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/387960
* 13:02 hashar: gerrit: marked mediawiki/extensions/WikibaseJavaScriptApi.git read-only - [[phab:T178226|T178226]]
* 12:17 hashar: gerrit: created wikibase/javascript-api inheriting from wikibase.git  - [[phab:T178226|T178226]]
* 07:05 legoktm: mwext-VisualEditor-publish got stuck for 15 hours, deleted a job in jenkins to kick it
== 2017-11-01 ==
* 16:31 hashar: docker push wmfreleng/tox:v2017.11.01.16.28 {{!}} add libmysqlclient-dev {{!}} [[phab:T179392|T179392]]
* 15:46 hashar: docker push wmfreleng/tox:v2017.11.01.15.29  {{!}} https://gerrit.wikimedia.org/r/#/c/387723/
* 00:12 legoktm: deployed https://gerrit.wikimedia.org/r/387650
== 2017-10-31 ==
* 22:53 hashar: docker push wmfreleng/tox:v2017.10.31.22.51  ( tox 2.6.0 https://gerrit.wikimedia.org/r/#/c/387682/ )
* 22:14 hashar: docker push wmfreleng/tox:v2017.10.31.21.03  {{!}} for ebernhardson  / https://gerrit.wikimedia.org/r/#/c/387682/
* 21:34 hashar: [[phab:T144961|T144961]] : sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/composer-*php70*'
* 21:32 hashar: [[phab:T144961|T144961]] : sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/composer-package-php70-docker/*'
* 18:06 legoktm: deployed https://gerrit.wikimedia.org/r/387624
* 16:52 hashar: Migrated some tox jobs to Docker via https://gerrit.wikimedia.org/r/387582
* 16:08 hashar: integration: sudo cumin --force 'name:docker' 'rm -fR  /srv/jenkins-workspace/workspace/*tox-docker*'
* 02:24 legoktm: moved mwgate-npm jobs over to docker - https://lists.wikimedia.org/pipermail/wikitech-l/2017-October/089046.html
* 01:19 legoktm: deployed https://gerrit.wikimedia.org/r/387500
== 2017-10-30 ==
* 10:56 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted.  Unmounting /mnt and restarting elastcisearch - [[phab:T178722|T178722]]
* 10:53 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted.  Unmounting /mnt and restarting elastcisearch - T 178722
* 10:52 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted.  Unmounting /mnt and restarting elastcisearch
* 09:55 hashar: gerrit: deleted graphs/shared.git  unused / emtpy repo
* 09:27 hashar: gerrit: deleted  /nfsd.git  (unused / no changes, created on October 4th 2016)
* 09:22 hashar: gerrit: prefix mediawiki/extensions/AutomaticBoardWelcome description with '[ARCHIVED] '  - [[phab:T179196|T179196]]
* 09:21 hashar: gerrit: prefix mediawiki/extensions/AWS description with '[ARCHIVED] '  - [[phab:T174864|T174864]]
== 2017-10-28 ==
* 09:12 Krenair: fixed puppet on deployment-kafka01 by installing ldap-utils
== 2017-10-27 ==
* 13:11 godog: provision deployment-redis{03,04} with stretch - [[phab:T148637|T148637]]
* 13:06 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project wikidata/query/rdf --change 383791,15
== 2017-10-26 ==
* 16:27 greg-g: fixed it. Had to offline/reonline deployment-tin repeatedly to get through the mediawiki-config-update post-merge backlog one by one. Now jobs are running on deployment-tin again
* 15:42 greg-g: tried the gearman disable/enable, got one beta-scap run, but that's it... in a 1:1 now
* 15:30 greg-g: Jenkins slave agent won't start: https://phabricator.wikimedia.org/P6195
* 15:27 greg-g: doing the dance: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update
== 2017-10-24 ==
* 17:59 madhuvishy: Ran `sudo cumin -b 5 --backend openstack "project:deployment-prep" "apt-get install git --yes"`
* 11:19 elukey: removed several roles mistakenly applied to puppet prefix deployment-aqs in Horizon (causing puppet failures for AQS nodes)
* 08:35 hashar: beta: cherry pick https://gerrit.wikimedia.org/r/#/c/386077/4 "hieradata for varnish caches" - [[phab:T178841|T178841]]
== 2017-10-23 ==
* 20:29 Krinkle: Puppet still failing, now with: "Error 400 on SERVER: Could not find data item cache::fe_transient_gb in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/cache/text.pp:12 on node deployment-cache-text04.deployment-prep.eqiad.wmflabs"
* 20:29 Krinkle: Previous edit failed. Horizon saved the field as blank. Presumably because the class is unknown in the current version of puppet manifests it has. Strange that it normalises in this way.
* 20:28 Krinkle: Edit horizon "Other classes" config for deployment-prep/deployment-cache-text04. Rename role::prometheus::varnish_exporter to profile::prometheus::varnish_exporter
* 20:13 Krinkle: Puppet run still failing on Beta cluster varnish: "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::prometheus::varnish_exporter"
* 09:29 hashar: fab docker_pull_image:wmfreleng/tox
* 09:26 hashar: docker push wmfreleng/tox:v2017.10.23.09.05 && docker push wmfreleng/tox:latest - https://gerrit.wikimedia.org/r/385950
== 2017-10-20 ==
* 10:00 elukey: cherry pick https://gerrit.wikimedia.org/r/#/c/385339 to the operations/puppet git repo on puppetmaster02
* 03:34 Krinkle: Beta Cluster varnish (text04) has not had a Puppet run for over 10 days (15165 minutes ago). Error: " puppet-agent: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::prometheus::varnish_exporter for deployment-cache-text04 .. Not using cache on failed catalog .. Could not retrieve catalog; skipping run"
== 2017-10-19 ==
* 11:21 zeljkof: Reloading Zuul to deploy {{Gerrit|26f4ff51f9b119f040b32fea4b45910cf14635c4}}
== 2017-10-18 ==
* 18:32 greg-g: MaxSem ran `foreachwiki extensions/LoginNotify/maintenance/migratePreferences.php` on deployment-prep
* 09:14 dcausse: deployment-prep: upgrading elasticsearch to 5.5.2
* 08:41 hashar: deployment-mediawiki07: install --owner=nutcracker -d /var/run/nutcracker  && systemctl start nutcracker # [[phab:T178457|T178457]]
* 08:38 hashar: deployment-videoscaler01: install --owner=nutcracker -d /var/run/nutcracker  && systemctl start nutcracker # [[phab:T178457|T178457]]
== 2017-10-17 ==
* 22:08 addshore: replaced integration-slave-docker-c2-m4-d40-1005 with integration-slave-docker-1005 [[phab:T178409|T178409]]
* 21:48 addshore: added slave integration-slave-docker-1006 (1x 4GB ram executor)
* 21:47 addshore: delete wmfreleng/mediawiki-extensions-phan from docker hub
* 14:05 addshore: deleted slave integration-slave-docker-1004
* 13:35 addshore: swapped integration-slave-docker-1004 for integration-slave-docker-c2-m4-d40-1004 (So we have more 4GB executors)
* 09:45 addshore: reload zuul for https://gerrit.wikimedia.org/r/384673
* 08:55 addshore: delete unused mwext-php70-phan-jessie-docker 'project' in jenkins UI
* 08:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/384614
== 2017-10-16 ==
* 20:48 halfak: deploying ores {{Gerrit|fb55ab8}} [[phab:T175180|T175180]] (fixes eswiki)
* 20:36 halfak: deploying ores {{Gerrit|42c5663}} [[phab:T175180|T175180]] (rolling back)
* 20:11 halfak: deploying ores {{Gerrit|0f3fe9f}} [[phab:T175180|T175180]] (second attempt)
* 20:10 no_justification: deployment-prep Both repos date from July
* 20:10 no_justification: deployment-prep Dropped 2 deploy-cache entries for ORES from deployment-sca03
* 19:57 halfak: deploying ores {{Gerrit|0f3fe9f}} [[phab:T175180|T175180]]
* 17:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/384565
* 17:09 addshore: addshore@integration-slave-docker-c2-m4-d40-1005:/srv/git/mediawiki$ sudo git clone --bare https://gerrit.wikimedia.org/r/p/mediawiki/core.git
== 2017-10-14 ==
* 00:48 MaxSem: reverting
* 00:47 MaxSem: Trying PHP7 mode on depoyment-prep with https://wikitech.wikimedia.org/w/index.php?diff=1772791 (ping [[phab:T173786|T173786]])
== 2017-10-13 ==
* 16:34 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki ([[phab:T177857|T177857]])
* 13:41 zeljkof: Reloading Zuul to deploy {{Gerrit|b5b1dc2c580a465a3fd81f0dda6144dcb1e8f5f7}}
* 10:43 zeljkof: Reloading Zuul to deploy {{Gerrit|320f06501ea72cccdd2d012b06edf2dfb35cd986}}
== 2017-10-11 ==
* 19:59 hashar: deployment-prep: deploying jobrunner to catchup with changes.
* 18:19 hashar: beta: rebased puppet master due to a conflict with {{Gerrit|b3c6968b3c}}
* 15:32 _joe_: removing deployment-pdf01, [[phab:T177931|T177931]]
* 08:33 hashar: Image snapshot-ci-jessie-1507710117 in wmflabs-eqiad is ready
* 08:22 hashar: nodepool: refreshing Jessie snapshot after some puppet patches got merged
== 2017-10-10 ==
* 17:51 Amir1: add "Ladsgroup" to oversight members in enwiki in beta cluster to test [[phab:T177705|T177705]]
* 16:29 Amir1: adding "Ladsgroup" to admins in wikidatawiki in beta cluster
== 2017-10-09 ==
* 13:26 hashar: Upgraded Jenkins to 2.73.1 earlier today
* 08:53 hashar: hard restart integration-slave-docker-1001 via horizon. It is deadlocked somehow. - [[phab:T177749|T177749]]
== 2017-10-06 ==
* 13:22 hashar: Jenkins: adding Maven-3.0.5 to the tool configuration https://integration.wikimedia.org/ci/configureTools/
* 11:58 hashar: Jenkins: installed Warnings plugin
* 11:54 hashar: Jenkins: removing the Violations plugin. It is not used.
* 09:22 hashar: integration: purged bunch of old containers:  sudo cumin 'name:slave-docker' 'yes {{!}} docker container prune'
== 2017-10-05 ==
* 19:15 hasharAway: rebooting integration-slave-docker-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 19:11 hasharAway: rebooting integration-slave-jessie-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 13:16 hashar: Image snapshot-ci-jessie-1507208677 in wmflabs-eqiad is ready
* 11:47 hashar: Refreshing Nodepool Jessie snapshot to get java 8 by default - [[phab:T162828|T162828]]
* 10:56 hashar: integration: unbreak the puppet master. Was stuck do a cherry pick that needed a rebase
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/382361
* 04:16 legoktm: deploying https://gerrit.wikimedia.org/r/382354
== 2017-10-04 ==
* 13:19 andrewbogott: migrating 'deployment-kafka-jumbo-1' to labvirt1017
== 2017-10-03 ==
* 22:38 thcipriani: git stash /srv/mediawiki-staging/php-master/extensions/Echo to fix beta-code-update-eqiad
* 14:26 hashar: Created https://github.com/wikimedia/analytics-wikistats2 - [[phab:T177288|T177288]]
* 14:23 hashar: Gerrit: created analytics/wikistats2.git for fdans - [[phab:T177288|T177288]]
* 07:15 legoktm: deploying https://gerrit.wikimedia.org/r/381937
* 05:33 legoktm: deploying https://gerrit.wikimedia.org/r/381812 https://gerrit.wikimedia.org/r/381378
== 2017-10-02 ==
* 15:52 zeljkof: Reloading Zuul to deploy {{Gerrit|895f33da2a8daeb1fa2634bcd4eddfbbbcd78819}}
* 14:57 addshore: docker push npm image from https://gerrit.wikimedia.org/r/#/c/381384/2
* 14:47 zeljkof: Reloading Zuul to deploy {{Gerrit|9ac7821ae384612ad8249d03e47f264004d1199a}}
* 14:22 zeljkof: Reloading Zuul to deploy {{Gerrit|00890a511274d4b215034979db91708fbf6b0d0f}}
* 12:32 addshore: docker push composer, mediawiki-phan & mediawiki-phpcs latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:23 addshore: docker push ci-jessie, php & php-mediawiki latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:16 addshore: marking integration-slave-docker-1002 as offline [[phab:T177039|T177039]]
* 11:08 zeljkof: Reloading Zuul to deploy {{Gerrit|c1d7b5f1f7db4968cc2acc85539c298865a3b5ff}}
== 2017-09-30 ==
* 14:28 zeljkof: Reloading Zuul to deploy {{Gerrit|c08a3ad67b5bc1cbdb900d6b4138034b4be83c0f}}
== 2017-09-29 ==
* 19:45 hashar: Deleting integration-slave-jessie-php55
* 17:34 zeljkof: Reloading Zuul to deploy {{Gerrit|0e26c8697438a88fbdd62884dcb40e664ba98fa2}}
* 16:42 zeljkof: Reloading Zuul to deploy {{Gerrit|09445b837a03a2ce906fe848aec8350f59ab5898}}
* 15:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7f66813dc0842dadfdb74c9257582aed26f35d60}}
* 14:15 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.logininput.org ( testing w/o delete [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 14:10 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.loginpartner.org --delete ( testing [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 13:00 hashar: github: created https://github.com/wikimedia/integration-quibble for gerrit replication
* 12:53 hashar: gerrit: marked labs/tools/grrrit archived
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'weeks ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'months ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:40 addshore: marking integration-slave-docker-1001 as online - [[phab:T177039|T177039]]
* 09:33 addshore: rebooting integration-slave-docker-1001
* 09:10 addshore: wm-ci-docker-push mediawiki-phpcs:v2017.09.29.09.08 & latest https://gerrit.wikimedia.org/r/381413
* 05:59 legoktm: marking integration-slave-docker-1001 as offline - [[phab:T177039|T177039]]
* 00:19 mutante: releases1001 - created user for "no_justification", dropped pass in home dir
* 00:12 mutante: jenkins now configured and running at https://releases.wikimedia.org/ci/ ([[phab:T164030|T164030]]) - but needs additional admin users and puppet is still disabled for temp hack fix
== 2017-09-28 ==
* 16:58 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381242
* 15:37 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.28.15.28
* 11:35 hashar: docker push wmfreleng/tox:v2017.09.28.11.34 Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 11:34 hashar: docker push wmfreleng/ci-jessie:v2017.09.28.11.33 . Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 09:20 moritzm: upgraded mediawiki04-mediawiki06 in deployment-prep to HHVM 3.18.5
* 08:39 hashar: Deleted integration-saltmaster and deployment-salt02 . Replaced by integration-cumin and deployment-cumin - [[phab:T176314|T176314]]
* 08:32 hashar: Migrated Hiera config from https://wikitech.wikimedia.org/wiki/Hiera:Integration to Horizon
* 08:31 hashar: Removing salt configuration from integration and deployment-prep projects. Replaced by cumin. - [[phab:T176314|T176314]]
== 2017-09-27 ==
* 21:25 hashar: salt is being replaced by cumin instances being deployment-cumin and integration-cumin . Check this out: https://wikitech.wikimedia.org/wiki/Cumin !
* 20:12 hashar: Deleted aptly.integration.eqiad.wmflabs and the https://integration-aptly.wmflabs.org/repo/ webproxy. They were for php5.5 packages on jessie, now available on apt.wm.o - [[phab:T174972|T174972]]
* 19:39 hashar: deployment-prep: purging "ferm" on hosts that no more have it applied via puppet. There were some old iptables rules left around blocking access
* 16:38 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381024
* 12:38 addshore: Reloading Zuul to deploy - Add lintr-docker-non-voting [integration/config] - https://gerrit.wikimedia.org/r/380746
* 12:27 hasharAway: docker push wmfreleng/tox:v2017.09.27.12.26  https://gerrit.wikimedia.org/r/#/c/380926/
* 11:01 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.27.10.53 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.27.10.51 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/composer:v2017.09.27.10.49 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.27.10.45 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/tox:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 10:08 addshore: docker push docker.io/wmfreleng/ci-jessie:v2017.09.27.09.59 & latest (from https://gerrit.wikimedia.org/r/#/c/378033/5)
== 2017-09-26 ==
* 16:15 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.26.16.09 & latest (from PS11 of https://gerrit.wikimedia.org/r/379479)
* 16:01 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.26.15.45 & latest (From PS10 of https://gerrit.wikimedia.org/r/379479)
* 13:58 addshore: added hashar to https://hub.docker.com/u/wmfreleng
* 13:07 moritzm: adding  deployment-videoscaler01 to deployment-prep (stretch-based video scaler)
* 12:45 addshore: Reloading Zuul to deploy - Low prio queue for libraryupdater [integration/config] - https://gerrit.wikimedia.org/r/380307
* 12:28 addshore: fab docker_pull_image:wmfreleng/lintr:v2017.09.26.12.04
* 12:26 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/379818
* 12:06 addshore: docker push docker.io/wmfreleng/ & latest (PS6 of https://gerrit.wikimedia.org/r/378831 )
== 2017-09-25 ==
* 17:07 mutante: Greg is now a contint-admin
* 12:36 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker rmi wmfreleng/operations-puppet:0.0.1 wmfreleng/operations-puppet:0.1.0'
* 12:30 addshore: Reloading Zuul to deploy Refactor 'operations-puppet-tests-docker' into macros for easy reuse [integration/config] - https://gerrit.wikimedia.org/r/379959
* 09:12 moritzm: added deployment-mediawiki07 to deployment-prep (stretch-based app server, WIP)
== 2017-09-24 ==
* 10:49 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run "sudo docker images --no-trunc --format '{{.ID}} {{.CreatedSince}}' {{!}} grep ' months' {{!}} awk '{ print $1 }' {{!}} xargs --no-run-if-empty docke r rmi"
* 10:37 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.24.10.33 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2 (actually this time))
* 09:58 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.22.18.51 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2)
== 2017-09-22 ==
* 21:55 tabbycat: Granted Greg G. 'staff' global rights on the beta cluster per request
* 20:37 hashar: Image snapshot-ci-jessie-1506112074 in wmflabs-eqiad is ready
* 20:28 hashar: updating nodepool image for jessie [2/x]
* 20:03 hasharAway: Updating nodepool image for jessie
* 17:22 addshore: docker push docker.io/wmfreleng/tox:v2017.09.22.17.16 & latest # (From current master)
* 15:24 hashar: Restarted Jenkins (out of memory)
* 10:06 hashar: deployement-salt02 migrated hiera config from wikitech to horizon. Removed the class role::deployment::salt_masters
* 08:44 hashar: Upgraded docker on integration-slave-docker-1001 and integration-slave-docker-1002 - [[phab:T176267|T176267]]
* 07:13 greg-g: some jsduck jobs are running now, serially, for the backlogged queue. Unsure of starved jobs (integration-config-qa, pywikibot-beta-cluster, etc)
* 07:04 greg-g: deleting stuck mediawiki-core-jsduck-publish jobs in Jenkins UI
* 06:57 greg-g: pinged an opsen, hopefully they'll restart zuul shortly
* 06:45 greg-g: Zuul is stuck, no jobs are processing
== 2017-09-21 ==
* 10:23 elukey: removed {{Gerrit|6fdf6ee653}} from deployment-prep's puppet master cherry picks (seemed an old version of https://gerrit.wikimedia.org/r/#/c/357985)
== 2017-09-20 ==
* 15:46 addshore: reloading zuul for https://gerrit.wikimedia.org/r/#/c/379250/
* 13:59 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.20.13.49 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/composer:v2017.09.20.13.44 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.20.13.44 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.20.13.43 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/php:v2017.09.20.13.40 & latest # built from master
* 13:07 tabbycat: deployment-prep Ran cleanupSpam.php on deploymentwiki. Further testing with regards to ongoing development and updating of the script.
* 11:53 addshore: Reloading Zuul (Testing)
== 2017-09-19 ==
* 17:26 legoktm: removed rights from User:Sau226 on beta cluster due to block of account used for browser tests
* 09:13 tabbycat: Re-run previous script and it worked this time, see https://deployment.wikimedia.beta.wmflabs.org/wiki/Template_talk:Rotate/en
* 09:11 tabbycat: Ran mwscript cleanupSpam.php on the beta cluster, but it didn't worked (looks it is not fetching the domains properly)
== 2017-09-18 ==
* 20:12 addshore: deleted unused images that were *months old* on docker slaves
* 19:01 addshore: addshore@contint1001:~$ sudo service zuul reload
* 18:45 thcipriani: reloading zuul to test https://gerrit.wikimedia.org/r/#/c/378665/2
* 16:53 elukey: removed https://gerrit.wikimedia.org/r/#/c/377753/ from the git cherry-picks in operations/puppet on puppetmaster02
== 2017-09-17 ==
* 18:59 addshore: Reloading Zuul to deploy archiving of 2 extensions
== 2017-09-14 ==
* 19:37 tgr: updated PrivateSettings.php for [[phab:T175868|T175868]]
* 10:38 elukey: cherry-pick https://gerrit.wikimedia.org/r/#/c/377753/7 on deployment-prep's puppetmaster02 to test it on the new kafka jumbo instances
* 10:35 hashar: CI puppet master:  added class geoip::data::package and parameters:  puppetmaster::geoip::fetch_private: false  puppetmaster::geoip::use_proxy: false    - [[phab:T175864|T175864]]
== 2017-09-13 ==
* 10:13 addshore: docker push docker.io/wmfreleng/operations-puppet:v2017.09.13.09.23 (#d693f74c9b3404220a2ad2934f526d4f4455914b)
* 09:25 hashar: Deleting integration-slave-trusty-1003 and integration-slave-trusty-1001  - [[phab:T175696|T175696]]
* 09:14 hashar: nodepool:  openstack image delete image-ci-trusty  - [[phab:T175696|T175696]]
* 07:49 hashar: Jenkins: removing the Ubuntu JDK from https://integration.wikimedia.org/ci/configureTools/
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - [[phab:T161882|T161882]]
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - T 161882
== 2017-09-12 ==
* 20:35 hashar: pooling integration-slave-jessie-1003 and integration-slave-jessie-1004
* 19:40 hashar: hacked integration-slave-jessie hosts to ship them php5.5
* 18:49 hasharAway: nodepool: deleted image image-ci-trusty_old_20170804  Keeping image-ci-trusty just in case
* 14:57 hashar: Deleted all left over jenkins jobs having ci-trusty-wikimedia label. - [[phab:T161882|T161882]]
* 14:46 hashar: provisionning integration-slave-jessie-1003 and integration-slave-jessie-1004 to move php55lint to them.  NOT READY YET - [[phab:T161882|T161882]]
* 14:05 hashar: Deleting integration-slave-trusty-1004  - [[phab:T161882|T161882]]
* 13:09 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-815477
* 11:09 hashar: Image snapshot-ci-jessie-1505213295 in wmflabs-eqiad is ready
* 10:48 hashar: nodepool: force updating jessie image to grab php5.5-luasandbox - [[phab:T161882|T161882]] [[phab:T174972|T174972]]
== 2017-09-11 ==
* 23:27 thcipriani: restarting jenkins
* 22:38 legoktm: deploying https://gerrit.wikimedia.org/r/377361
* 12:47 hashar: Nodepool: refreshing jessie snapshot to get php5.5-luasandbox installed
== 2017-09-10 ==
* 01:44 bd808: nodepool running steadily again, but has been heavily throttled to hopefully prevent another weekend thundering herd of doom failure for the OpenStack backend
== 2017-09-09 ==
* 22:15 bd808: `sudo journalctl -u nodepool --since today --no-pager` shows many LaunchStatusException failures.
== 2017-09-07 ==
* 13:02 hashar: nodepool: Image snapshot-ci-jessie-1504788047 in wmflabs-eqiad is ready {{!}} [[phab:T174972|T174972]]
* 11:58 hashar: nodepool: updating snapshot-ci-jessie  to add php5.5-redis {{!}} [[phab:T161882|T161882]] [[phab:T174972|T174972]]
* 11:10 addshore: Reloading Zuul to deploy "Add gate-submit jobs for analytics/wmde/* repos"
* 02:44 legoktm: deploying https://gerrit.wikimedia.org/r/376460
== 2017-09-06 ==
* 21:32 bearND: Update mobileapps to {{Gerrit|2cb6281}} ([[phab:T168848|T168848]] [[phab:T169277|T169277]] [[phab:T169274|T169274]] [[phab:T162179|T162179]] [[phab:T164033|T164033]] [[phab:T167921|T167921]] [[phab:T174698|T174698]] [[phab:T168848|T168848]] [[phab:T174808|T174808]])
== 2017-09-05 ==
* 23:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/376034
* 19:34 gilles: deployed PrivateSettings.php change to add Thumbor username to Swift configuration
== 2017-09-04 ==
* 15:59 zeljkof: Reloading Zuul to deploy {{Gerrit|ca1c6ec05122e6f50a5a52fecbbd2bd8e882ad87}}
* 12:21 hashar: Image snapshot-ci-jessie-1504527142 in wmflabs-eqiad is ready
* 11:37 hashar: nodepool: refreshing jessie snapshot
* 10:03 addshore: Reloading Zuul to deploy mwext-php70-phan-jessie-docker experimental job
* 00:42 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/libs$ sudo -u zuul rm -rf XMPReader
== 2017-09-02 ==
* 08:32 legoktm: rm -rf /var/logs/kafka on deployment-kafka01 to free up disk space
== 2017-08-31 ==
* 23:32 Krenair: fixed deployment-imagescaler0[12] puppet by installing a package and file manually, some puppetisation still needed - https://phabricator.wikimedia.org/T174746
* 23:04 Krenair: that also did deployment-cache-(upload{{!}}text)04
* 22:50 Krenair: fixed deployment-ms-be0[34] puppet by removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/371582/1 - details in a comment there
* 15:49 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/374970/
* 14:52 zeljkof: Reloading Zuul to deploy {{Gerrit|dd504839b3a48f431199785421751a5e3e38af57}}
* 10:06 zeljkof: Reloading Zuul to deploy {{Gerrit|e91b30ff6df435a6a3c55fe978be6bbf1f1cbde4}}
* 06:25 legoktm: deploying https://gerrit.wikimedia.org/r/374937
== 2017-08-30 ==
* 12:49 hashar: gerrit: marked wikimedia/communications/WMBlog as read-only - [[phab:T172372|T172372]]
== 2017-08-29 ==
* 15:39 hashar: Created integration-slave-jessie-php55 to try out a php5.5 package on Jessie - [[phab:T161882|T161882]]
* 15:06 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-793795
* 08:45 hashar: Restarting Jenkins for openjdk update
* 08:11 hashar: refreshing all Jenkins jobs with a newer version of JJB
== 2017-08-28 ==
* 14:54 hashar: integration: rebase integration puppet master. Got a conflict due to r -> r_lang renaming ( https://gerrit.wikimedia.org/r/#/c/363337/ )
* 08:52 hashar: gerrit: added ldap/ciadmin  to the 'integration' group.  [[phab:T169557|T169557]] [[phab:T173233|T173233]]
== 2017-08-25 ==
* 15:11 zeljkof: Reloading Zuul to deploy {{Gerrit|b6704e249f0d9e83de5835908eaddff40f104e2f}}
== 2017-08-24 ==
* 21:55 mdholloway: disk was full on integration-slave-jessie-android; deleted ~8gb of old screenshots from /tmp to clear some space
* 16:54 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/372780/7
* 15:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/371138/
* 11:56 zeljkof: Reloading Zuul to deploy {{Gerrit|c20a7402467efb669a30dc06ec70c41fc6853193}}
== 2017-08-22 ==
* 06:49 legoktm: deploying https://gerrit.wikimedia.org/r/370695
* 06:44 legoktm: deploying https://gerrit.wikimedia.org/r/372229
== 2017-08-21 ==
* 18:18 mutante: addshore is now a contint-admin
== 2017-08-18 ==
* 20:21 legoktm: deploying https://gerrit.wikimedia.org/r/372222
== 2017-08-16 ==
* 23:46 legoktm: reloading Zuul to deploy https://gerrit.wikimedia.org/r/372208 https://gerrit.wikimedia.org/r/371653 https://gerrit.wikimedia.org/r/371757 https://gerrit.wikimedia.org/r/371640
== 2017-08-15 ==
* 17:57 bearND: Update mobileapps to {{Gerrit|33b80dd}} ([[phab:T172829|T172829]] [[phab:T152441|T152441]] [[phab:T172021|T172021]] [[phab:T103362|T103362]])
== 2017-08-14 ==
* 09:46 TabbyCat: maurelio@deployment-tin:/srv/mediawiki/dblists$ expanddblist flow-computed > /home/maurelio/flow-test.dblist (to test expandblist for a patch I am working on)
== 2017-08-11 ==
* 20:25 addshore: added mediawiki::maintenance::wikidata to deployment-tin
== 2017-08-07 ==
* 15:11 thcipriani: restarting jenkins for plugin update
== 2017-08-06 ==
* 13:28 TabbyCat: Ran mwscript extensions/WikimediaMaintenance/dumpInterwiki.php deploymentwiki on the beta cluster
== 2017-08-04 ==
* 15:21 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/370222/2
== 2017-08-03 ==
* 12:02 hashar: Added  integration-slave-docker-1004 to the pool of jenkins slaves - [[phab:T150502|T150502]]
* 10:12 hashar: gerrit: marked wikimedia/communications/WP-Victor read-only and [ARCHIVED] - [[phab:T107430|T107430]]
* 04:50 SMalyshev: update cherry-pick for https://gerrit.wikimedia.org/r/#/c/299825/8 on deployment-puppetmaster02.deployment-prep.eqiad.wmflabs
== 2017-08-02 ==
* 22:08 MaxSem: Running rebuildall.php on beta ruwiki
* 20:17 bearND: Update mobileapps to {{Gerrit|2d8e8f6}}
* 11:31 hashar: Image snapshot-ci-jessie-1501673225 in wmflabs-eqiad is ready [[phab:T169602|T169602]]
* 10:51 hashar: Image snapshot-ci-jessie-1501670727 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 09:02 hashar: Regenerating Nodepool Jessie image from scratch to get rid of tox 1.9.2 installed under /usr/local - [[phab:T169602|T169602]]
* 08:44 hashar: Image snapshot-ci-jessie-1501662758 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 08:42 hashar: - [[phab:T169602|T169602]]
* 08:32 hashar: Regenerating Nodepool jessie image to upgrade tox from 1.9.2 to 2.5.0 - [[phab:T169602|T169602]]
== 2017-08-01 ==
* 15:45 hashar: Image snapshot-ci-jessie-1501601670 in wmflabs-eqiad is ready  && purging old instances [[phab:T161861|T161861]]
* 15:44 hashar: Debug: Executing '/usr/bin/npm install -g npm@3.8.3' -  [[phab:T161861|T161861]]
* 15:34 hashar: Refreshing nodepool Jessie image to bump npm from 2.x to 3.8.x  [[phab:T161861|T161861]]
* 10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
== 2017-07-28 ==
* 21:11 MaxSem: Dropped table wikigrok_questions from beta enwiki
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|47a07e06e5e348902a3cdc51cff9c2bed87ab6df}}
* 00:17 Krinkle: Testing job insertion on beta cluster from deployment-tin triggers PHP Notice: Undefined index: uuid in EventBus/JobQueueEventBus.php:102, PHP Notice: Undefined index: sha1 in EventBus/JobQueueEventBus.php:99
== 2017-07-26 ==
* 21:35 Reedy: kill two long running update.php jobs on deployment-tin
* 13:39 zeljkof: Reloading Zuul to deploy {{Gerrit|8787b4bbe75691e35792864434f8e09f3aaf6e53}}
* 12:04 zeljkof: Reloading Zuul to deploy {{Gerrit|79781d8b08a6dbd4d2e2b5554fbb04154c0acf48}}
* 11:39 zeljkof: Reloading Zuul to deploy {{Gerrit|723ab49039714ae2507ef43a94884de1a6bb347e}}
* 11:31 hashar: realign installed debian packages on integration-slave-jessie-1001 and integration-slave-jessie-1002 -  [[phab:T171724|T171724]]
* 09:25 hashar: deployment-tin deleting temporary l10n cache from July 19th 20:09 at /tmp/scap_l10n_3608512748  1.5G
* 09:24 hashar: deployment-cache-upload04 deployment-cache-text04 upgraded logster 0.0.10-1~jessie1 -> 0.0.10-2~jessie1 - [[phab:T171318|T171318]]
== 2017-07-25 ==
* 16:51 halfak: deploying ores {{Gerrit|835d848}} [[phab:T171505|T171505]]
== 2017-07-24 ==
* 21:56 bearND: Update mobileapps to {{Gerrit|b608ec8}}
* 15:03 hashar: Added webperformance Jenkins slave https://integration.wikimedia.org/ci/computer/webperformance/ with a single executor - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk
* 14:40 hashar: Booting integration-webperf instance 2CPU / 2GB RAM / 40G disk. Intended to host webperformance long running jobs . [[phab:T166756|T166756]]
* 11:02 hashar: Removing profile::swift::storage::labs class from deployment-ms-be03 and deployment-ms-be04 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]  [[phab:T171454|T171454]]
* 10:59 hashar: Removing class from deployment-trending01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:54 hashar: Removing classes from deployment-sca02 and deployment-sca03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:32 hashar: Removing profile::etcd from deployment-conf03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:12 hashar: Removing role::mathoid from deployment-mathoid to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:09 hashar: Removing role::changeprop from deployment-changeprop to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:06 hashar: Removing role::ocg from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:02 hashar: Removing role::mobileapps from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
== 2017-07-21 ==
* 14:55 hashar: Jenkins: upgraded Android Emulator plugin with https://gerrit.wikimedia.org/r/#/c/366253/ && https://gerrit.wikimedia.org/r/#/c/366484/ -  [[phab:T150623|T150623]]
* 14:12 hashar: added novaadmin to deployment-prep as a regular user. That lets MediaWiki OpenStack API list the instances [[phab:T171280|T171280]]
* 13:56 hashar: Created github mirror repo https://github.com/wikimedia/wikibase-wikiba.se  [[phab:T171160|T171160]]
* 10:46 hashar: Gerrit: created wikibase/wikibase.se repo for Amir1 / [[phab:T171160|T171160]]
== 2017-07-20 ==
* 16:42 hashar: How to fix ssh access on beta cluster instances: https://phabricator.wikimedia.org/T171174#3456966
* 15:30 hashar: deployment-prep : removing project wide puppet classes from https://horizon.wikimedia.org/project/puppet/  All are role::eventlogging::analytics::*
* 15:08 hashar: removed profile::recommendation_api  from deployment-sca01  to try to fix the ssh access for mobrovac  [[phab:T171173|T171173]]  [[phab:T171174|T171174]]
* 14:57 zeljkof: reloading Zuul to deploy {{Gerrit|80b9d85}}
* 14:31 hashar: deployment-prep: manually cleaned out the puppet master configuration. It was all screwed up.  Notably I removed bits about the puppetdb
* 10:20 zeljkof: Reloading Zuul to deploy {{Gerrit|80b9d855443a2f572d877b280783110684344c5d}}
* 09:17 hashar: Spawning and pooling integration-slave-docker-1003  as replacement to integration-slave-docker-1000 (broken)  - [[phab:T150502|T150502]]
* 09:03 hashar: Restoring castorby updating all jobs  to point to castor02 ( https://gerrit.wikimedia.org/r/366524 ) Starts with a cold cache :( - [[phab:T171148|T171148]]
* 08:53 hashar: Created castor02.integration.eqiad.wmflabs with puppet role role::ci::castor::server and adding it to Jenkins. Will then update the Jenkins jobs to point to it - [[phab:T171148|T171148]]
* 08:00 hashar: Disabled castor entirely via https://gerrit.wikimedia.org/r/366520 . The instance is broken - [[phab:T171148|T171148]]
* 07:55 hashar: Refreshing all Jenkins jobs defined in JJB in order to then disable castor entirely for [[phab:T171148|T171148]]
* 07:09 _joe_: rebooting castor, jobs are failing, and no one seems able to login
* 07:05 _joe_: adding myself to projectadmins for integration, trying to troubleshoot castor
* 01:38 thcipriani: scap on beta was failing because during the ldap downtime puppet created a shadow mwdeploy user, fixed using vipw and vigr
== 2017-07-19 ==
* 14:43 hashar: Jenkins: uploaded a patched android-emulator plugin for [[phab:T150623|T150623]]  and restarting Jenkins
* 13:55 hashar: Jenkins: added JDK "Debian - OpenJdk 7" with  JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
* 12:54 hashar: Gerrit: created repo integration/jenkinsci/android-emulator-plugin.git owned by access group integration-jenkinsci-android-emulator-plugin  which has Mholloway - [[phab:T170904|T170904]]
== 2017-07-18 ==
* 16:26 halfak: manually restarted uwsgi-ores and celery-ores-worker on deployment-sca03
* 16:19 halfak: manually installed "aspell-el" on deployment-sca03 (work around for ongoing puppet issues)
* 09:04 hashar: deleted integration-slave-trusty-1006
* 03:57 twentyafterfour: Fixed deployment-imagescaler01 by cherry-picking https://gerrit.wikimedia.org/r/#/c/365891/ on deployment-puppetmaster02
== 2017-07-17 ==
* 18:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/365198
== 2017-07-14 ==
* 20:16 Amir1: cpan[1]> install LWP::UserAgent on tin
== 2017-07-13 ==
* 17:04 thcipriani: restarting jenkins for updates
== 2017-07-12 ==
* 20:07 bearND: Update mobileapps to {{Gerrit|d30dae2}}
* 18:19 greg-g: where "things" == nodepool instance delete/creation
* 18:18 greg-g: things are back to a bad state, chase etc investigating
* 17:52 greg-g: nodepool is back to making instances and running jobs, thanks Cloud team
* 17:22 greg-g: CI is backed up, only one nodepoll instance running for the last long while, many in building
* 00:35 legoktm: deploying https://gerrit.wikimedia.org/r/364628
== 2017-07-11 ==
* 21:30 legoktm: deploying https://gerrit.wikimedia.org/r/364601
== 2017-07-09 ==
* 01:15 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki ([[phab:T170026|T170026]], [[phab:T165716|T165716]])
== 2017-07-07 ==
* 14:53 hashar: deployment-prep: change webproxy http://recommendation-api-beta.wmflabs.org/ to deployment-sca02 (has the proper security rule) - [[phab:T148129|T148129]]
* 14:53 hashar: deployment-prep: add port 9632 to security group "sca" https://horizon.wikimedia.org/project/access_and_security/security_groups/593/  -  [[phab:T148129|T148129]]
* 14:03 hashar: Image snapshot-ci-trusty-1499435837 in wmflabs-eqiad is ready
* 13:57 hashar: Nodepool: updating snapshot-ci-trusty
* 13:56 hashar: Nodepool: uploaded new Ubuntu Trusty image
== 2017-07-06 ==
* 17:28 thcipriani: committed changes to modules/kafkatee on deployment-puppetmaster02 since having them uncommitted broke git-sync-upstream
* 16:20 hashar: Deleting Nodepool snapshot snapshot-ci-jessie-1499350442 -  faulty php7.0-sqlite package that breaks phan jobs - [[phab:T169904|T169904]]
* 15:29 hashar: deployment-cache-upload04 manually ran apt-get upgrade to downgrade ldap-utils and libldap-2.4-2  (caused puppet failure)
* 14:14 hashar: regenerating mediawiki-core-qunit-selenium-jessie jenkins job
* 12:05 hashar: deployment-prep created Web proxy for recommendation-api-beta.wmflabs.org -> http://10.68.20.183:9632  (deployment-sca01) for schana
* 02:38 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/363519
== 2017-07-04 ==
* 14:10 hashar: manually upgraded apache2 on deployment-puppetmaster02  see [[phab:T159254|T159254]]
* 13:33 hashar: beta cluster puppet is broken: Error: Could not send report: Connection refused - connect(2) for "deployment-puppetmaster02.deployment-prep.eqiad.wmflabs" port 8140
* 09:28 hashar: gerrit:  marking read-only  mediawiki/extensions/Nonlinear - [[phab:T169519|T169519]]
== 2017-07-03 ==
* 11:34 hashar: jenkins: refreshing all jobs and updating the castor-save bit ( https://gerrit.wikimedia.org/r/#/c/361843/ )
== 2017-06-30 ==
* 08:16 hashar: Gerrit: changing repos to read-only: analytics/kraken analytics/kraken/deploy  analytics/vagrant/kraken - [[phab:T169303|T169303]]
== 2017-06-29 ==
* 23:17 legoktm: deploying https://gerrit.wikimedia.org/r/362314
== 2017-06-28 ==
* 15:55 hashar: beta: git gc  mediawiki repos in /srv/mediawiki-staging
* 15:47 hashar: beta: git -C /srv/deployment/ores/deploy/submodules/editquality gc  (saving 380MBytes)
* 15:33 hashar: running git gc under /srv/mediawiki-staging
* 14:43 hashar: pypi.python.org is back again - [[phab:T169091|T169091]]
* 14:33 elukey: running alter tables on the EL database in deployment-eventlogging03.deployment-prep.eqiad.wmflabs
* 14:06 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates and [[phab:T169091|T169091]]
* 14:04 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates
* 10:06 hashar: Unblocked beta cluster jenkins job. Have been stalled for a while
== 2017-06-27 ==
* 22:58 Amir1: cherry-picking gerrit:360891/3
* 22:42 Amir1: cherry-picking gerrit:360891/2
* 21:58 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'string' --property-id P34
* 18:31 hashar: Image snapshot-ci-jessie-1498587497 in wmflabs-eqiad is ready - [[phab:T169004|T169004]]
* 18:18 hashar: Regenerating Jessie nodepool image to hopefulyl bring back hhvm-tidy package  - [[phab:T169004|T169004]]
* 17:39 Amir1: running mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'external-id' --property-id P34
== 2017-06-26 ==
* 22:24 halfak: deploying ores-prod-deploy:82dfd56 to beta (note: [[phab:T168099|T168099]])
* 22:20 halfak: deploying ores-prod-deploy:82dfd56 to beta
* 20:33 bearND: Update mobileapps to {{Gerrit|0b05026}}
* 18:44 hashar: nodepool image-delete 1636  # Deletes snapshot-ci-trusty-1498491445 which lack nodejs when we still need it.
* 18:23 twentyafterfour: renamed previously active image to 'image-ci-trusty_bad_20170626'
* 18:22 twentyafterfour: reverted nodepool image-ci-trusty to previous version 'image-ci-trusty-old_20170626'
* 15:41 hashar: Image snapshot-ci-trusty-1498491445 in wmflabs-eqiad is ready
* 15:34 hashar: Rebuilding nodepool image for trusty and regenerating snapshots
* 09:19 hashar: gerrit: marked wikimedia/bugzilla/* repos read-only
== 2017-06-24 ==
* 06:02 legoktm: deployment-flourine02 /srv partition is alerting on low disk space but once logs get automatically gzip'd it should be fine
== 2017-06-23 ==
* 20:59 hasharAway: deployment-db03 reinstall ldap-utils, libldap-2.4-2  2.4.44+dfsg-4~bpo8+1 > 2.4.41+dfsg-1+wmf1
* 20:54 hasharAway: apt-get upgrade deployment-elastic06
== 2017-06-22 ==
* 19:02 Amir1: cherry-picking gerrit:360891/1 ([[phab:T163922|T163922]])
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group .  That also makes him an owner to wikimedia/discovery/* - [[phab:T168588|T168588]]
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group .  That also makes him an owner to wikimedia/discovery/*
* 08:18 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x! - [[phab:T168511|T168511]]
* 08:16 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x!
* 08:12 hashar: deployment-prep: upgraded puppet to 3.8.5  on all instances
== 2017-06-21 ==
* 20:03 bearND: Update mobileapps to {{Gerrit|21f771d}}
* 19:54 hashar: deployment-tin stopped keyholder and armed it
* 19:25 hashar: hard rebooting deployment-db04
* 19:20 hashar: hard rebooting deployment-db03
* 18:52 hashar: Removing /etc/apt/sources.list.d/wikimedia_mariadb.list  (content: deb http://apt.wikimedia.org/wikimedia precise-wikimedia mariadb  )
* 18:51 hashar: fixing up apt config on deployment-db03 and deployment-db04 /  upgrade packages and kernel / reboot
* 17:02 hashar: upgrading kernel and puppet on deployment-mcs01 deployment-restbase01 and deployment-restbase02 - [[phab:T168541|T168541]]
* 17:00 hashar: upgrading kernel and puppet on deployment-changeprop and deployment-conf03 - [[phab:T168541|T168541]]
* 16:56 hashar: upgrading kernel and puppet on deployment-aqs01 deployment-aqs02 and deployment-aqs03 - [[phab:T168541|T168541]]
* 16:38 hashar: rebooting deployment-cache-upload04 and deployment-cache-text-04 - [[phab:T168541|T168541]]
* 16:29 hashar: upgrading deployment-apertium02 and deployment-eventlogging04  - [[phab:T168541|T168541]]
* 16:23 hashar: upgrade and reboot deployment-prometheus01
* 16:11 hashar: rebooting deployment-ms-fe02
* 16:11 hashar: rebooting deployment-ms-be04
* 16:09 hashar: rebooting deployment-ms-be03
* 16:03 hashar: upgrading deployment-ms-fe02  deployment-ms-be03 and deployment-ms-be04
* 15:57 hashar: apt-get upgrade and reboot of deployment-memc04 and deployment-memc05
* 15:52 hashar: rebooting deployment-etcd-01
* 15:48 hashar: apt-get upgrade deployment-etcd-01
* 15:35 hashar: deployment-prep changing Varnish director for citoid from citoid.wmflabs.org to citoid-beta.wmflabs.org  ( via https://horizon.wikimedia.org/project/prefixpuppet/ ) - [[phab:T168519|T168519]]
* 14:41 hashar: deployment-tmh01 is down for some reason
* 14:21 hashar: deployment-prep: force running puppet on all instances
* 14:17 hashar: finally fixed puppet on deployment-prep !
* 14:02 hashar: deployment-puppmaster (cd /etc/puppet && ln -s /var/lib/git/operations/puppet/manifests && ln -s /var/lib/git/operations/puppet/modules)
* 13:26 hashar: deployment-prep: puppet master got erroneously upgrade to puppet* 4.8.  Roll it back to 3.8 which fail, and then back to 3.7!
* 12:47 hashar: broke deployment-prep puppet master while upgrading it :(
* 12:28 hashar: deployment-imagescaler01 removed puppetmaster and puppetmaster-common packages
* 12:04 hashar: apt-get dist-upgrade on deployment-mediawiki hosts
* 11:59 hashar: armed keyholder on deployment-tin and deployment-mira
* 11:15 hashar: deployment-cache-text04 : apt-get dist-upgrade
* 11:12 hashar: varnish fails on deployment-cache-text04
* 11:08 hashar: deployment-prep : rebooting deployment-tin deployment-mira deployment-cache-text04 deployment-cache-upload04
* 11:00 hashar: deployment-prep apt-get upgrade and reboot all hosts
* 10:21 hashar: deployment-zotero01  apt-get upgrade and rebooted
* 09:59 hashar: integration: removing swift / python-swift from integration-puppetmaster01
* 09:57 hashar: Upgrading puppet 3.7.2 .. 3.8.5 on integration-slave-docker-1001 and integration-slave-docker-1002
* 09:39 hashar: integration: deleting swift and and swift-storage-01  unused
* 09:38 hashar: upgrading/Rebooting all instances from integration project to catch up with Linux kernel upgrades
== 2017-06-20 ==
* 19:25 hashar: Nodepool rate being bumped from 1 query per 6 seconds to 1 query per 5 seconds ( https://gerrit.wikimedia.org/r/#/c/358601/ )
* afk: deployment-tin stuck on post-merge queue for the past 13 hours, unstuck now
== 2017-06-19 ==
* afk: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/360091/
* 08:29 hashar: Gerrit: added Ladsgroup to 'mediawiki' group - [[phab:T165860|T165860]]
== 2017-06-18 ==
* 19:26 Reedy: Re-enabled beta-update-databases-eqiad as wikidatawiki takes < 10 minutes [[phab:T168036|T168036]] [[phab:T167981|T167981]]
* 19:25 Reedy: A lot of items on beta wikidatawiki deleted [[phab:T168036|T168036]] [[phab:T167981|T167981]]
== 2017-06-16 ==
* 23:41 Reedy_: also deleting a lot of Property:P* pages on beta wikidatawiki [[phab:T168106|T168106]]
* 22:55 Reedy: deleting Q100000-Q200000 on beta wikidatawiki [[phab:T168106|T168106]]
* 19:04 Reedy: disabled beta-update-databases-eqiad because it's not doing much useful atm
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|18a50a707eac0bcdd88f48f2321af78ee399a4eb}}
* 14:40 hashar: integration-slave-jessie-1001  apt-get upgrade to downgrade python-pbr to 0.8.2 as pinned since [[phab:T153877|T153877]].  /usr/bin/unattended-upgrade magically upgraded it for some reason
* 06:49 Reedy: script upto `Processed up to page 336425 (Q235372)`... hopefully it's finished by morning
* 03:13 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 03:10 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 02:23 Reedy: cherry-picked https://gerrit.wikimedia.org/r/#/c/354932/ onto beta puppetmaster
== 2017-06-15 ==
* 16:34 RainbowSprinkles: deployment-prep: Disabled database updates for awhile, running it by hand
* 10:39 hashar: apt-get upgrade on deployment-tin
* 00:52 thcipriani: deployment-tin jenkins agent borked for 4 hours, should be fixed now
== 2017-06-14 ==
* 12:24 hashar: gerrit: marked mediawiki/skins/Donate has read-only ( https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/skins/Donate ) - [[phab:T124519|T124519]]
== 2017-06-13 ==
* 22:05 hashar: Zuul resarted manually from a terminal on contint1001. It does not have any statsd configuration so we will miss metrics for a bit till it is restarted properly.
* 21:13 hashar: Gracefully restarting Zuul
* 20:37 hashar: Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4)
* 20:31 hashar: Nodepool: deleted a bunch of Trusty instances.  It scheduled lot of them that are taking slots in the pool. Better have jessie nodes to be spawned instead since there is high demand for them
* 20:19 hashar: deployment-prep: added Polishdeveloper to the "importer" global group. https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/Polishdeveloper  - [[phab:T167823|T167823]]
* 18:47 andrewbogott: root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter"
* 18:46 andrewbogott: using salt to "apt-get -y install facter" on all deployment-prep instances
* 18:38 andrewbogott: restarting apache2 on deployment-puppetmaster02
* 18:37 andrewbogott: doing a git fetch and rebase for deployment-puppetmaster02
* 17:00 elukey: hacking apache on mediawiki05 to test rewrite rules
* 16:04 Amir1: cherry-picked 357985/4 on puppetmaster
* 15:59 halfak: deployed ores-prod-deploy:862aea9
* 13:47 hashar: nodepool force running puppet for: lower min-ready for trusty [puppet] - https://gerrit.wikimedia.org/r/356466
* 10:53 elukey: rolling restart of all kafka brokers to pick up the new zookeper change (only deployment-zookeeper02 available)
* 10:36 elukey: delete deployment-zookeeper01 (old trusty instance, replaced with a jessie one)
* 09:50 elukey: big refactoring for zookeeper merged in operations/puppet - https://gerrit.wikimedia.org/r/#/c/354449 - ping the Analytics team for any issue
== 2017-06-12 ==
* 14:22 hashar: Image snapshot-ci-trusty-1497276913 in wmflabs-eqiad is ready
* 14:15 hashar: Nodepool: regenerating Trusty images to confirm that removal of keystone admin_token is a noop for nodepool - [[phab:T165211|T165211]]
* 12:44 hashar: Image snapshot-ci-jessie-1497270581 in wmflabs-eqiad is ready
* 12:30 hashar: nodepool: refreshing Jessie snapshot to upgrade HHVM from 3.12 to 3.18 - [[phab:T167493|T167493]] [[phab:T165074|T165074]]
* 08:47 hashar: deployment-prep : salt -v '*' cmd.run 'apt-get clean'
== 2017-06-09 ==
* 20:30 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358092/1
* 18:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358067/3
== 2017-06-07 ==
* 17:49 elukey: forced /usr/local/bin/git-sync-upstream manually on puppetmaster02
* 17:30 elukey: manually fixed rebase issue for operations/puppet on puppetmaster02 (empty commit due to the change for scap3 and jobrunners)
* 09:33 elukey: restart kafka brokers to pick up the new zookeeper settings
* 09:00 elukey: adding deployment-zookeeper02.eqiad.wmflabs to Hiera:deployment-prep
* 08:43 gehel: upgrading kibana to v5.3.3 on deployment-logstash2
* 08:35 gehel: rolling back to kibana 5.3.2, incompatible elasticsearch version
* 08:28 gehel: upgrading kibana to v5.4.1 on deployment-logstash2
== 2017-06-06 ==
* 14:34 hashar: deleting buildlog.integration.eqiad.wmflabs  was mean to receive Jenkins logs in ElasticSearch. We are experimenting with relforge1001.eqiad.wmnet now - [[phab:T78705|T78705]]
* 12:37 hashar: Removing HHVM from permanent Trusty slaves
* 10:44 elukey: running eventlogging_cleaner.py (https://gerrit.wikimedia.org/r/#/c/356383/) on eventlogging to test the cleaning of old events
* 09:24 hashar: Deleting deployment-phab02 instance. Has been shut off since April 23rd -  [[phab:T167090|T167090]]
* 07:51 hashar_: Fixed puppet on deployment-aqs instances
== 2017-06-05 ==
* 15:38 elukey: manually hacking deployment-jobrunner02.deployment-prep.eqiad.wmflabs to test a new config
== 2017-06-02 ==
* 19:51 hashar: integration: granted ebernhardson sudo
* 12:12 hashar: jenkins: rebuild logstash plugin from HEAD of master for jenkins 2 back compat. logstash-1.2.0-4-gbcbc19e - [[phab:T78705|T78705]]
== 2017-06-01 ==
* 20:14 bearND: Update mobileapps to {{Gerrit|c4dc72d}}
* 20:12 mdholloway: killed the running emulator processes on integration-slave-jessie-android to get it booting again following yesterday's gerrit outage
* 13:39 hashar: Gerrit: change integration.git project to "Rebase if Necessary" with "Allow content merges" - [[phab:T131008|T131008]]
* 13:10 hashar: Gerrit allow content merge for integration/config ( https://gerrit.wikimedia.org/r/#/admin/projects/integration/config )  - [[phab:T131008|T131008]]
* 08:03 hashar: Purged all mysql bin files from deployment-db03 ( rm -fR /srv/sqldata/T166060 ) - [[phab:T166060|T166060]]
== 2017-05-31 ==
* 20:21 hashar: Jenkins: upgrading git-client-plugin 2.4.5..2.4.6 [[phab:T166557|T166557]]
* 07:50 hashar: deployment-db04:  mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
* 07:49 hashar: deployment-db03:  mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
== 2017-05-30 ==
* 22:08 hasharAway: Changed integration/config.git submit type from "Fast forward only" to "Rebase if Necessary"  [[phab:T131008|T131008]]
== 2017-05-29 ==
* 14:44 elukey: reverted previous config on redis01
* 14:36 elukey: set redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 300 on redis01 as test (rollback: redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 0)
* 10:22 hashar: force refreshed Nodepool Trusty images. Was stuck somehow
* 10:06 hashar: deployment-tin rm -fR /usr/src/hhvm  [[phab:T166492|T166492]]
* 09:51 hashar: deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json  [[phab:T166492|T166492]]
== 2017-05-26 ==
* 09:20 elukey: installing hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02
* 07:20 elukey: hacking on jobrunner02 in deployment-prep
* 01:28 bearND: Update mobileapps to {{Gerrit|db6493c}}
== 2017-05-25 ==
* 19:46 hashar: deployment-tin manually cleaning disk space
* 16:44 elukey: restored hhvm on jobrunner02
* 16:03 bearND: Update mobileapps to {{Gerrit|946fe1f}}
* 10:33 elukey: manual install of hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02 to test a fix for the Redis.php lib
* 02:46 RainbowSprinkles: running `mwscript extensions/Flow/maintenance/FlowUpdateUserWiki.php --wiki=enwiki` in a screen on deployment-tin, probably going to take all night
== 2017-05-24 ==
* 16:04 hashar: rebooting integration-slave-trusty-1003 to catch up with kernel upgrade
* 12:22 hashar: deployment-prep: finished rebase of puppet.git
* 10:19 hashar: deployment-prep rebased  puppet repo with: git rebase -X theirs
* 10:10 hashar: deployment-prep : resetting puppet master to last known snapshot snapshot-20170523T0010  .  All cherry picks got deleted
* 10:09 hashar: deployment-etcd-01: fixed puppet run
* 08:38 moritzm: updated puppet on deployment-puppetmaster02 to 3.8.5-2~bpo8+2
== 2017-05-23 ==
* 16:55 RainbowSprinkles: there was no data
* 16:55 RainbowSprinkles: dropped flow_ext_ref from commonswiki on beta. schema migration is busted, going to let it recreate table
* 08:20 hashar: Updating Nodepool snapshot-ci-trusty
* 08:19 hashar: Regenerated Nodepool base image for Trusty. Got rid of hhvm from it
== 2017-05-22 ==
* 12:11 greg-g: ran git prune and rm'd the gc.log file
* 11:40 greg-g: gjg@deployment-tin:/srv/mediawiki/.git/gc.log has warning: There are too many unreachable loose objects; run 'git prune' to remove them.
== 2017-05-21 ==
* 12:05 Reedy: deployment-tin is back online
* 10:41 Reedy: disabled jerkins on deployment-tin again
* 09:10 greg-g: beta-update-database-eqiad has been hitting the timelimit since May 19th
* 09:02 Reedy: brought deployment-tin back online a while ago
== 2017-05-20 ==
* 09:10 greg-g: executers are running again
* 09:02 greg-g: All executers in Jenkins are "offline" including the permament ones
== 2017-05-19 ==
* 19:05 mutante: fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab.  remove again from instance level for phab-01
* 18:40 mutante: deployment-phab01 still has puppet error "Could not find class role::phabricator::main" and that should simply be removed from it, but i can NOT find it in Horizon, i checked instance config, project config, the "Other" section, the "All classes" tab. Because it's gone. But how do i fix the instance config then?
* 18:39 mutante: applying role::phabricator_server on instance deployment-phab01 (it had error, could not find role::phabricator::main and the name changed in role/profile conversion)
== 2017-05-15 ==
* 10:46 addshore: enabled beta-code-update-eqiad for some testing
* 10:38 addshore: temporarily disabled beta-code-update-eqiad for some testing
== 2017-05-13 ==
* 20:31 bd808: Deleted stuck mediawiki-core-doxygen-publish job. Jenkins had it marked for a particular nodepool instance that was offline.
== 2017-05-12 ==
* 13:12 hashar: Trying to refresh Nodepool Jessie image. Should get HHVM pinned to 'experimental' component => 3.12.x
== 2017-05-11 ==
* 20:43 hashar: nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit.  Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. [[phab:T165074|T165074]]
* 12:57 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/353282/
== 2017-05-10 ==
* 20:28 bearND: Update mobileapps to {{Gerrit|75b135e}}
* 18:32 mutante: deployment-tin/mira: the change of the role class name was because of https://gerrit.wikimedia.org/r/#/c/344728/ which moved deployment::server to profile/role structure. both instances configured accordingly now. the remaining issue with "id_rsa.bromine" should be all unrelated
* 18:28 mutante: deployment-mira: configure puppet config in horizon, remove "role::deployment::server", use correct new name "role::deployment_server" (moved to profile). (a bit tricky because then in Horizon it seems to disappear from the "others" section, but if you click the "all" tab you get to see the class names
* 18:12 mutante: deployment-tin: puppet run now ok, except ":Upload/File[/var/lib/releases/.ssh/id_rsa.bromine.eqiad.wmnet]: Could not evaluate:" this should be an unrelated issue
* 18:05 mutante: deployment-tin: configure to use role::deployment_server (instead of deployment::server), for some reason now Horizon shows _nothing_ under "other classes" where this was before
* 17:58 mutante: deployment-tin: deleting puppet lock file (claimed it was running but also didnt run since > 900 min), looking at fixing deployment::server role name change
* 15:26 elukey: refresh cherry pick gerrit/352582 on puppet master (rebase -i to remove, then cherry pick)
* 14:34 elukey: cherry pick gerrit/352582 to puppet master
* 12:35 hashar: deployment-prep: git -C /srv/mediawiki-staging/php-master/extensions rm --cached SemanticFormsInputs
* 08:04 hashar: merging 'composer test' into mwext-testextension-* jobs https://gerrit.wikimedia.org/r/#/c/352160/ - [[phab:T161895|T161895]]
== 2017-05-09 ==
* 12:44 hashar: deployment-ircd upgrading puppet 3.7.2 => 3.8.5
* 12:19 hashar: Unbroke puppet on deployment-irc and deployment-urldownloader . Both choked on a ruby one-liner, fixed via https://gerrit.wikimedia.org/r/#/c/336840/
== 2017-05-08 ==
* 21:42 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/351131/
* 00:57 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/351130
== 2017-05-06 ==
* 01:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/352154
== 2017-05-05 ==
* 20:11 hasharDisappear: Mass pushing addition of jakub-onderka/php-console-highlighter  to all mediawiki extensions having php-parallel-lint ( example: https://gerrit.wikimedia.org/r/#/c/352215/ )
* 09:20 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350817 on deployment-puppetmaster02
* 09:17 addshore: temporarily enabled beta-code-update-eqiad
* 09:12 addshore: temporarily disabled beta-code-update-eqiad
* 08:30 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350220/ on deployment-puppetmaster02
== 2017-05-04 ==
* 10:47 hashar: puppet ca destroy deployment-zookeeper01.eqiad.wmflabs
* 10:46 hashar: puppet ca destroy deployment-ores-redis-02.deployment-prep.eqiad.wmflabs  (no such instance)
* 10:46 hashar: puppet ca sign deployment-ores-redis-02.deployment-prep.eqiad.wmflabs
* 10:39 hashar: Removing puppetmaster: puppetmaster.thumbor.eqiad.wmflabs  from deployment-imagescaler01 - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'  - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'
* 10:31 hashar: deployment-phab01 / deployment-imagescaler01 rm /etc/puppet/puppet.conf.d/10-self.conf  - [[phab:T153319|T153319]]
* 10:29 hashar: Unbroke puppet on deployment-imagescaler01 and removing role::puppetmaster::self - [[phab:T153319|T153319]]
* 10:16 hashar: Unbroke puppet on deployment-phab01 - [[phab:T153319|T153319]]
* 07:30 hashar: deployment-prep: adding TTO (This, that and the other) as a project member to grant shell access - [[phab:T163887|T163887]]
== 2017-05-03 ==
* 17:39 mdholloway: (this concerns integration-slave-jessie-android)
* 17:37 mdholloway: enabled automatic Android component installation for the Android Gradle plugin, rebuilt the SDK, and deleted the old one
* 15:54 hashar: Granted sudo right for Niedzielski accounts on Android CI slave.  Already has it with the other labs account Sniedzielski - [[phab:T164388|T164388]]
* 15:38 hashar: Granted mdholloway (mobile team) full sudo access on integration labs project so he can reach integration-slave-jessie-android -  [[phab:T164388|T164388]]
== 2017-05-02 ==
* 21:14 hashar: Manually cancelled a few mediawiki-core-jsduck-publish and mediawiki-core-doxygen-publish job in Jenkins build queue. They seems to deadlock Jenkins somehow :(
* 19:59 hashar: Regenerate jobs selenium-GettingStarted from JJB - [[phab:T164296|T164296]]
* 19:51 hashar: Jenkins: rolling back Performance plugin from 2.2 to 2.0 due to an exception / failure to find a junit xml file.  [[phab:T164296|T164296]]
* 19:02 hashar: Added multichill ( https://github.com/multichill ) to the Wikimedia Github organization
* 10:21 godog: bounce varnish and varnish-frontend on deployment-cache-upload04
* 10:16 godog: upgrade scap on deployment-tin to overcome AttributeError: Lock instance has no attribute 'get_lock_excuse'
* 09:41 godog: flip deployment-cache-upload04 to deployment-ms-fe02 - [[phab:T162247|T162247]]
* 08:17 hashar: Reconfigured all Jenkins jobs via jjb
== 2017-05-01 ==
* 20:39 hashar: Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ )
* 20:26 hashar: nodepool: deleting alien instance ci-trusty-wikimedia-631443 4e66ad7e-b9d3-4af1-b559-{{Gerrit|3f54968d376e}}
* 02:49 TimStarling: on puppetmaster02 manually updating /etc/conftool/data-local
* 02:37 TimStarling: on puppetmaster02 updated cherry pick for https://gerrit.wikimedia.org/r/#/c/347360
== 2017-04-27 ==
* 18:18 urandom: deployment-prep: restarting cassandra-metrics-collector on deployment-restbase0[1-2]
* 07:26 Amir1: cherry-picking 348184/4 ([[phab:T161563|T161563]])
== 2017-04-26 ==
* 23:36 urandom: removing r/350485 from deployment-prep
* 21:53 urandom: cherry-picking r/350485 to deployment-prep
* 20:20 bearND: Update mobileapps to {{Gerrit|14bd4a5}}
* 15:24 godog: add new deployment-ms-be0[34] backends to swift in deployment-prep - [[phab:T162247|T162247]]
== 2017-04-25 ==
* 21:57 halfak: deployed ores {{Gerrit|cc12103}}
* 06:46 Amir1: uncherry-pick {{Gerrit|f6ce64e99a}} and {{Gerrit|225b8d4e82}} ([[phab:T161563|T161563]])
== 2017-04-22 ==
* 20:17 hashar: Added FlorianSW to Github organization "wikimedia" (no team though)
== 2017-04-21 ==
* 12:25 hashar: [[phab:T104048|T104048]]  zuul enqueue --trigger gerrit --pipeline postmerge --project AhoCorasick --change 345433,1
* 09:32 hashar: Zuul: deploying  "Decouple repos from mediawiki gate queue"  {{Gerrit|7a79f752363a}}  / [[phab:T107529|T107529]]
* 09:30 elukey: hack reverted on tin and scap pull performed on jobrunner02
== 2017-04-20 ==
* 17:09 elukey: reverted hack on deployment-tin (apparently no effects on the jobrunner)
* 16:41 elukey: temporary disable puppet on deployment-tin to remove jobrunner02 from scap dsh; manually enable persistent connection between it and rdb redis hosts
== 2017-04-19 ==
* 16:34 hashar: deleted nodepool alien ci-jessie-wikimedia-613597
* 09:20 hashar: apt-get upgrade deployment-tin  deployment-mira
* 09:16 hashar: apt-get upgrade on deployment-mx deployment-redis01 deployment-redis02 deployment-cache-text04
* 02:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/348896
== 2017-04-18 ==
* 14:29 hashar: unbreaking integration puppetmaster. Broke it when upgrading the puppet package :(
* 14:09 hashar: integration: upgrade puppet on Jessie permanent slaves  3.7.2 -> 3.8.5  (and add ruby-rgen).  Done via: salt -v '*' pkg.upgrade
* 13:17 elukey: upgrade deployment-jobrunner02 to hhvm 3.18.2+wmf2 - [[phab:T162354|T162354]]
* 10:07 godog: upgrade swift to 2.2.0 on deployment-ms*
== 2017-04-14 ==
* 12:29 hashar: Delete integration-c1 instance (32GB RAM) on labvirt1004. It was used as a workaround for [[phab:T161006|T161006]]
* 08:17 hashar: beta: cherry picking again 348184/4 'service: use gzip for logging in uwsgi' for [[phab:T161563|T161563]]
* 08:03 hashar: beta: resetting puppetmaster to last good tag snapshot-20170414T0030 A cherry pick for [[phab:T161563|T161563]] end up dropping three patches which broke other parts of the infrastructure
* 07:52 hashar_: Puppet failing on deployment-tin and deployment-mira . Some patches have been dropped from the puppet master :-((
* 00:59 Amir1: three cherry-picks failed to merge, skipped them {{Gerrit|93dad5bec8e937ef93bdd63046b0bbbf14ad9722}} {{Gerrit|92c7d0b002a02ff46a00e79d6d89fe83d5f65c17}} {{Gerrit|21d60a478ffb21160049679eea235bdb1010a489}}
* 00:45 Amir1: cherry-picking 348184/1 ([[phab:T161563|T161563]])
== 2017-04-13 ==
* 15:37 hashar: deployment-mediawiki04 clearing /var/cache/hhvm/fcgi.hhbc.sq3
* 15:15 hashar: Deployed  mediawiki-core-qunit-selenium-jessie job (runs qunit + selenium with webdriverio)  https://gerrit.wikimedia.org/r/#/c/347587/ - [[phab:T139740|T139740]]
== 2017-04-12 ==
* 15:14 hashar: rm -fR  /mnt/home/jenkins-deploy/.android/build-cache/*  # [[phab:T162635|T162635]]
* 14:56 hashar: integration-slave-jessie-1001 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 14:54 hashar: integration-slave-jessie-1002 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 10:37 hashar: Jenkins email-ext plugin got upgraded.  Some groovy templating might be prevented and would have to be reviewed/approved via https://integration.wikimedia.org/ci/scriptApproval/
* 08:52 hashar: Cancelled bunch of mediawiki-core-doxygen-publish jobs that were keeping the queue busy/deadlocked builds.  Should be moved to poll scm instead ( [[phab:T115755|T115755]] )
== 2017-04-11 ==
* 15:59 hashar: integration-config-tox-jessie job is broken due to the JJB upgrade
* 15:40 hashar: Upgraded JJB to latest master {{Gerrit|4f77324f}}  with a couple cherrypicks on top of that. 022738f8...edebce7f  [[phab:T162674|T162674]]
* 15:36 hashar: Updating selenium-* jobs configuration for the performance plugin due to JJB upgrade [[phab:T162674|T162674]]
* 15:24 hashar: Adding parameter ZUUL_VOTING to all Jenkins jobs due to JJB upgrade [[phab:T162674|T162674]]
* 15:13 hashar: Forced updated jenkins-job-builder 86478421...022738f8 - [[phab:T162674|T162674]]
* 13:44 hashar: Forced updated jenkins-job-builder 1639a86e...86478421 - [[phab:T162674|T162674]]
* 13:44 hashar: Updating all Jenkins jobs using the git plugin due to JJB change  {{Gerrit|cdfeb7bf66b0eacfed3eaf2a77813d65ab0e29f2}} -  [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e  - [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e
* 10:41 hashar: Enable webdriver.io  browser tests for MediaWiki core - https://gerrit.wikimedia.org/r/#/c/324719/ - [[phab:T139740|T139740]]
* 09:50 hashar: Regenerating MediaWiki doxygen documentations for all 1.23.x releases.
* 08:55 hashar: Retriggering MediaWiki doxygen publishing job for 1.26.0 - [[phab:T162506|T162506]] : zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.26.0 --newrev {{Gerrit|981ec62244e4806a16bda804dda1d14cb5d7f193}}
== 2017-04-10 ==
* 21:17 hashar: marked a nodepool node online manually. The instance was up but Jenkins failed to reach it due to some SEVERE: I/O error in channel
* 20:52 hashar: integration-slave-jessie-1001 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete # [[phab:T162635|T162635]]
* 20:49 hashar: integration-slave-jessie-1002 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete  # [[phab:T162635|T162635]]
* 20:08 bearND: Update mobileapps to {{Gerrit|1695900}}
== 2017-04-06 ==
* 16:36 halfak: staging ores:554ea12
* 12:23 hashar: Image snapshot-ci-trusty-1491480759 in wmflabs-eqiad is ready
* 12:13 hashar: Updating Nodepool Trusty image to let Linux overcommit memory ( https://gerrit.wikimedia.org/r/#/c/346634/ )
== 2017-04-05 ==
* 13:34 ema: testing possible fix for [[phab:T162035|T162035]] on deployment-ms-fe01
== 2017-04-04 ==
* 21:29 hashar: contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 21:26 hashar: contint2001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 20:58 hashar: integration: purging precise cow images from integration-slave-jessie-1001 and integration-slave-jessie-1002  ( https://gerrit.wikimedia.org/r/#/c/345836/ )
* 20:58 hashar: rebased integration puppet master
* 20:02 legoktm: deploying https://gerrit.wikimedia.org/r/346348
== 2017-04-03 ==
* 20:43 bearND: Update mobileapps to {{Gerrit|fdd4e31}}
* 20:39 hashar: Nodepool: holding instance ci-trusty-wikimedia-597386  in an attempt debug Wikibase/Scribunto memory usage exploding [[phab:T125050|T125050]]
* 20:37 hashar: jenkins: disabled/reenabled gearman plugin to unlock the beta cluster related jobs
* 09:17 hashar: deployment-jobrunner02 : cherry picked a monkey patch for Redis::close() to prevent it from sending QUIT command ( https://gerrit.wikimedia.org/r/#/c/346117/ ) - [[phab:T125735|T125735]]
== 2017-04-01 ==
* 09:48 Sagan: puppet on deployment-tin looks like it is not running properly
== 2017-03-29 ==
* 23:51 Krinkle: Free up space on integration-slave-jessie-1001 by removing old /srv/jenkins-workspace and /srv/pbuilder dirs
* 19:57 thcipriani: added --force flag for scap in beta-scap-eqiad temporarily
* 18:41 ebernhardson: upgrading elasticsearch and kibana to 5.1.2 on deployment-logstash2 to test puppet+integration prior to prod deployment
* 15:18 hashar: Delete a 32GB instance integration-ci  - [[phab:T161006|T161006]]
== 2017-03-28 ==
* 19:53 hashar: Populating package manager cache of oojs-ui-npm-run-jenkins-node-6-jessie  by manually triggering a build with ZUUL_PIPELINE=postmerge  [[phab:T155483|T155483]]
* 19:34 hashar: Migrate oojs/ui to just run 'npm jenkins'  https://gerrit.wikimedia.org/r/345203 /  [[phab:T155483|T155483]]
* 16:05 halfak: deployed ores:18beebf ([[phab:T160638|T160638]])
* 13:22 gehel: restarting elasticsearch on deployment-elastic05 to reload log4j configuration
* 10:28 hashar: Jenkins: installing Android Lint plugin 2.4 - [[phab:T161305|T161305]]
* 07:42 hashar: nodepool cleared a couple alien instances
== 2017-03-27 ==
* 17:02 ebernhardson: cherry pick https://gerrit.wikimedia.org/r/344964 to puppetmaster to test upgrade to logstash 5.x
* 11:10 hashar: Image snapshot-ci-jessie-1490612363 in wmflabs-eqiad is ready
* 10:59 hashar: Updating Nodepool Jessie image to include PhantomJS (take two) - [[phab:T137112|T137112]]
* 10:58 hashar: Image snapshot-ci-jessie-1490611594 in wmflabs-eqiad is ready
* 10:47 hashar: Updating Nodepool Jessie image to include PhantomJS - [[phab:T137112|T137112]]
* 10:20 hashar: Restarting Jenkins to drop the Throttle Concurrent Builds plugin - [[phab:T158596|T158596]]
== 2017-03-25 ==
* 10:46 Amir1: deleting deployment-ores-redis ([[phab:T160762|T160762]])
* 10:39 Amir1: changing ores redis address to deployment-ores-redis-01 ([[phab:T160762|T160762]])
* 10:02 Amir1: deleted deployment-ores-redis-02
== 2017-03-24 ==
* 21:34 Amir1: launching deployment-ores-redis-02 ([[phab:T160762|T160762]])
== 2017-03-23 ==
* 16:07 mobrovac: restbase deploying {{Gerrit|752ca4b7}}
* 15:52 hashar: Deleting integration-slave-trusty-1011 m1.large. One less perm slave to take care about
* 14:02 hashar: deployment-ms-be01 and deployment-ms-be02 : Lower Swift replicator on, upgrade package, reboot hosts.  [[phab:T160990|T160990]]
== 2017-03-22 ==
* 09:45 hashar: beta: purging all Linux kernel from Swift instances
* 08:48 hashar: deployment-ms-be01: swift-init reload all - [[phab:T160990|T160990]]
* 08:45 hashar: deployment-ms-be01: swift-init reload container - [[phab:T160990|T160990]]
* 08:43 hashar: deployment-ms-be01: swift-init reload object - [[phab:T160990|T160990]]
== 2017-03-21 ==
* 16:47 halfak: halfak@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 16:07 Amir1: ladsgroup@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 11:27 hashar: integration: purging old packages on permanent slaves, mostly old kernels:  apt-get autoremove --purge
* 09:06 hashar: CI deploying config hack "High priority test pipeline"  :  https://gerrit.wikimedia.org/r/343318  - [[phab:T160667|T160667]]
== 2017-03-20 ==
* 20:51 andrewbogott: migrating deployment-urldownloader to labvirt1013
* 20:45 andrewbogott: migrating deployment-pdf01 to labvirt1011
* 20:14 andrewbogott: migrating deployment-puppetmaster02 to a different labvirt
* 20:09 bearND: Update mobileapps to {{Gerrit|c0ab01d}}
* 08:51 hashar: Jenkins: depooling / deleting Precise instances.
== 2017-03-17 ==
* 14:08 hashar: salt -v '*precise*' cmd.run 'puppet agent --disable "Pending shutdown on March 20th - [[phab:T158652|T158652]]"'
== 2017-03-16 ==
* 21:48 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/343113/
* 15:03 hashar: deployment-prep setting role::logging::mediawiki::udp2log::rotate: 15  in project wide hiera configuration
== 2017-03-15 ==
* 20:29 bearND: Update mobileapps to {{Gerrit|bb8fcf2}}
* 19:02 niedzielski: Reloading Zuul to deploy {{Gerrit|f1c9073c6bda0bc83329c5178cb53613479ba200}}
* 15:55 Reedy: Removed hhvm statcache cherrypick from beta puppetmaster
* 11:09 elukey: Restore prod version of memcached on deployment-memc04 after experiment (I installed a new version a while ago)
* 10:22 elukey: created instances deployment-aqs0[23] to have better testing for the AQS beta environment
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=hewiktionary
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=dewiktionary
* 09:08 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary
* 08:56 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary // (ParameterTypeException, [[phab:T160503|T160503]])
* 08:50 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (3 sites added)
* 08:49 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https --load-from=https://deployment.wikimedia.beta.wmflabs.org/w/api.php
* 08:49 addshore: addshore@deployment-tin mwscript sql.php --wiki=enwiktionary "TRUNCATE sites; TRUNCATE site_identifiers;"
* 08:44 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=dewiktionary --site-group=wiktionary // (0 sites added)
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (1 site added)
== 2017-03-14 ==
* 19:22 thcipriani: removed alien nodepool instance via: openstack server delete ci-jessie-wikimedia-566503
* 10:15 hashar: Added Niedzielski to integration.
* 09:54 hashar: Jenkins: dropping Sniedzielski  more specific permissions. Account is already in wmf ldap group
== 2017-03-13 ==
* 13:19 hashar: Depooled Precise instances from Jenkins [[phab:T158652|T158652]]  leaving the instances up for now.
* 11:38 hashar: Deleting php53lint jobs. Replacing them with php55 equivalents
* 09:39 hashar: upgrading puppet on deployment-pdf01
* 09:30 hashar: Removing old kernel packages from deployment-pdf01 to free up disk space
* 08:55 hashar: Deleting deployment-copper  Fails puppet due to broken OpenStack metadata  http://169.254.169.254/openstack/2015-10-15/meta_data.json  (fails)  and  no more needed (per elukey )
== 2017-03-10 ==
* 12:29 kart_: Beta: [[phab:T159800|T159800]]: Update DB index for [[phab:T146450|T146450]]
== 2017-03-09 ==
* 16:20 gehel: upgrading elasticsearch on deployment-prep to v5.1.2
* 09:39 hashar: deployment-prep: rebasing puppet master. Got stall due to a submodule update apparently
== 2017-03-08 ==