You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - T189365)
imported>Stashbot
(dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/838249)
 
(936 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== 2018-03-10 ==
== 2022-10-04 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - [[phab:T189365|T189365]]
* 19:46 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/838249


== 2018-03-09 ==
== 2022-10-03 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 14:22 TheresNoTime: set `ring_manager` host to `deployment-ms-fe03` in deployment-prep's _.yaml. [[phab:T316845|T316845]]
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 13:22 hashar: Triggering CI for design/codex@v0.2.1 using `zuul enqueue-ref --trigger gerrit --pipeline publish --project design/codex --ref refs/tags/v0.2.1 --newrev 4abb7677b3ea076bbd6778977d9a9374cf45015c`  # [[phab:T313767|T313767]]
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0  https://gerrit.wikimedia.org/r/#/c/417960/
* 13:15 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/822144 # [[phab:T313767|T313767]]
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 12:47 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/833061
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container {{!}} [[phab:T188686|T188686]]
* 09:02 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/837494/
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343


== 2018-03-08 ==
== 2022-09-30 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 .  Upgrade packages and reboot.
* 18:43 James_F: Triggering graceful restart of zuul to see if that fixes on-going merge/gerrit connection issues.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 17:07 James_F: Zuul: Make PHP 8.1 non-voting for all skins and extensions [[phab:T316078|T316078]]
* 18:36 bearND: Update mobileapps to {{Gerrit|afb0167}}
* 16:46 James_F: Zuul: Make PHP 8.0 and PHP 8.1 voting for all skins and extensions in master for [[phab:T300463|T300463]] and [[phab:T316078|T316078]]
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 15:12 James_F: Docker: Building and publishing PHP 8.0.24 images for [[phab:T315167|T315167]]
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 02:33 James_F: Zuul: [mediawiki/core] Clean up REL1_35 and REL1_37 PHP 8 jobs
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc  # 1.4GBytes
* 02:30 James_F: Zuul: [mediawiki/core] Upgrade PHP 8.0 and 8.1 jobs to full vendor jobs for [[phab:T300463|T300463]] and [[phab:T316078|T316078]]
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 02:27 James_F: Zuul: Drop FIXME messages for [[phab:T318093|T318093]], being Declined
* 10:27 hashar: Deploy docker images for /deploy repositories {{!}} https://gerrit.wikimedia.org/r/#/c/417217/


== 2018-03-07 ==
== 2022-09-29 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" {{!}} [[phab:T188686|T188686]]
* 23:54 TheresNoTime: samtar@deployment-jobrunner04:~$ sudo systemctl stop php7.2-fpm.service && sudo systemctl start php7.4-fpm.service
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 23:47 TheresNoTime: cherry pick 836953 to deployment-prep
* 02:28 legoktm: manually triggering jenkins jobs
* 23:09 TheresNoTime: [samtar@deployment-deploy03 ~]$ sudo puppet agent -tv
* 23:08 TheresNoTime: deployment-deploy03, `sudo systemctl stop php7.2-fpm.service`, `sudo systemctl start php7.4-fpm.service`
* 23:03 TheresNoTime: ran `sudo puppet agent -tv` on deployment -deploy03, -mediawiki11, -mediawiki12
* 13:56 James_F: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches
* 13:41 James_F: Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf for [[phab:T261872|T261872]]
* 13:34 James_F: Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones
* 12:30 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/836696


== 2018-03-06 ==
== 2022-09-28 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets  parent permission group to scoring/ores
* 17:32 brennen: trying a re-publish of dev-images in case https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/eb82162b4bf443df20998a53bfb06460bfc6a365 didn't get picked up
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic  scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from  scoring/ores itself inheriting from scoring/
* 00:30 James_F: Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job here too
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 00:20 James_F: Zuul: [mediawiki/services/function-evaluator] Use direct coverage job, for [[phab:T302608|T302608]]
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to {{Gerrit|afbe9af}}
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h


== 2018-03-05 ==
== 2022-09-27 ==
* 16:38 Reedy: deleted the stack traces too
* 08:11 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/835182
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *


== 2018-03-04 ==
== 2022-09-26 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 21:46 Daimona: Applying schema changes to the wikishared DB on beta for the CampaignEvents extension # [[phab:T318379|T318379]] [[phab:T318120|T318120]]
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 21:31 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T318120|T318120]]
* 02:12 Krenair: Regenerated captcha images for [[phab:T164047|T164047]]
* 20:00 dduvall: regenerating 314 jobs for deployment of https://gerrit.wikimedia.org/r/835262
* 11:40 James_F: Docker: Building and publishing quibble-buster-php74-bundle
* 11:40 James_F: Docker
* 10:52 hashar: Rolling quibble/ruby jobs from php 7.4 to 7.2: `mediawiki-selenium-integration-docker` `legacy-quibble-rubyselenium-docker` # [[phab:T318525|T318525]]
* 09:35 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/834717/


== 2018-03-02 ==
== 2022-09-23 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 18:09 James_F: Zuul: [wikimedia-cz/web-*] Migrate tests from php73+ to php74+
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 18:06 James_F: Zuul: [labs/tools/guc] Migrate tests from php73+ to php74+
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 18:04 James_F: Zuul: [labs/tools/coverme] Migrate tests from php73+ to php74+
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 15:55 James_F: Docker: Building and publishing php74 versions of composer-security-check, mediawiki-phan, mediawiki-phan-testrun, and phpmetrics
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 13:26 James_F: Zuul: Run php 7.4 phan for extensions and skins
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs


== 2018-03-01 ==
== 2022-09-22 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 20:40 zabe: shutoff deployment-db07 # [[phab:T318126|T318126]]
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 20:36 zabe: take deployment-prep out of read-only # [[phab:T318126|T318126]]
* 02:11 legoktm: manually queued jenkins jobs
* 20:32 zabe: failover deployment-prep master from deployment-db07 to deployment-db09 # [[phab:T318126|T318126]]
* 20:25 zabe: set deployment-prep as read-only # [[phab:T318126|T318126]]
* 16:26 dancy: Upgrading scap to latest code revision in beta cluster
* 10:38 zabe: deployment-db10: start replication # [[phab:T318126|T318126]]


== 2018-02-28 ==
== 2022-09-21 ==
* 16:31 legoktm: manually queued jenkins jobs
* 23:34 zabe: shutoff deployment-db08 # [[phab:T318126|T318126]]
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 23:00 jeena: restarting zuul to try and fix CI issues
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 20:46 zabe: clone deployment-db10 from deployment-db08 # [[phab:T318126|T318126]]
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 18:49 TheresNoTime: cherry-picked [[gerrit:833839]] to deployment-puppetmaster04, testing [[phab:T317417|T317417]]
* 07:29 legoktm: mass queuing jenkins jobs again
* 18:19 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db10 # [[phab:T318126|T318126]]
* 17:55 zabe: create volume db10 and attach to deployment-db10 # [[phab:T318126|T318126]]
* 17:54 zabe: create deployment-db10 as g3.cores8.ram16.disk20 # [[phab:T318126|T318126]]
* 14:21 zabe: deployment-db09: restart mariadb # [[phab:T318126|T318126]]
* 13:55 TheresNoTime: modified deployment-prep "prometheus" security group - port 80, [[phab:T315699|T315699]]
* 13:18 James_F: Jenkins: Dropped 16 more old node jobs left on the server.
* 13:11 James_F: Jenkins: Dropped four old node10 jobs left on the server (oojs-core-node10-browser-docker, ooui-special-node10-plus-php80-composer-docker, wikipeg-special-node10-plus-php72-composer-docker, wikipeg-special-node10-plus-php80-composer-docker)
* 13:05 James_F: Jenkins: Dropped scap-pipeline-stretch and trigger-scap-pipeline-stretch following {{Gerrit|26c74a1}}
* 12:36 hashar: Reloaded Zuul for Remove Stretch from mediawiki/tools/scap - https://gerrit.wikimedia.org/r/833705
* 09:46 andrewbogott: removed some stray whitespace in /var/lib/git/operations/puppet that was preventing rebase on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs


== 2018-02-27 ==
== 2022-09-20 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 - https://gerrit.wikimedia.org/r/#/c/415102/
* 22:00 zabe: deployment-db09: start replication # [[phab:T318126|T318126]]
* 10:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d9ed9d4dded7d646fc9c4b54155613eef99752a9}}
* 20:06 zabe: deployment-db09: import dump into mariadb # [[phab:T318126|T318126]]
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 20:04 zabe: rsynced dump from deployment-db08 to deployment-db09 # [[phab:T318126|T318126]]
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 08:08 hashar: Upgrading CI and releases Jenkins plugins notably to update the git client [[phab:T315897|T315897]]
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 02:06 zabe: created backup of all databases on deployment-db08 # [[phab:T318126|T318126]]
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.


== 2018-02-26 ==
== 2022-09-19 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 23:58 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db09 # [[phab:T318126|T318126]]
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 23:57 zabe: create volume db09 and attach to deployment-db09 # [[phab:T318126|T318126]]
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx ([[phab:T188288|T188288]])
* 23:57 zabe: create deployment-db09 as g3.cores8.ram16.disk20 # [[phab:T318126|T318126]]
* 20:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/833066
* 16:54 James_F: Zuul: [operations/mediawiki-config] Switch to PHP 7.4 jobs
* 16:24 James_F: Zuul: [mediawiki/core] Add php80 and php81 to `check php` command
* 15:36 James_F: Zuul: [mediawiki/core] run phan on PHP 7.4 for [[phab:T316518|T316518]]
* 13:50 James_F: Zuul: [mediawiki/core] Add a non-vendor php81 job for main branch for [[phab:T316078|T316078]]
* 12:06 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T316128|T316128]]
* 11:57 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T316128|T316128]]


== 2018-02-24 ==
== 2022-09-16 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster


== 2018-02-23 ==
== 2022-09-15 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 19:56 thcipriani: Updating development images on contint primary
* 11:50 hashar: deployment-mediawiki04 : apt-get clean && apt-get autoremove --purge
* 17:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/832374
* 11:50 hashar: deployment-mediawiki04 :  rm /var/cache/hhvm/*.sq3  and restarting hhvm
* 11:03 TheresNoTime: soft reboot deployment-parsoid12, unresponsive
* 11:48 hashar: deployment-mediawiki04 is out of disk space on /  causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only {{!}} [[phab:T187995|T187995]]


== 2018-02-22 ==
== 2022-09-13 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/413374/
* 22:14 zabe: delete deployment-urldownloader02
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to {{Gerrit|a1339a9}}


== 2018-02-21 ==
== 2022-09-12 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0  and reloading Zuul {{!}} https://gerrit.wikimedia.org/r/413227
* 22:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/19
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 20:38 RhinosF1: (for lack of a better place) added Cyberpower678 to acl*userdisable. has enough clue, fairly active and trusted.
* 16:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d90d617e0794725280f9f2660effabea74533cbe}}
* 08:14 James_F: Zuul: [mediawiki/extensions/Realnames] Enable quibble composer jobs
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis


== 2018-02-20 ==
== 2022-09-09 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 17:46 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T311126|T311126]]
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 17:25 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T311126|T311126]]
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 17:08 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T316409|T316409]]
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 16:36 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T316409|T316409]]
* 10:49 hashar: devtools: fixed fqdn of instances puppetmaster-1001 and gerrit-prod-1001 by manually editing `/etc/hosts` # [[phab:T317404|T317404]]


== 2018-02-19 ==
== 2022-09-08 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 20:50 dduvall: running `./fab deploy_docker` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/830919
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions


== 2018-02-18 ==
== 2022-09-07 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
* 15:05 TheresNoTime: making hack changes to beta to test [[phab:T317195|T317195]] resolution


== 2018-02-16 ==
== 2022-09-06 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 15:42 bd808: Promoted user 'StrikerBot' to admin on gitlab.wikimedia.org so that Striker can use the account to attach Developer accounts to gitlab via API.
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 02:05 James_F: Running REL1_39 branch commands for [[phab:T313920|T313920]]
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 00:20 Krinkle: Prune various old mediawiki/core wmf branches for Gerrit usability, ref [[phab:T303828|T303828]]
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely {{!}} https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *


== 2018-02-15 ==
== 2022-09-02 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 15:59 zabe: added vwalters as member of the deployment-prep project [[phab:T316943|T316943]]
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/  (no jobs touched)
* 13:40 Krinkle: " ENOENT: no such file or directory, lstat " failing quibble jobs on integration-agent-docker-1024
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only {{!}} [[phab:T187428|T187428]]
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255


== 2018-02-14 ==
== 2022-09-01 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to {{Gerrit|f23519f}}
* 09:36 zabe: shutoff deployment-urldownloader02
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 07:43 hashar: Updating Jenkins jobs for Quibble 1.4.5 > 1.4.6  +  php 7.4 update {{!}} [[phab:T305525|T305525]] {{!}} [[phab:T314586|T314586]] {{!}} [[phab:T316601|T316601]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/828611
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB


== 2018-02-13 ==
== 2022-08-31 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 23:41 zabe: deleted shutoff deployment-restbase03
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 16:39 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension # [[phab:T308738|T308738]]
* 10:51 moritzm: shutdown deployment-videoscaler01 ([[phab:T187063|T187063]])
* 16:37 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # [[phab:T312870|T312870]]
* 10:49 moritzm: shutdown deployment-tmh01 ([[phab:T187063|T187063]])
* 16:21 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # [[phab:T312870|T312870]]
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 16:18 hashar: Tag Quibble 1.4.6 @ {{Gerrit|8828487d0}} # [[phab:T305525|T305525]] [[phab:T314586|T314586]]
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 15:27 James_F: Docker: Building and publishing quibble-fresnel image based on php74 for [[phab:T316525|T316525]]
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/  # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089


== 2018-02-12 ==
== 2022-08-30 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 16:00 James_F: Zuul: [labs/tools/heritage] Switch postmerge job to tox-py37-coverage-publish for [[phab:T316627|T316627]]
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 09:32 hashar: doc: on doc1002: `sudo -u doc-uploader rm -fR /srv/doc/mw-tools-scap/` That got moved to `/srv/doc` and a redirect has been set. # [[phab:T315541|T315541]]
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for [[phab:T186870|T186870]]
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. [[phab:T54919|T54919]]
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - [[phab:T186870|T186870]])
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - [[phab:T186870|T186870]])
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - [[phab:T186870|T186870]]
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/


== 2018-02-10 ==
== 2022-08-29 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - [[phab:T186870|T186870]])
* 21:25 inflatador: ES6->7 upgrade in beta-cluster [[phab:T315604|T315604]]
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 13:39 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/826985
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
* 12:34 James_F: Zuul: [mediawiki/core] Don't run phan on PHP 7.4, it doesn't pass; for [[phab:T316518|T316518]]
* 12:08 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-re-try] for [[phab:T293924|T293924]]
* 11:28 James_F: Zuul: [mediawiki/core] Make PHP 8.1 voting on REL1_38 and REL1_39 for [[phab:T316080|T316080]]
* 09:00 hashar: Updated Jenkins job mediawiki-quibble-composer-mysql-php80-docker to capture core dumps using https://gerrit.wikimedia.org/r/c/integration/config/+/496392  # [[phab:T315167|T315167]]
* 00:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/827015


== 2018-02-09 ==
== 2022-08-25 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 15:53 dancy: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/c/integration/config/+/826593
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 07:37 James_F: Zuul: [mediawiki/extensions/UnlinkedWikibase] Add quibble for [[phab:T316183|T316183]]
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 [[phab:T186856|T186856]]
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. {{!}} [[phab:T179378|T179378]] [[phab:T186856|T186856]]
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for {{Gerrit|I73a3142271d76f9640005752b1a336123b921b9f}}
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - [[phab:T186856|T186856]]
* 00:59 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
* 00:58 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])


== 2018-02-08 ==
== 2022-08-23 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 17:40 hashar: Stopping Gerrit
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test {{!}} [[phab:T179552|T179552]]
* 11:54 hashar: Manually applied a `docker-pkg` fix on contint2001 to prevent it from downloading unrelated images [[phab:T310458|T310458]]


== 2018-02-07 ==
== 2022-08-22 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 07:17 taavi: trying to disconnect jenkins from gearman and then re-connect to see if it helps with [[phab:T315818|T315818]]
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 07:12 taavi: restart zuul-merger on contint2001 [[phab:T315818|T315818]]
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230


== 2018-02-06 ==
== 2022-08-21 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix  [[phab:T186381|T186381]]
* 13:07 Reedy: looks live various CI jobs (coverage etc) have been stuck for about 8.5 hours
* 21:14 legoktm: restarted zuul due to patch being stuck ([[phab:T186381|T186381]])
* 13:00 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/824862
* 19:25 hashar: Restarted Zuul due to [[phab:T186381|T186381]]
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480


== 2018-02-05 ==
== 2022-08-19 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 23:04 TheresNoTime: resized deployment-mwlog01's /srv volume, restarted
* 16:09 mdholloway: mobileapps deployment to BC failed with error ([[phab:T186532|T186532]])
* 22:57 TheresNoTime: shutting down deployment-mwlog01 for [[phab:T315707|T315707]]
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 15:50 James_F: Docker: Build stalled out for 30 minutes; terminated and re-started.
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 {{!}} [[phab:T186494|T186494]]
* 15:15 dancy: Upgrading scap to latest code revision in beta cluster
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 {{!}} [[phab:T186494|T186494]]
* 15:11 James_F: Docker: Building and publishing images with PHP 8.0.22 for [[phab:T315167|T315167]]


== 2018-02-04 ==
== 2022-08-18 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
* 17:01 hashar: Restarted zuul-merger on contint1001 # [[phab:T315586|T315586]]
* 16:42 hashar: Reloaded Zuul for {{Gerrit|Ie83b19699a8526bf67f5610a0aa89dcedc0e3979}}
* 13:14 awight: [beta] Deploying new kartotherian version


== 2018-02-03 ==
== 2022-08-17 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 14:18 zabe: fix merge conflicts in deployment-prep private repo # [[phab:T315394|T315394]]
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 10:27 hashar: Built image docker-registry.discovery.wmnet/releng/commit-message-validator:1.0.0  # [[phab:T315159|T315159]]
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins


== 2018-02-01 ==
== 2022-08-16 ==
* 16:14 Amir1: deleting deployment-sca03 ([[phab:T184501|T184501]])
* 20:51 RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:30 hashar: Repooled integration-agent-docker-1028 , it was mysteriously unreachable [[phab:T315372|T315372]]
* 19:18 Krinkle: mediawiki/extensions/EventLogging$ git remote-wildcard-br-d 'wmf/1.35*' 'wmf/1.36*'  'wmf/1.37*' 'wmf/1.38*'
* 19:17 Krinkle: mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref [[phab:T303828|T303828]]
* 19:16 TheresNoTime: manually running `/usr/local/bin/wmf-beta-update-databases.py` on `deployment-deploy03`
* 17:16 TheresNoTime: soft-rebooting deployment-mediawiki12


== 2018-01-31 ==
== 2022-08-12 ==
* 22:01 mdholloway: updated mobileapps to {{Gerrit|3d717fa}} on beta cluster
* 17:47 dancy: Restarting zuul
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 17:42 dancy: Restarting Jenkins in an attempt to get CI jobs running again
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 00:54 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service . Certificate was close to expiry


== 2018-01-30 ==
== 2022-08-11 ==
* 20:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/404778
* 21:11 mutante: restarted phd service on phab2001
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
* 19:12 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16
* 12:26 jnuche: Reenabled CI beta sync jobs after cluster incident
* 11:48 jnuche: Temporarily disabled CI beta sync jobs until issue in cluster is resolved
* 10:25 zabe: take deployment-prep out of read-only mode


== 2018-01-29 ==
== 2022-08-10 ==
* 23:24 awight: Experiment with versioned ORES venv, [[phab:T181071|T181071]]
* 11:36 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/822052


== 2018-01-24 ==
== 2022-08-09 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
* 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
* 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
* 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
* 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for [[phab:T293924|T293924]]
* 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for [[phab:T293851|T293851]]


== 2018-01-23 ==
== 2022-08-08 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for [[phab:T309872|T309872]]
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for [[phab:T314630|T314630]]
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from  integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase {{!}} https://wikitech.wikimedia.org/wiki/Keyholder


== 2018-01-22 ==
== 2022-08-05 ==
* afk: restarting jenkins
* 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for [[phab:T296967|T296967]]
* 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
* 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for [[phab:T314630|T314630]]
* 14:48 James_F: Zuul: Add WelpThatWorked to allow list
* 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin


== 2018-01-20 ==
== 2022-08-04 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
* 15:21 dancy: Deleting beta-mediawiki-config-update-eqiad job
* 15:16 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405
* 10:01 TheresNoTime: clearing out stuck beta deployment jobs [[phab:T314378|T314378]] [[phab:T72597|T72597]]


== 2018-01-19 ==
== 2022-08-03 ==
* 17:23 zeljkof: Reloading Zuul to deploy {{Gerrit|25cab6fdb53f1e5d284192159c43f0ce28f6e604}}
* 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
* 17:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7d6b4eecc6d7854de7db6714bcfd006d860f3e28}}
* 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
* 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
* 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
* 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for [[phab:T309872|T309872]]
* 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
* 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too


== 2018-01-18 ==
== 2022-08-02 ==
* 18:29 bearND: (beta): Update mobileapps to {{Gerrit|2690899}}
* 19:26 James_F: Zuul: [design/codex] Switch coverage job back to -direct
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* 15:23 dancy: Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 15:22 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 07:55 TheresNoTime: cleared stuck beta deployment jobs [[phab:T72597|T72597]]
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state


== 2018-01-17 ==
== 2022-08-01 ==
* 16:24 zeljkof: Reloading Zuul to deploy {{Gerrit|5f757310f499a6a2cdf036dde3d258046377186f}}
* 23:16 James_F: Zuul: [design/codex] Switch to node16
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for [[phab:T185024|T185024]] : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
* 22:53 TheresNoTime: remove stuck beta deployment jobs
* 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
* 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
* 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
* 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
* 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
* 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for [[phab:T314306|T314306]]
* 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
* 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
* 09:14 TheresNoTime: clearing stuck beta CI jobs


== 2018-01-16 ==
== 2022-07-29 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for [[phab:T313919|T313919]]
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images{{!}}grep wmfreleng{{!}}awk '{print \$3}'{{!}}sort{{!}}uniq{{!}}xargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images {{!}} https://gerrit.wikimedia.org/r/#/c/404422/


== 2018-01-15 ==
== 2022-07-28 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 17:54 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/818189/
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy {{Gerrit|ff0a02decf1c75b2cef4f438c622d548c45fd270}}
* 17:14 zeljkof: Reloading Zuul to deploy {{Gerrit|fff6431628437cd9b3ccd5bafe5b58480b2a3714}}
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for [[phab:T180841|T180841]]
* 13:37 kart_: Ran update.php on metawiki betacluster ([[phab:T180841|T180841]])
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git {{!}} [[phab:T174338|T174338]]
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo


== 2018-01-12 ==
== 2022-07-27 ==
* 14:11 zeljkof: Reloading Zuul to deploy {{Gerrit|3816837e6d70ee7eb276c9b00034a5d2012a8afd}}
* 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch [[phab:T300463|T300463]]
* 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for [[phab:T274965|T274965]]
* 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
* 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
* 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref [[phab:T143162|T143162]], [[phab:T193565|T193565]]


== 2018-01-11 ==
== 2022-07-25 ==
* 17:50 tgr: added Groovier1 to project members for [[phab:T158909|T158909]]
* 22:16 dduvall: re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403654
* 21:25 dduvall: disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 13:45 hashar: Migrate  composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403647


== 2018-01-10 ==
== 2022-07-23 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 17:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/816251
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers {{!}} https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826


== 2018-01-09 ==
== 2022-07-21 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' {{!}} [[phab:T158243|T158243]] [[phab:T162191|T162191]]
* 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git {{!}} [[phab:T183569|T183569]]
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki ([[phab:T184276|T184276]])
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores ([[phab:T184282|T184282]])
* 08:48 Amir1: stopping ores services in deployment-sca03 ([[phab:T184282|T184282]])
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 ([[phab:T184282|T184282]])


== 2018-01-08 ==
== 2022-07-20 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
* 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline


== 2018-01-05 ==
== 2022-07-19 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
* 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # [[phab:T310905|T310905]]


== 2018-01-03 ==
== 2022-07-18 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:43 dancy: Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 13:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # [[phab:T311752|T311752]]
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 10:40 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3  the version in apt.wm.o
* 09:58 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included
* 19:09 hashar: apt-get upgrade on deployment-tin  . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 09:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/814728
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)


== 2018-01-01 ==
== 2022-07-17 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for [[phab:T183862|T183862]]
* 13:00 taavi: reloading zuul for https://gerrit.wikimedia.org/r/814356
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki ([[phab:T183862|T183862]])
* 19:56 Amir1: restarting ores services in deployment-sca03 ([[phab:T183862|T183862]])


== 2017-12-27 ==
== 2022-07-16 ==
* 19:42 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt


== 2017-12-25 ==
== 2022-07-15 ==
* 04:54 legoktm: deployed https://gerrit.wikimedia.org/r/400153 https://gerrit.wikimedia.org/r/400103
* 15:59 hashar: Built pcov php docker images [[phab:T280170|T280170]]
* 15:46 hashar: contint2001: `docker-system-prune-dangling.service`  it failed overnight cause Docker was not running. That should clear Icinga state # [[phab:T313119|T313119]]
* 14:05 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Switch to node16 jobs
* 13:05 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]], this time actually.
* 12:30 hashar: Starting docker on contint2001.wikimedia.org # [[phab:T313119|T313119]]
* 12:20 hashar: rebuilding `php??` images for pcov https://gerrit.wikimedia.org/r/c/integration/config/+/694621 # [[phab:T280170|T280170]]
* 10:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813967
* 10:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813932


== 2017-12-24 ==
== 2022-07-14 ==
* 16:07 Amir1: ladsgroup@deployment-tin$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=wikidatawiki --load-from https://en.wikipedia.beta.wmflabs.org/w/api.php ([[phab:T183633|T183633]])
* 18:50 James_F: Docker: Building node16 images for CI for [[phab:T313075|T313075]]
* 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for [[phab:T203215|T203215]]
* 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
* 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for [[phab:T268085|T268085]]
* 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI


== 2017-12-23 ==
== 2022-07-13 ==
* 14:10 addshore: made ladsgroup owner of the github org
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813720
* 20:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813707


== 2017-12-22 ==
== 2022-07-12 ==
* 22:39 thcipriani: integration-slave-jessie-1004 removed mediawiki-core-jsduck, mwgate-php55lint, mediawikicore-php55lint as /srv mount was full [[phab:T179963|T179963]]
* 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki ([[phab:T312865|T312865]])
* 16:37 hashar: cancelled update of npm and npm-test images. npm is broken when used with a proxy {{!}} https://gerrit.wikimedia.org/r/#/c/399837/
* 16:12 hashar: rebuilding npm and npm-test docker images https://gerrit.wikimedia.org/r/#/c/388450/
* 09:17 hashar: Fixed mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs that used a wrong Docker image name {{!}} https://gerrit.wikimedia.org/r/399789
* 09:08 hashar: updating mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs to the new ci-src-setup docker image https://gerrit.wikimedia.org/r/399754
* 09:08 hashar: updating mediawiki-core-php70-phan-docker mwext-php70-phan-docker jobs to the new ci-src-setup docker image mediawiki-core-php70-phan-docker mwext-php70-phan-docker


== 2017-12-21 ==
== 2022-07-11 ==
* 16:54 awight: Update ORES to {{Gerrit|eb0f776bb}}
* 22:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/812934
* 15:14 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399612/ "fix hhvm docker-pkg definitions"
* 19:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/812467
* 14:50 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399611/ "fix php55 definitions"
* 14:20 hashar: fab deploy_docker for https://gerrit.wikimedia.org/r/#/c/399609/ "fix zuul-cloner docker-pkg definition"
* 13:51 hashar: wikitech: change email of PortalsBuilder user from releng@lists.wikimedia.org to portals@lists.wikimedia.org {{!}} Credentials come from https://phabricator.wikimedia.org/D872 {{!}} [[phab:T179694|T179694]]


== 2017-12-20 ==
== 2022-07-10 ==
* 20:51 hashar: Rebuilding hhvm Docker containers https://gerrit.wikimedia.org/r/399406 {{!}} [[phab:T183324|T183324]]
* 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet
* 18:00 RoanKattouw: Importing dump from deployment-db03 on deployment-db04
* 15:30 RoanKattouw: Restarting dump again, failed due to lack of disk space
* 15:07 RoanKattouw: Dropped invalid view labswiki.updates, restarting dump
* 14:59 RoanKattouw: Dumping all databases on deployment-db03 so I can restore replication on deployment-db04. This may cause MediaWiki writes to fail while the dump runs


== 2017-12-19 ==
== 2022-07-09 ==
* 20:10 RoanKattouw: (Earlier today) Depooled deployment-db04, it needs fixing after replication broke badly. It's out of sync with deployment-db03, where I manually fixed inconsistencies
* 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
* 18:11 awight: Update beta ORES service to {{Gerrit|f109792}}
* 17:25 ori: Cherry-picked {{Gerrit|Ief73cc553}} (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
* 17:00 awight: Disable ORES UI for beta wikidatawiki, [[phab:T183266|T183266]]
* 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
* 15:58 zeljkof: Reloading Zuul to deploy {{Gerrit|2f514e4d9731c1793be1b95bb601315c5966ffd6}}
* 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
* 15:37 zeljkof: Reloading Zuul to deploy {{Gerrit|fb9327ef2be7ca9ff50f74095b87fa6d449f09ef}}
* 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump  # ref [[phab:T312689|T312689]]
* 15:34 hashar: Switched tox jobs from wmfreleng/tox to docker-registry.wikimedia.org/releng/tox {{!}} https://gerrit.wikimedia.org/r/388449
* 14:50 zeljkof: Reloading Zuul to deploy {{Gerrit|2fbfc1d90aefd0c01c83d1eed31b59af62339253}}
* 12:12 hashar: CI: switching mwgate-composer-php70 job from Nodepool to Docker {{!}} https://gerrit.wikimedia.org/r/#/c/398921/
* 11:59 hashar: CI: switching composer-php55 / composer-package-php55 jobs from Nodepool to Docker {{!}} https://gerrit.wikimedia.org/r/#/c/398920/
* 11:30 hashar: building php55 docker images on contint1001 {{!}} https://gerrit.wikimedia.org/r/#/c/397634/


== 2017-12-18 ==
== 2022-07-07 ==
* 19:43 thcipriani: checkout master on deployment-tin:/srv/mediawiki-staging/php-master to fix beta-code-update-eqiad
* 22:42 TheresNoTime: clear stuck beta deployment jobs (again), [[phab:T72597|T72597]]
* 17:36 addshore: paused beta-code-update-eqiad for a while while I test something
* 21:10 TheresNoTime: clear stuck beta deployment jobs, [[phab:T72597|T72597]]
* 09:58 hashar: gerrit: deleted unused user-metrics-2 repo. Been created 4 years and 7 months ago but otherwise unused
* 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
* 09:53 hashar: gerrit: deleted wikibase/data-model and wikibase/data-model-services . They are on Github https://github.com/wmde/WikibaseDataModel and https://github.com/wmde/WikibaseDataModelServices
* 12:22 hashar: integration: rebooting `integration-agent-docker-1039` [[phab:T312534|T312534]]


== 2017-12-13 ==
== 2022-07-05 ==
* 22:08 mdholloway: deployed mobileapps@ddddebb to BC
* 14:17 dwalden: restarted mathoid service on deployment-docker-mathoid01
* 18:25 thcipriani: failed Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/394551/ permissions errors with fabfile.py
* 11:39 hashar: Reloaded Zuul for `skip selenium for Wikibase repo/rest-api` https://gerrit.wikimedia.org/r/c/integration/config/+/811258
* 18:18 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/394551/
* 08:49 hauskatze: Diffusion rORES repository. Changed URI settings: enabled SSH push for mirroring; disabled HTTP {{!}} [[phab:T311390|T311390]]
* 17:24 awight: Install aspell-is for ORES
* 17:06 awight: Deploy ORES service {{Gerrit|b67bba7}}
* 00:38 mdholloway: deployed mobileapps@bfc3588 to BC


== 2017-12-12 ==
== 2022-06-30 ==
* 18:22 mdholloway: deployed mobileapps@5b8796d to BC
* 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at [[phab:T72597|T72597]]
* 15:34 addshore: deploy zuul for parameter_functions update
* 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
* 15:27 addshore: unblocked beta scaps and files syncs on jenkins
* 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053


== 2017-12-11 ==
== 2022-06-29 ==
* 23:18 mdholloway: deployed mobileapps@e290b17 to BC
* 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
* 21:06 mdholloway: deployed mobileapps@61ca333 to BC
* 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')
* 19:03 andrewbogott: upgraded deployment-puppetmaster02 to puppet v4
* 17:24 hashar: Update node-6-docker jobs so the entry point recognizes setting  NPM_RUN_SCRIPT=doc {{!}}  https://gerrit.wikimedia.org/r/#/c/397576/
* 10:00 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-GitHub {{!}} [[phab:T182231|T182231]]
* 09:59 hashar: github: archiving https://github.com/wikimedia/mediawiki-extensions-SwiftCloudFiles - [[phab:T182384|T182384]]


== 2017-12-09 ==
== 2022-06-28 ==
* 03:13 legoktm: deployed https://gerrit.wikimedia.org/r/396486
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]


== 2017-12-08 ==
== 2022-06-27 ==
* 21:20 legoktm: deployed https://gerrit.wikimedia.org/r/396480
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 19:47 legoktm: deployed https://gerrit.wikimedia.org/r/396453
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
* 18:09 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=enwiki --fix - [[phab:T182356|T182356]]
* 13:01 hashar: github: disable vulnerability alerts and archived https://github.com/wikimedia/wikimedia-lobbypop/ {{!}} [[phab:T180878|T180878]]
* 13:01 hashar: github: disable vulnerability alerts and archived https://github.com/wikimedia/labs-tools-Wikimedia-Emoji-Bot/ {{!}} [[phab:T180878|T180878]]
* 13:00 hashar: github: disable vulnerability alerts for the archived repo https://github.com/wikimedia/labs-tools-Wikimedia-Emoji-Bot/
* 11:06 zeljkof: Reloading Zuul to deploy {{Gerrit|1faf444d1710eecb43edf5a40c58377d31d61713}}


== 2017-12-07 ==
== 2022-06-24 ==
* 20:53 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=dewiki --fix
* 20:52 taavi: added `denisse` as a member
* 20:25 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=deploymentwiki --fix --add-prefix=Broken/
* 20:18 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.loginidol.org
* 17:56 mdholloway: deployed mobileapps@71f581c to beta cluster
* 10:09 hashar: integration: sudo cumin --force 'name:integration-slave-jessie-100*' /usr/local/sbin/run-puppet-agent  {{!}} https://gerrit.wikimedia.org/r/395961
* 10:06 hashar: integration: unbroke puppet on some permanent slaves. Add been broken since Nov 29th ~ 19:50UTC {{!}}  https://gerrit.wikimedia.org/r/#/c/395961/
* 09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase.  wmf/* and REL branches are going to be broken though {{!}} https://gerrit.wikimedia.org/r/395704 {{!}} [[phab:T181838|T181838]]


== 2017-12-06 ==
== 2022-06-23 ==
* 21:43 awight: Update ORES to {{Gerrit|42cf532}}
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
* 17:54 gehel: logstash upgrade on deployment-logstash2 completed, 5 minutes of logs lost during upgrade - [[phab:T178412|T178412]]
* 17:26 gehel: upgrading ELK on deployment-logstash2 - [[phab:T178412|T178412]]
* 16:48 Hauskatze: Ran cleanupSpam.php on deploymentwiki
* 10:03 hashar: docker push wmfreleng/npm:v2017.12.06.09.55 wmfreleng/npm-stretch:v2017.12.06.09.55 wmfreleng/npm-test:v2017.12.06.09.55 wmfreleng/npm-test-stretch:v2017.12.06.09.55 !!! wmfreleng/npm-browser-test:v2017.12.06.09.55 {{!}} https://gerrit.wikimedia.org/r/#/c/395555/


== 2017-12-05 ==
== 2022-06-22 ==
* 12:18 hashar: deployment-videoscaler01: rm /var/log/hhvm/* /var/log/apache2/*  . Restarted apache2/hhvm/syslog
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 12:16 hashar: integration: sudo cumin --force '*' 'apt-get clean'
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 12:16 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 12:15 hashar: deployment-videoscaler01: apt-get clean  to free up disk space
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 08:51 hashar: jenkins: adding global property FORCE_COLOR=1 to https://integration.wikimedia.org/ci/configure . That forces webdriver.io to spurts color in the Jenkins console when not using a TTY
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 06:37 kart_: Updated cxserver to {{Gerrit|1693bcf}}
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2017-12-04 ==
== 2022-06-21 ==
* 17:44 awight: ORES: Try enwiki models on simplewiki, [[phab:T181848|T181848]] ({{Gerrit|6baed71}})
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster


== 2017-12-03 ==
== 2022-06-20 ==
* 21:27 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * # to clear out full /srv
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 14:07 ori: restarted acme-chief on deployment-acme-chief03


== 2017-12-01 ==
== 2022-06-17 ==
* 21:15 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/394655
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 13:46 godog: deployment-prep bounce elasticsearch on logstash2 to test jmx_exporter
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 11:55 hashar: updating *npm-browser-node-6-docker jobs to use a new container based on Stretch with Chromium/Firefox {{!}} https://gerrit.wikimedia.org/r/#/c/394340/ {{!}} [[phab:T179360|T179360]]
* 10:08 hashar: docker push wmfreleng/npm-browser-test-stretch:v2017.11.30.21.30 && docker push wmfreleng/npm-browser-test-stretch:latest {{!}} https://gerrit.wikimedia.org/r/#/c/394340/ {{!}} [[phab:T179360|T179360]]
* 08:40 hashar: rebased operations/puppet on deployment-prep and integration puppetmasters
* 08:40 hashar: deployment-prep: removed a hack to puppetmaster environments/future/environment.conf containing:  parser = future \n manifest = $confdir/manifests\n
* 08:38 hashar: integration: removed a hack to puppetmaster environments/future/environment.conf containing:  parser = future \n manifest = $confdir/manifests\n


== 2017-11-30 ==
== 2022-06-16 ==
* 23:08 addshore: turned beta-scap-eqiad back on
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 23:03 addshore: reload zuul to deploy Revert "Use gate-and-submit-swat for mediawiki-config" [integration/config] - https://gerrit.wikimedia.org/r/394484
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 22:58 addshore: also reloaded with hashar Switch ArticlePlaceholder to npm-browser-test & Remove mwgate-npm-node-6-jessie
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]
* 22:57 addshore: reloaded zuul for Use gate-and-submit-swat for mediawiki-config [integration/config] - https://gerrit.wikimedia.org/r/394464
* 21:05 hashar: docker push wmfreleng/npm-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-stretch:latest && docker push wmfreleng/npm-test-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-test-stretch:latest {{!}} https://gerrit.wikimedia.org/r/#/c/394338/ {{!}} [[phab:T179360|T179360]]
* 20:50 addshore: temp disable beta-scap-eqiad so that it doesnt block me doing my own scaps
* 18:59 bd808: Testing stashbot fix for double phab logging ([[phab:T181731|T181731]])
* 17:49 anomie: Finished running cleanupUsersWithNoId.php on Beta Cluster for [[phab:T181731|T181731]]
* 16:58 anomie: Running cleanupUsersWithNoId.php on Beta Cluster, see [[phab:T181731|T181731]]


== 2017-11-29 ==
== 2022-06-15 ==
* 21:27 awight: Update ores submodule, for RevIdScorer statistics
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 21:17 awight: deployment-prep Verbose logging for ORES Celery
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 14:32 chasemp: git pull on /var/lib/git/labs/private and resolve one merge conflict.  (the root key file is too old here)
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
* 09:18 hashar: gerrit: forcing replication: ssh -p 29418 hashar@gerrit.wikimedia.org replication start operations/software/druid_exporter  # [[phab:T181219|T181219]]
* 09:14 hashar: github: created wikimedia/operations-debs-contenttranslation-apertium-crh-tur and wikimedia/operations-debs-prometheus-openldap-exporter
* 09:08 hashar: github: created repo operations-software-druid_exporter {{!}} [[phab:T181219|T181219]]
* 03:56 legoktm: deleted all workspaces on integration-slave-jessie-1003 /srv ran out of space
* 03:23 Krinkle: Jenkins jobs for mediawiki-core-php55lint consistently failing on integration-slave-jessie ("git: stderr: error: failed to write..")
* 00:02 halfak: deploy-prep awight enabled ORES service
* 00:01 halfak: deploy-prep awight disabled ORES service


== 2017-11-28 ==
== 2022-06-14 ==
* 17:42 awight: Remove stale ORES customizations for the beta cluster.
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 17:31 awight: Remove beta cluster customizations for ORES
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production


== 2017-11-27 ==
== 2022-06-13 ==
* 19:06 awight: Update beta ORES to latest, {{Gerrit|e58bfbf}}
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 08:38 hashar: reactivating https://phabricator.wikimedia.org/source/iegreview/ , it still developped https://phabricator.wikimedia.org/D894
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore


== 2017-11-24 ==
== 2022-06-12 ==
* 08:16 hashar: pooling integration-slave-docker-1003 again {{!}} [[phab:T179378|T179378]]
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 08:14 hashar: nodepool: Image snapshot-ci-jessie-1511510623 in wmflabs-eqiad is ready
* 08:13 hashar: upgrading blubber on contint2001
* 08:03 hashar: nodepool: manually rebuilding snapshot-ci-jessie


== 2017-11-23 ==
== 2022-06-10 ==
* 19:34 hashar: migrating pywikibot/core jobs to Docker https://gerrit.wikimedia.org/r/#/c/393091/
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 19:34 hasharAway: migrating pywikibot/core jobs to Docker https://gerrit.wikimedia.org/r/#/c/393091/
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check


== 2017-11-22 ==
== 2022-06-09 ==
* 18:55 greg-g: beta update jobs are back
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 18:48 greg-g: hung beta updates, doing the monthly dance: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration
* 16:58 halfak: deploying ores-prod-deploy:5084251 [[phab:T181168|T181168]]


== 2017-11-21 ==
== 2022-06-08 ==
* 23:29 mdholloway: deployed mobileapps@52d6a83 on the beta cluster
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 21:28 TabbyCat: deployment-prep Ran cleanupSpam.php on deploymentwiki.
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 18:38 mdholloway: deployed mobileapps@9d1602d on the beta cluster
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134
* 17:06 hasharAway: docker push wmfreleng/tox-cergen:v2017.11.21.16.52 {{!}} https://gerrit.wikimedia.org/r/392678 {{!}} For https://integration.wikimedia.org/ci/job/cergen-tox-docker/ which pass !
* 13:24 hashar: gerrit: adding Jdrewniak to wmf-deployment group https://gerrit.wikimedia.org/r/#/admin/groups/21,members {{!}} [[phab:T180639|T180639]]
* 13:08 hashar: gerrit: created wikimedia/portals/deploy  https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/portals/deploy  for jan_drewniak {{!}} [[phab:T180777|T180777]]
* 13:02 hashar: docker push wmfreleng/ci-src-setup:v2017.11.21.12.57 && docker push wmfreleng/ci-src-setup:latest {{!}} https://gerrit.wikimedia.org/r/392632 {{!}} [[phab:T177684|T177684]]
* 02:42 Krinkle: Adding relative time to [[Deployments]] calendar (Common.js), e.g. "4 hours from now" or "soon"


== 2017-11-20 ==
== 2022-06-06 ==
* 15:57 hashar: gerrit: deleted operations/network-diagrams mostly empty and no changes. Created back in 2012.
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 15:03 hashar: integration: pass all environment variables to the docker run commands {{!}} https://gerrit.wikimedia.org/r/#/c/390432/ {{!}} [[phab:T177684|T177684]]
* 10:06 hashar: nodepool: manually deleted left over instances ci-jessie-wikimedia-894187 and ci-jessie-wikimedia-894188 . Jenkins fails to ssh to it and they were left ready for 72 hours.
* 10:05 hashar: deployment-phab : set hiera 'phabricator_cluster_search: []'  trying to unblock puppet and soft rebooted the instance {{!}} [[phab:T180935|T180935]]
* 09:39 hashar: deployment-prep added missing key between_bytes_timeout to cache::app_def_be_opts  for deployment-cache-text04 and deployment-cache-upload04 {{!}} [[phab:T180935|T180935]]
* 09:29 hashar: deployment-tin:  apt-mark hold scap  {{!}} the apt-repo on deployment-tin is out of date {{!}} [[phab:T180935|T180935]]


== 2017-11-16 ==
== 2022-06-02 ==
* 23:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/391902
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 00:10 thcipriani: removed old differential-docker-test images on integration-slave-docker-1001
* 11:26 hashar: Restarting Jenkins on contint2001
* 11:19 hashar: Restarting Jenkins on releases1002


== 2017-11-15 ==
== 2022-05-31 ==
* 21:05 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/391079/6
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 17:29 thcipriani: updating docker-pkg dockerfiles on contint1001 for https://gerrit.wikimedia.org/r/#/c/388448/
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 09:55 addshore: created wmf/1.31.0-wmf.8 branch of Wikidata extension repo [[phab:T180539|T180539]]
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 07:30 Krinkle: Aborting jobs in 'test' pipeline for backport REL commits that are already merged meanwhile in 'submit' pipeline
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2017-11-13 ==
== 2022-05-30 ==
* 23:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/391136/2
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]


== 2017-11-10 ==
== 2022-05-28 ==
* 22:17 hashar: docker push wmfreleng/npm:v2017.11.10.22.15 && docker push wmfreleng/npm:latest && docker push wmfreleng/npm-test:v2017.11.10.22.15 && docker push wmfreleng/npm-test:latest {{!}} BABEL_CACHE_PATH {{!}} [[phab:T179425|T179425]]
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]
* 21:39 hashar: docker push wmfreleng/tox-pyspark:v2017.11.10.21.37 && docker push wmfreleng/tox-pyspark:latest {{!}} {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint  {{!}} [[phab:T176747|T176747]]
* 21:36 hasharAway: docker push wmfreleng/tox:v2017.11.10.21.35 && docker push wmfreleng/tox:latest {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint  {{!}} [[phab:T176747|T176747]]
* 21:36 hasharAway: docker push wmfreleng/tox:v2017.11.10.21.35 && docker push wmfreleng/tox:latest {{!}} https://gerrit.wikimedia.org/r/389937 docker: handle signals in tox entrypoint
* 16:18 hashar: docker push wmfreleng/tox:v2017.11.10.16.17 && docker push wmfreleng/tox:latest  for https://gerrit.wikimedia.org/r/#/c/388084/
* 12:39 hashar: Updated Jenkins tox jobs {{!}} https://gerrit.wikimedia.org/r/#/c/389924/ {{!}} Does not quite fix [[phab:T176747|T176747]] yet thoughaaaaaaaaaaaa
* 11:56 hashar: docker push wmfreleng/tox:v2017.11.10.11.49 && docker push wmfreleng/tox:latest && docker push wmfreleng/tox-pyspark:v2017.11.10.11.49 && docker push wmfreleng/tox-pyspark:latest {{!}} https://gerrit.wikimedia.org/r/#/c/389924/


== 2017-11-09 ==
== 2022-05-27 ==
* 21:45 legoktm: deployed https://gerrit.wikimedia.org/r/390325
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 16:54 zeljkof: Reloading Zuul to deploy {{Gerrit|baceaeb09d5b924a323e372559de00e29fc5e698}}
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 15:20 hashar: docker push wmfreleng/npm-test:v2017.11.09.15.15 {{!}} https://gerrit.wikimedia.org/r/390261 {{!}} [[phab:T176747|T176747]]
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 12:52 hashar: Jenkins: fixed/changed the global git user.name / user.email  (now: "Wikimedia CI" and "releng@lists.wikimedia.org" )
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 10:59 hashar: Drop jsduck doc/ri from npm/npm-test images {{!}} https://gerrit.wikimedia.org/r/#/c/389943 {{!}} docker push wmfreleng/npm:v2017.11.09.10.57 && docker push wmfreleng/npm-test:v2017.11.09.10.57
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 07:53 hasharAway: Killed stuck containers wmfreleng/npm-test on integration-slave-docker-1001  - [[phab:T176747|T176747]]
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2017-11-08 ==
== 2022-05-26 ==
* 13:43 Reedy: ran apt-get clean{{!}}autoclean on deplyoment-mediawiki04 to free up some space
* 18:33 dancy: Updated Jenkins beta-* job configs
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch


== 2017-11-07 ==
== 2022-05-25 ==
* 18:45 twentyafterfour: cowboy-committed and pushed rMSCAc1f2ac2 to hopefully unbreak `scap deploy` in beta
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 17:56 legoktm: integration-slave-jessie-1003 /srv full, legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mwgate-* mediawiki-*
* 15:46 dancy: Restarted apache2 on gerrit1001
* 17:27 hashar: Image snapshot-ci-jessie-1510074928 in wmflabs-eqiad is ready - [[phab:T179772|T179772]]
* 17:15 hashar: Updating Nodepool snapshot to get php5.5-zip  - [[phab:T179772|T179772]]
* 16:15 hashar: Created portalsbuilder in Gerrit, generated a ssh key pair for it and stored in Jenkins credentials store - [[phab:T179694|T179694]]
* 15:15 hashar: Created VPS account "PortalsBuilder"  - [[phab:T179694|T179694]]


== 2017-11-06 ==
== 2022-05-24 ==
* 23:49 thcipriani: ssh-keyscan deployment-videoscaler01.deployment-prep.eqiad.wmflabs >> /etc/ssh/ssh_known_hosts
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 22:29 hashar: killed stuck npm Docker containers on integration-slave-docker-1002 (due to  [[phab:T176747|T176747]] ). Pooled the instance back, the slowness it experienced is probably not related to labvirt CPU usage ( [[phab:T179378|T179378]] )
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 20:35 Amir1: deploy ores:93e8846 in beta cluster
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 16:02 thcipriani: Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/388546/ and https://gerrit.wikimedia.org/r/#/c/389463/


== 2017-11-03 ==
== 2022-05-23 ==
* 13:51 hashar: pooled integration-slave-docker-1004 and integration-slave-docker-1007
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 13:30 hashar: Unpool integration-slave-docker-1002 and integration-slave-docker-1003 . They are slow CPU wise, most probably due to the underlying labvirt being CPU starved. - [[phab:T179378|T179378]]
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 12:38 hashar: [[phab:T179593|T179593]] generate doc for cumin@v1.2.2 : contint1001$ zuul enqueue-ref --trigger gerrit --pipeline publish --project operations/software/cumin --ref refs/tags/v1.2.2 --newrev {{Gerrit|f7453873ff7c242743a22b119fc5e2171321c38b}}
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 11:20 hashar: generate doc for cumin@v1.2.2 : contint1001$ zuul enqueue-ref --trigger gerrit --pipeline publish --project operations/software/cumin --ref refs/tags/v1.2.2
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 11:17 addshore: zuul reload for zuul: add noop jobs for new analytics/wmde/WDCM-* repos [integration/config] - https://gerrit.wikimedia.org/r/388423
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 11:17 hashar: generate doc for cumin ( [[phab:T179593|T179593]] ) : contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/cumin --change 388261,2
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 02:04 legoktm: integration-slave-jessie-1004 deleted mwgate-php55lint (5.2GB) and mediawiki-core-php55lint (2.5GB) workspaces due to low disk space in /srv
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2017-11-02 ==
== 2022-05-22 ==
* 22:30 halfak: deploying ores-deploy {{Gerrit|82a13ae}}
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 16:58 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/387960
* 13:02 hashar: gerrit: marked mediawiki/extensions/WikibaseJavaScriptApi.git read-only - [[phab:T178226|T178226]]
* 12:17 hashar: gerrit: created wikibase/javascript-api inheriting from wikibase.git  - [[phab:T178226|T178226]]
* 07:05 legoktm: mwext-VisualEditor-publish got stuck for 15 hours, deleted a job in jenkins to kick it


== 2017-11-01 ==
== 2022-05-21 ==
* 16:31 hashar: docker push wmfreleng/tox:v2017.11.01.16.28 {{!}} add libmysqlclient-dev {{!}} [[phab:T179392|T179392]]
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 15:46 hashar: docker push wmfreleng/tox:v2017.11.01.15.29  {{!}} https://gerrit.wikimedia.org/r/#/c/387723/
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]
* 00:12 legoktm: deployed https://gerrit.wikimedia.org/r/387650


== 2017-10-31 ==
== 2022-05-20 ==
* 22:53 hashar: docker push wmfreleng/tox:v2017.10.31.22.51  ( tox 2.6.0 https://gerrit.wikimedia.org/r/#/c/387682/ )
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
* 22:14 hashar: docker push wmfreleng/tox:v2017.10.31.21.03  {{!}} for ebernhardson  / https://gerrit.wikimedia.org/r/#/c/387682/
* 21:34 hashar: [[phab:T144961|T144961]] : sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/composer-*php70*'
* 21:32 hashar: [[phab:T144961|T144961]] : sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/composer-package-php70-docker/*'
* 18:06 legoktm: deployed https://gerrit.wikimedia.org/r/387624
* 16:52 hashar: Migrated some tox jobs to Docker via https://gerrit.wikimedia.org/r/387582
* 16:08 hashar: integration: sudo cumin --force 'name:docker' 'rm -fR  /srv/jenkins-workspace/workspace/*tox-docker*'
* 02:24 legoktm: moved mwgate-npm jobs over to docker - https://lists.wikimedia.org/pipermail/wikitech-l/2017-October/089046.html
* 01:19 legoktm: deployed https://gerrit.wikimedia.org/r/387500


== 2017-10-30 ==
== 2022-05-19 ==
* 10:56 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted. Unmounting /mnt and restarting elastcisearch - [[phab:T178722|T178722]]
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 10:53 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted.  Unmounting /mnt and restarting elastcisearch - T 178722
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already
* 10:52 hashar: deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted.  Unmounting /mnt and restarting elastcisearch
* 09:55 hashar: gerrit: deleted graphs/shared.git  unused / emtpy repo
* 09:27 hashar: gerrit: deleted  /nfsd.git  (unused / no changes, created on October 4th 2016)
* 09:22 hashar: gerrit: prefix mediawiki/extensions/AutomaticBoardWelcome description with '[ARCHIVED] '  - [[phab:T179196|T179196]]
* 09:21 hashar: gerrit: prefix mediawiki/extensions/AWS description with '[ARCHIVED] '  - [[phab:T174864|T174864]]


== 2017-10-28 ==
== 2022-05-18 ==
* 09:12 Krenair: fixed puppet on deployment-kafka01 by installing ldap-utils
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]


== 2017-10-27 ==
== 2022-05-17 ==
* 13:11 godog: provision deployment-redis{03,04} with stretch - [[phab:T148637|T148637]]
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 13:06 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project wikidata/query/rdf --change 383791,15


== 2017-10-26 ==
== 2022-05-16 ==
* 16:27 greg-g: fixed it. Had to offline/reonline deployment-tin repeatedly to get through the mediawiki-config-update post-merge backlog one by one. Now jobs are running on deployment-tin again
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 15:42 greg-g: tried the gearman disable/enable, got one beta-scap run, but that's it... in a 1:1 now
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 15:30 greg-g: Jenkins slave agent won't start: https://phabricator.wikimedia.org/P6195
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809
* 15:27 greg-g: doing the dance: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update


== 2017-10-24 ==
== 2022-05-14 ==
* 17:59 madhuvishy: Ran `sudo cumin -b 5 --backend openstack "project:deployment-prep" "apt-get install git --yes"`
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 11:19 elukey: removed several roles mistakenly applied to puppet prefix deployment-aqs in Horizon (causing puppet failures for AQS nodes)
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 08:35 hashar: beta: cherry pick https://gerrit.wikimedia.org/r/#/c/386077/4 "hieradata for varnish caches" - [[phab:T178841|T178841]]
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request


== 2017-10-23 ==
== 2022-05-13 ==
* 20:29 Krinkle: Puppet still failing, now with: "Error 400 on SERVER: Could not find data item cache::fe_transient_gb in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/cache/text.pp:12 on node deployment-cache-text04.deployment-prep.eqiad.wmflabs"
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 20:29 Krinkle: Previous edit failed. Horizon saved the field as blank. Presumably because the class is unknown in the current version of puppet manifests it has. Strange that it normalises in this way.
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 20:28 Krinkle: Edit horizon "Other classes" config for deployment-prep/deployment-cache-text04. Rename role::prometheus::varnish_exporter to profile::prometheus::varnish_exporter
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 20:13 Krinkle: Puppet run still failing on Beta cluster varnish: "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::prometheus::varnish_exporter"
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 09:29 hashar: fab docker_pull_image:wmfreleng/tox
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 09:26 hashar: docker push wmfreleng/tox:v2017.10.23.09.05 && docker push wmfreleng/tox:latest - https://gerrit.wikimedia.org/r/385950
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2017-10-20 ==
== 2022-05-12 ==
* 10:00 elukey: cherry pick https://gerrit.wikimedia.org/r/#/c/385339 to the operations/puppet git repo on puppetmaster02
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 03:34 Krinkle: Beta Cluster varnish (text04) has not had a Puppet run for over 10 days (15165 minutes ago). Error: " puppet-agent: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::prometheus::varnish_exporter for deployment-cache-text04 .. Not using cache on failed catalog .. Could not retrieve catalog; skipping run"
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])


== 2017-10-19 ==
== 2022-05-11 ==
* 11:21 zeljkof: Reloading Zuul to deploy {{Gerrit|26f4ff51f9b119f040b32fea4b45910cf14635c4}}
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster


== 2017-10-18 ==
== 2022-05-10 ==
* 18:32 greg-g: MaxSem ran `foreachwiki extensions/LoginNotify/maintenance/migratePreferences.php` on deployment-prep
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 09:14 dcausse: deployment-prep: upgrading elasticsearch to 5.5.2
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 08:41 hashar: deployment-mediawiki07: install --owner=nutcracker -d /var/run/nutcracker  && systemctl start nutcracker # [[phab:T178457|T178457]]
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 08:38 hashar: deployment-videoscaler01: install --owner=nutcracker -d /var/run/nutcracker  && systemctl start nutcracker # [[phab:T178457|T178457]]
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377


== 2017-10-17 ==
== 2022-05-09 ==
* 22:08 addshore: replaced integration-slave-docker-c2-m4-d40-1005 with integration-slave-docker-1005 [[phab:T178409|T178409]]
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 21:48 addshore: added slave integration-slave-docker-1006 (1x 4GB ram executor)
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 21:47 addshore: delete wmfreleng/mediawiki-extensions-phan from docker hub
* 14:05 addshore: deleted slave integration-slave-docker-1004
* 13:35 addshore: swapped integration-slave-docker-1004 for integration-slave-docker-c2-m4-d40-1004 (So we have more 4GB executors)
* 09:45 addshore: reload zuul for https://gerrit.wikimedia.org/r/384673
* 08:55 addshore: delete unused mwext-php70-phan-jessie-docker 'project' in jenkins UI
* 08:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/384614


== 2017-10-16 ==
== 2022-05-08 ==
* 20:48 halfak: deploying ores {{Gerrit|fb55ab8}} [[phab:T175180|T175180]] (fixes eswiki)
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 20:36 halfak: deploying ores {{Gerrit|42c5663}} [[phab:T175180|T175180]] (rolling back)
* 20:11 halfak: deploying ores {{Gerrit|0f3fe9f}} [[phab:T175180|T175180]] (second attempt)
* 20:10 no_justification: deployment-prep Both repos date from July
* 20:10 no_justification: deployment-prep Dropped 2 deploy-cache entries for ORES from deployment-sca03
* 19:57 halfak: deploying ores {{Gerrit|0f3fe9f}} [[phab:T175180|T175180]]
* 17:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/384565
* 17:09 addshore: addshore@integration-slave-docker-c2-m4-d40-1005:/srv/git/mediawiki$ sudo git clone --bare https://gerrit.wikimedia.org/r/p/mediawiki/core.git


== 2017-10-14 ==
== 2022-05-06 ==
* 00:48 MaxSem: reverting
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]
* 00:47 MaxSem: Trying PHP7 mode on depoyment-prep with https://wikitech.wikimedia.org/w/index.php?diff=1772791 (ping [[phab:T173786|T173786]])


== 2017-10-13 ==
== 2022-05-05 ==
* 16:34 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki ([[phab:T177857|T177857]])
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 13:41 zeljkof: Reloading Zuul to deploy {{Gerrit|b5b1dc2c580a465a3fd81f0dda6144dcb1e8f5f7}}
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 10:43 zeljkof: Reloading Zuul to deploy {{Gerrit|320f06501ea72cccdd2d012b06edf2dfb35cd986}}
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins  builds from https://integration.wikimedia.org/ci/view/Beta/


== 2017-10-11 ==
== 2022-05-04 ==
* 19:59 hashar: deployment-prep: deploying jobrunner to catchup with changes.
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 18:19 hashar: beta: rebased puppet master due to a conflict with {{Gerrit|b3c6968b3c}}
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 15:32 _joe_: removing deployment-pdf01, [[phab:T177931|T177931]]
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 08:33 hashar: Image snapshot-ci-jessie-1507710117 in wmflabs-eqiad is ready
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 08:22 hashar: nodepool: refreshing Jessie snapshot after some puppet patches got merged
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2017-10-10 ==
== 2022-05-03 ==
* 17:51 Amir1: add "Ladsgroup" to oversight members in enwiki in beta cluster to test [[phab:T177705|T177705]]
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 16:29 Amir1: adding "Ladsgroup" to admins in wikidatawiki in beta cluster
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2017-10-09 ==
== 2022-05-02 ==
* 13:26 hashar: Upgraded Jenkins to 2.73.1 earlier today
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 08:53 hashar: hard restart integration-slave-docker-1001 via horizon. It is deadlocked somehow. - [[phab:T177749|T177749]]
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]


== 2017-10-06 ==
== 2022-04-29 ==
* 13:22 hashar: Jenkins: adding Maven-3.0.5 to the tool configuration https://integration.wikimedia.org/ci/configureTools/
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 11:58 hashar: Jenkins: installed Warnings plugin
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]
* 11:54 hashar: Jenkins: removing the Violations plugin. It is not used.
* 09:22 hashar: integration: purged bunch of old containers: sudo cumin 'name:slave-docker' 'yes {{!}} docker container prune'


== 2017-10-05 ==
== 2022-04-28 ==
* 19:15 hasharAway: rebooting integration-slave-docker-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 19:11 hasharAway: rebooting integration-slave-jessie-1002 to catch with kernel upgrade and pooling it back in Jenkins - [[phab:T177039|T177039]]
* 18:57 brennen: finished editing mediawiki-new-errors
* 13:16 hashar: Image snapshot-ci-jessie-1507208677 in wmflabs-eqiad is ready
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 11:47 hashar: Refreshing Nodepool Jessie snapshot to get java 8 by default - [[phab:T162828|T162828]]
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC
* 10:56 hashar: integration: unbreak the puppet master. Was stuck do a cherry pick that needed a rebase
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/382361
* 04:16 legoktm: deploying https://gerrit.wikimedia.org/r/382354


== 2017-10-04 ==
== 2022-04-27 ==
* 13:19 andrewbogott: migrating 'deployment-kafka-jumbo-1' to labvirt1017
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029


== 2017-10-03 ==
== 2022-04-26 ==
* 22:38 thcipriani: git stash /srv/mediawiki-staging/php-master/extensions/Echo to fix beta-code-update-eqiad
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 14:26 hashar: Created https://github.com/wikimedia/analytics-wikistats2 - [[phab:T177288|T177288]]
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 14:23 hashar: Gerrit: created analytics/wikistats2.git for fdans - [[phab:T177288|T177288]]
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 07:15 legoktm: deploying https://gerrit.wikimedia.org/r/381937
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 05:33 legoktm: deploying https://gerrit.wikimedia.org/r/381812 https://gerrit.wikimedia.org/r/381378
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204


== 2017-10-02 ==
== 2022-04-25 ==
* 15:52 zeljkof: Reloading Zuul to deploy {{Gerrit|895f33da2a8daeb1fa2634bcd4eddfbbbcd78819}}
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 14:57 addshore: docker push npm image from https://gerrit.wikimedia.org/r/#/c/381384/2
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI
* 14:47 zeljkof: Reloading Zuul to deploy {{Gerrit|9ac7821ae384612ad8249d03e47f264004d1199a}}
* 14:22 zeljkof: Reloading Zuul to deploy {{Gerrit|00890a511274d4b215034979db91708fbf6b0d0f}}
* 12:32 addshore: docker push composer, mediawiki-phan & mediawiki-phpcs latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:23 addshore: docker push ci-jessie, php & php-mediawiki latest tags built from https://gerrit.wikimedia.org/r/381392
* 12:16 addshore: marking integration-slave-docker-1002 as offline [[phab:T177039|T177039]]
* 11:08 zeljkof: Reloading Zuul to deploy {{Gerrit|c1d7b5f1f7db4968cc2acc85539c298865a3b5ff}}


== 2017-09-30 ==
== 2022-04-20 ==
* 14:28 zeljkof: Reloading Zuul to deploy {{Gerrit|c08a3ad67b5bc1cbdb900d6b4138034b4be83c0f}}
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service


== 2017-09-29 ==
== 2022-04-18 ==
* 19:45 hashar: Deleting integration-slave-jessie-php55
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 17:34 zeljkof: Reloading Zuul to deploy {{Gerrit|0e26c8697438a88fbdd62884dcb40e664ba98fa2}}
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 16:42 zeljkof: Reloading Zuul to deploy {{Gerrit|09445b837a03a2ce906fe848aec8350f59ab5898}}
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 15:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7f66813dc0842dadfdb74c9257582aed26f35d60}}
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.
* 14:15 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.logininput.org ( testing w/o delete [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 14:10 tabbycat: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.loginpartner.org --delete ( testing [[phab:T176206|T176206]] / {{Gerrit|7f842058602c}} )
* 13:00 hashar: github: created https://github.com/wikimedia/integration-quibble for gerrit replication
* 12:53 hashar: gerrit: marked labs/tools/grrrit archived
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'weeks ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:53 addshore: addshore@integration-slave-docker-1001:~$ sudo docker ps --filter "status=exited" {{!}} grep 'months ago' {{!}} awk '{print $1}' {{!}} xargs --no-run-if-empty sudo docker rm
* 09:40 addshore: marking integration-slave-docker-1001 as online - [[phab:T177039|T177039]]
* 09:33 addshore: rebooting integration-slave-docker-1001
* 09:10 addshore: wm-ci-docker-push mediawiki-phpcs:v2017.09.29.09.08 & latest https://gerrit.wikimedia.org/r/381413
* 05:59 legoktm: marking integration-slave-docker-1001 as offline - [[phab:T177039|T177039]]
* 00:19 mutante: releases1001 - created user for "no_justification", dropped pass in home dir
* 00:12 mutante: jenkins now configured and running at https://releases.wikimedia.org/ci/ ([[phab:T164030|T164030]]) - but needs additional admin users and puppet is still disabled for temp hack fix


== 2017-09-28 ==
== 2022-04-14 ==
* 16:58 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381242
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 15:37 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.28.15.28
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 11:35 hashar: docker push wmfreleng/tox:v2017.09.28.11.34 Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 11:34 hashar: docker push wmfreleng/ci-jessie:v2017.09.28.11.33 . Adds XDG_CACHE_HOME=/cache https://gerrit.wikimedia.org/r/#/c/380961/
* 09:20 moritzm: upgraded mediawiki04-mediawiki06 in deployment-prep to HHVM 3.18.5
* 08:39 hashar: Deleted integration-saltmaster and deployment-salt02 . Replaced by integration-cumin and deployment-cumin - [[phab:T176314|T176314]]
* 08:32 hashar: Migrated Hiera config from https://wikitech.wikimedia.org/wiki/Hiera:Integration to Horizon
* 08:31 hashar: Removing salt configuration from integration and deployment-prep projects. Replaced by cumin. - [[phab:T176314|T176314]]


== 2017-09-27 ==
== 2022-04-12 ==
* 21:25 hashar: salt is being replaced by cumin instances being deployment-cumin and integration-cumin . Check this out: https://wikitech.wikimedia.org/wiki/Cumin !
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 20:12 hashar: Deleted aptly.integration.eqiad.wmflabs and the https://integration-aptly.wmflabs.org/repo/ webproxy. They were for php5.5 packages on jessie, now available on apt.wm.o - [[phab:T174972|T174972]]
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 19:39 hashar: deployment-prep: purging "ferm" on hosts that no more have it applied via puppet. There were some old iptables rules left around blocking access
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 16:38 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/381024
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 12:38 addshore: Reloading Zuul to deploy - Add lintr-docker-non-voting [integration/config] - https://gerrit.wikimedia.org/r/380746
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141
* 12:27 hasharAway: docker push wmfreleng/tox:v2017.09.27.12.26  https://gerrit.wikimedia.org/r/#/c/380926/
* 11:01 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.27.10.53 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.27.10.51 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/composer:v2017.09.27.10.49 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.27.10.45 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/php:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 11:01 addshore: docker push docker.io/wmfreleng/tox:v2017.09.27.10.21 & latest (From https://gerrit.wikimedia.org/r/#/c/380940)
* 10:08 addshore: docker push docker.io/wmfreleng/ci-jessie:v2017.09.27.09.59 & latest (from https://gerrit.wikimedia.org/r/#/c/378033/5)


== 2017-09-26 ==
== 2022-04-08 ==
* 16:15 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.26.16.09 & latest (from PS11 of https://gerrit.wikimedia.org/r/379479)
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 16:01 addshore: docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.26.15.45 & latest (From PS10 of https://gerrit.wikimedia.org/r/379479)
* 13:58 addshore: added hashar to https://hub.docker.com/u/wmfreleng
* 13:07 moritzm: adding  deployment-videoscaler01 to deployment-prep (stretch-based video scaler)
* 12:45 addshore: Reloading Zuul to deploy - Low prio queue for libraryupdater [integration/config] - https://gerrit.wikimedia.org/r/380307
* 12:28 addshore: fab docker_pull_image:wmfreleng/lintr:v2017.09.26.12.04
* 12:26 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/379818
* 12:06 addshore: docker push docker.io/wmfreleng/ & latest (PS6 of https://gerrit.wikimedia.org/r/378831 )


== 2017-09-25 ==
== 2022-04-07 ==
* 17:07 mutante: Greg is now a contint-admin
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 12:36 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker rmi wmfreleng/operations-puppet:0.0.1 wmfreleng/operations-puppet:0.1.0'
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]
* 12:30 addshore: Reloading Zuul to deploy Refactor 'operations-puppet-tests-docker' into macros for easy reuse [integration/config] - https://gerrit.wikimedia.org/r/379959
* 09:12 moritzm: added deployment-mediawiki07 to deployment-prep (stretch-based app server, WIP)


== 2017-09-24 ==
== 2022-04-06 ==
* 10:49 addshore: addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run "sudo docker images --no-trunc --format '{{.ID}} {{.CreatedSince}}' {{!}} grep ' months' {{!}} awk '{ print $1 }' {{!}} xargs --no-run-if-empty docke r rmi"
* 20:03 thcipriani: rebooting phabricator
* 10:37 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.24.10.33 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2 (actually this time))
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]
* 09:58 addshore: docker push docker.io/wmfreleng/lintr:v2017.09.22.18.51 & latest (https://gerrit.wikimedia.org/r/#/c/378831/2)


== 2017-09-22 ==
== 2022-04-04 ==
* 21:55 tabbycat: Granted Greg G. 'staff' global rights on the beta cluster per request
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 20:37 hashar: Image snapshot-ci-jessie-1506112074 in wmflabs-eqiad is ready
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 20:28 hashar: updating nodepool image for jessie [2/x]
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 20:03 hasharAway: Updating nodepool image for jessie
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 17:22 addshore: docker push docker.io/wmfreleng/tox:v2017.09.22.17.16 & latest # (From current master)
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 15:24 hashar: Restarted Jenkins (out of memory)
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 10:06 hashar: deployement-salt02 migrated hiera config from wikitech to horizon. Removed the class role::deployment::salt_masters
* 08:54 jnuche: redeploying Zuul
* 08:44 hashar: Upgraded docker on integration-slave-docker-1001 and integration-slave-docker-1002 - [[phab:T176267|T176267]]
* 07:13 greg-g: some jsduck jobs are running now, serially, for the backlogged queue. Unsure of starved jobs (integration-config-qa, pywikibot-beta-cluster, etc)
* 07:04 greg-g: deleting stuck mediawiki-core-jsduck-publish jobs in Jenkins UI
* 06:57 greg-g: pinged an opsen, hopefully they'll restart zuul shortly
* 06:45 greg-g: Zuul is stuck, no jobs are processing


== 2017-09-21 ==
== 2022-04-02 ==
* 10:23 elukey: removed {{Gerrit|6fdf6ee653}} from deployment-prep's puppet master cherry picks (seemed an old version of https://gerrit.wikimedia.org/r/#/c/357985)
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases


== 2017-09-20 ==
== 2022-03-31 ==
* 15:46 addshore: reloading zuul for https://gerrit.wikimedia.org/r/#/c/379250/
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
* 13:59 addshore: docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.20.13.49 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/composer:v2017.09.20.13.44 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/zuul-cloner:v2017.09.20.13.44 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/php-mediawiki:v2017.09.20.13.43 & latest # built from master
* 13:59 addshore: docker push docker.io/wmfreleng/php:v2017.09.20.13.40 & latest # built from master
* 13:07 tabbycat: deployment-prep Ran cleanupSpam.php on deploymentwiki. Further testing with regards to ongoing development and updating of the script.
* 11:53 addshore: Reloading Zuul (Testing)


== 2017-09-19 ==
== 2022-03-29 ==
* 17:26 legoktm: removed rights from User:Sau226 on beta cluster due to block of account used for browser tests
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 09:13 tabbycat: Re-run previous script and it worked this time, see https://deployment.wikimedia.beta.wmflabs.org/wiki/Template_talk:Rotate/en
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 09:11 tabbycat: Ran mwscript cleanupSpam.php on the beta cluster, but it didn't worked (looks it is not fetching the domains properly)
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771


== 2017-09-18 ==
== 2022-03-28 ==
* 20:12 addshore: deleted unused images that were *months old* on docker slaves
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 19:01 addshore: addshore@contint1001:~$ sudo service zuul reload
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 18:45 thcipriani: reloading zuul to test https://gerrit.wikimedia.org/r/#/c/378665/2
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 16:53 elukey: removed https://gerrit.wikimedia.org/r/#/c/377753/ from the git cherry-picks in operations/puppet on puppetmaster02
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]


== 2017-09-17 ==
== 2022-03-27 ==
* 18:59 addshore: Reloading Zuul to deploy archiving of 2 extensions
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]


== 2017-09-14 ==
== 2022-03-26 ==
* 19:37 tgr: updated PrivateSettings.php for [[phab:T175868|T175868]]
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 10:38 elukey: cherry-pick https://gerrit.wikimedia.org/r/#/c/377753/7 on deployment-prep's puppetmaster02 to test it on the new kafka jumbo instances
* 10:35 hashar: CI puppet master:  added class geoip::data::package and parameters:  puppetmaster::geoip::fetch_private: false  puppetmaster::geoip::use_proxy: false    - [[phab:T175864|T175864]]


== 2017-09-13 ==
== 2022-03-25 ==
* 10:13 addshore: docker push docker.io/wmfreleng/operations-puppet:v2017.09.13.09.23 (#d693f74c9b3404220a2ad2934f526d4f4455914b)
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 09:25 hashar: Deleting integration-slave-trusty-1003 and integration-slave-trusty-1001  - [[phab:T175696|T175696]]
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 09:14 hashar: nodepool:  openstack image delete image-ci-trusty  - [[phab:T175696|T175696]]
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 07:49 hashar: Jenkins: removing the Ubuntu JDK from https://integration.wikimedia.org/ci/configureTools/
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - [[phab:T161882|T161882]]
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord
* 07:40 hashar: jenkins: on nodes, removing the labels phpflavor-*  they are no more needed - T 161882


== 2017-09-12 ==
== 2022-03-24 ==
* 20:35 hashar: pooling integration-slave-jessie-1003 and integration-slave-jessie-1004
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 19:40 hashar: hacked integration-slave-jessie hosts to ship them php5.5
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 18:49 hasharAway: nodepool: deleted image image-ci-trusty_old_20170804  Keeping image-ci-trusty just in case
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 14:57 hashar: Deleted all left over jenkins jobs having ci-trusty-wikimedia label. - [[phab:T161882|T161882]]
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
* 14:46 hashar: provisionning integration-slave-jessie-1003 and integration-slave-jessie-1004 to move php55lint to them.  NOT READY YET - [[phab:T161882|T161882]]
* 14:05 hashar: Deleting integration-slave-trusty-1004  - [[phab:T161882|T161882]]
* 13:09 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-815477
* 11:09 hashar: Image snapshot-ci-jessie-1505213295 in wmflabs-eqiad is ready
* 10:48 hashar: nodepool: force updating jessie image to grab php5.5-luasandbox - [[phab:T161882|T161882]] [[phab:T174972|T174972]]


== 2017-09-11 ==
== 2022-03-23 ==
* 23:27 thcipriani: restarting jenkins
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 22:38 legoktm: deploying https://gerrit.wikimedia.org/r/377361
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 12:47 hashar: Nodepool: refreshing jessie snapshot to get php5.5-luasandbox installed


== 2017-09-10 ==
== 2022-03-22 ==
* 01:44 bd808: nodepool running steadily again, but has been heavily throttled to hopefully prevent another weekend thundering herd of doom failure for the OpenStack backend
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945


== 2017-09-09 ==
== 2022-03-21 ==
* 22:15 bd808: `sudo journalctl -u nodepool --since today --no-pager` shows many LaunchStatusException failures.
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]


== 2017-09-07 ==
== 2022-03-18 ==
* 13:02 hashar: nodepool: Image snapshot-ci-jessie-1504788047 in wmflabs-eqiad is ready {{!}} [[phab:T174972|T174972]]
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 11:58 hashar: nodepool: updating snapshot-ci-jessie  to add php5.5-redis {{!}} [[phab:T161882|T161882]] [[phab:T174972|T174972]]
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 11:10 addshore: Reloading Zuul to deploy "Add gate-submit jobs for analytics/wmde/* repos"
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
* 02:44 legoktm: deploying https://gerrit.wikimedia.org/r/376460


== 2017-09-06 ==
== 2022-03-17 ==
* 21:32 bearND: Update mobileapps to {{Gerrit|2cb6281}} ([[phab:T168848|T168848]] [[phab:T169277|T169277]] [[phab:T169274|T169274]] [[phab:T162179|T162179]] [[phab:T164033|T164033]] [[phab:T167921|T167921]] [[phab:T174698|T174698]] [[phab:T168848|T168848]] [[phab:T174808|T174808]])
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]


== 2017-09-05 ==
== 2022-03-16 ==
* 23:03 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/376034
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 19:34 gilles: deployed PrivateSettings.php change to add Thumbor username to Swift configuration
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep


== 2017-09-04 ==
== 2022-03-15 ==
* 15:59 zeljkof: Reloading Zuul to deploy {{Gerrit|ca1c6ec05122e6f50a5a52fecbbd2bd8e882ad87}}
* 18:26 brennen: gitlab: removed most existing /people groups
* 12:21 hashar: Image snapshot-ci-jessie-1504527142 in wmflabs-eqiad is ready
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 11:37 hashar: nodepool: refreshing jessie snapshot
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 10:03 addshore: Reloading Zuul to deploy mwext-php70-phan-jessie-docker experimental job
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 00:42 legoktm: legoktm@contint1001:/srv/zuul/git/mediawiki/libs$ sudo -u zuul rm -rf XMPReader
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2017-09-02 ==
== 2022-03-14 ==
* 08:32 legoktm: rm -rf /var/logs/kafka on deployment-kafka01 to free up disk space
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2017-08-31 ==
== 2022-03-11 ==
* 23:32 Krenair: fixed deployment-imagescaler0[12] puppet by installing a package and file manually, some puppetisation still needed - https://phabricator.wikimedia.org/T174746
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
* 23:04 Krenair: that also did deployment-cache-(upload{{!}}text)04
* 22:50 Krenair: fixed deployment-ms-be0[34] puppet by removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/371582/1 - details in a comment there
* 15:49 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/374970/
* 14:52 zeljkof: Reloading Zuul to deploy {{Gerrit|dd504839b3a48f431199785421751a5e3e38af57}}
* 10:06 zeljkof: Reloading Zuul to deploy {{Gerrit|e91b30ff6df435a6a3c55fe978be6bbf1f1cbde4}}
* 06:25 legoktm: deploying https://gerrit.wikimedia.org/r/374937


== 2017-08-30 ==
== 2022-03-10 ==
* 12:49 hashar: gerrit: marked wikimedia/communications/WMBlog as read-only - [[phab:T172372|T172372]]
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 20:03 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2017-08-29 ==
== 2022-03-09 ==
* 15:39 hashar: Created integration-slave-jessie-php55 to try out a php5.5 package on Jessie - [[phab:T161882|T161882]]
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 15:06 hashar: nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-793795
* 08:45 hashar: Restarting Jenkins for openjdk update
* 08:11 hashar: refreshing all Jenkins jobs with a newer version of JJB


== 2017-08-28 ==
== 2022-03-08 ==
* 14:54 hashar: integration: rebase integration puppet master. Got a conflict due to r -> r_lang renaming ( https://gerrit.wikimedia.org/r/#/c/363337/ )
* 20:31 brennen: requiring 2fa for all users under /repos
* 08:52 hashar: gerrit: added ldap/ciadmin  to the 'integration' group.  [[phab:T169557|T169557]] [[phab:T173233|T173233]]


== 2017-08-25 ==
== 2022-03-07 ==
* 15:11 zeljkof: Reloading Zuul to deploy {{Gerrit|b6704e249f0d9e83de5835908eaddff40f104e2f}}
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2017-08-24 ==
== 2022-03-04 ==
* 21:55 mdholloway: disk was full on integration-slave-jessie-android; deleted ~8gb of old screenshots from /tmp to clear some space
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 16:54 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/372780/7
* 19:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768068
* 15:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/371138/
* 11:56 zeljkof: Reloading Zuul to deploy {{Gerrit|c20a7402467efb669a30dc06ec70c41fc6853193}}


== 2017-08-22 ==
== 2022-03-03 ==
* 06:49 legoktm: deploying https://gerrit.wikimedia.org/r/370695
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 06:44 legoktm: deploying https://gerrit.wikimedia.org/r/372229
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2017-08-21 ==
== 2022-03-02 ==
* 18:18 mutante: addshore is now a contint-admin
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 15:27 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/767493
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04


== 2017-08-18 ==
== 2022-02-28 ==
* 20:21 legoktm: deploying https://gerrit.wikimedia.org/r/372222
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin


== 2017-08-16 ==
== 2022-02-26 ==
* 23:46 legoktm: reloading Zuul to deploy https://gerrit.wikimedia.org/r/372208 https://gerrit.wikimedia.org/r/371653 https://gerrit.wikimedia.org/r/371757 https://gerrit.wikimedia.org/r/371640
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases


== 2017-08-15 ==
== 2022-02-24 ==
* 17:57 bearND: Update mobileapps to {{Gerrit|33b80dd}} ([[phab:T172829|T172829]] [[phab:T152441|T152441]] [[phab:T172021|T172021]] [[phab:T103362|T103362]])
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2017-08-14 ==
== 2022-02-23 ==
* 09:46 TabbyCat: maurelio@deployment-tin:/srv/mediawiki/dblists$ expanddblist flow-computed > /home/maurelio/flow-test.dblist (to test expandblist for a patch I am working on)
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 20:10 dancy: Updating scap in beta
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/  # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2017-08-11 ==
== 2022-02-22 ==
* 20:25 addshore: added mediawiki::maintenance::wikidata to deployment-tin
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])


== 2017-08-07 ==
== 2022-02-21 ==
* 15:11 thcipriani: restarting jenkins for plugin update
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 07:27 hashar: Refreshing all Jenkins jobs


== 2017-08-06 ==
== 2022-02-20 ==
* 13:28 TabbyCat: Ran mwscript extensions/WikimediaMaintenance/dumpInterwiki.php deploymentwiki on the beta cluster
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo


== 2017-08-04 ==
== 2022-02-19 ==
* 15:21 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/370222/2
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]


== 2017-08-03 ==
== 2022-02-18 ==
* 12:02 hashar: Added  integration-slave-docker-1004 to the pool of jenkins slaves - [[phab:T150502|T150502]]
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 10:12 hashar: gerrit: marked wikimedia/communications/WP-Victor read-only and [ARCHIVED] - [[phab:T107430|T107430]]
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 04:50 SMalyshev: update cherry-pick for https://gerrit.wikimedia.org/r/#/c/299825/8 on deployment-puppetmaster02.deployment-prep.eqiad.wmflabs
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}


== 2017-08-02 ==
== 2022-02-17 ==
* 22:08 MaxSem: Running rebuildall.php on beta ruwiki
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 20:17 bearND: Update mobileapps to {{Gerrit|2d8e8f6}}
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 11:31 hashar: Image snapshot-ci-jessie-1501673225 in wmflabs-eqiad is ready [[phab:T169602|T169602]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 10:51 hashar: Image snapshot-ci-jessie-1501670727 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 09:02 hashar: Regenerating Nodepool Jessie image from scratch to get rid of tox 1.9.2 installed under /usr/local - [[phab:T169602|T169602]]
* 11:12 hashar: Bringing deployment-deploy03 back
* 08:44 hashar: Image snapshot-ci-jessie-1501662758 in wmflabs-eqiad is ready - [[phab:T169602|T169602]]
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome
* 08:42 hashar: - [[phab:T169602|T169602]]
* 08:32 hashar: Regenerating Nodepool jessie image to upgrade tox from 1.9.2 to 2.5.0 - [[phab:T169602|T169602]]


== 2017-08-01 ==
== 2022-02-16 ==
* 15:45 hashar: Image snapshot-ci-jessie-1501601670 in wmflabs-eqiad is ready  && purging old instances [[phab:T161861|T161861]]
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 15:44 hashar: Debug: Executing '/usr/bin/npm install -g npm@3.8.3' -  [[phab:T161861|T161861]]
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 15:34 hashar: Refreshing nodepool Jessie image to bump npm from 2.x to 3.8.x  [[phab:T161861|T161861]]
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}


== 2017-07-28 ==
== 2022-02-15 ==
* 21:11 MaxSem: Dropped table wikigrok_questions from beta enwiki
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|47a07e06e5e348902a3cdc51cff9c2bed87ab6df}}
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 00:17 Krinkle: Testing job insertion on beta cluster from deployment-tin triggers PHP Notice: Undefined index: uuid in EventBus/JobQueueEventBus.php:102, PHP Notice: Undefined index: sha1 in EventBus/JobQueueEventBus.php:99
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}


== 2017-07-26 ==
== 2022-02-14 ==
* 21:35 Reedy: kill two long running update.php jobs on deployment-tin
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 13:39 zeljkof: Reloading Zuul to deploy {{Gerrit|8787b4bbe75691e35792864434f8e09f3aaf6e53}}
* 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 12:04 zeljkof: Reloading Zuul to deploy {{Gerrit|79781d8b08a6dbd4d2e2b5554fbb04154c0acf48}}
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 11:39 zeljkof: Reloading Zuul to deploy {{Gerrit|723ab49039714ae2507ef43a94884de1a6bb347e}}
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 11:31 hashar: realign installed debian packages on integration-slave-jessie-1001 and integration-slave-jessie-1002 -  [[phab:T171724|T171724]]
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon
* 09:25 hashar: deployment-tin deleting temporary l10n cache from July 19th 20:09 at /tmp/scap_l10n_3608512748  1.5G
* 09:24 hashar: deployment-cache-upload04 deployment-cache-text04 upgraded logster 0.0.10-1~jessie1 -> 0.0.10-2~jessie1 - [[phab:T171318|T171318]]


== 2017-07-25 ==
== 2022-02-12 ==
* 16:51 halfak: deploying ores {{Gerrit|835d848}} [[phab:T171505|T171505]]
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])


== 2017-07-24 ==
== 2022-02-10 ==
* 21:56 bearND: Update mobileapps to {{Gerrit|b608ec8}}
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 15:03 hashar: Added webperformance Jenkins slave https://integration.wikimedia.org/ci/computer/webperformance/ with a single executor - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk - [[phab:T166756|T166756]]
* 14:57 hashar: recreating integration-webperf instance has simply "webperformance"  Same 2CPU / 2GB RAM / 40G disk
* 14:40 hashar: Booting integration-webperf instance 2CPU / 2GB RAM / 40G disk. Intended to host webperformance long running jobs . [[phab:T166756|T166756]]
* 11:02 hashar: Removing profile::swift::storage::labs class from deployment-ms-be03 and deployment-ms-be04 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]  [[phab:T171454|T171454]]
* 10:59 hashar: Removing class from deployment-trending01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:54 hashar: Removing classes from deployment-sca02 and deployment-sca03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:32 hashar: Removing profile::etcd from deployment-conf03 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:12 hashar: Removing role::mathoid from deployment-mathoid to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:09 hashar: Removing role::changeprop from deployment-changeprop to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:06 hashar: Removing role::ocg from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]
* 10:02 hashar: Removing role::mobileapps from deployment-mcs01 to let puppet run. Reapplying it after. - [[phab:T171174|T171174]]


== 2017-07-21 ==
== 2022-02-09 ==
* 14:55 hashar: Jenkins: upgraded Android Emulator plugin with https://gerrit.wikimedia.org/r/#/c/366253/ && https://gerrit.wikimedia.org/r/#/c/366484/ -  [[phab:T150623|T150623]]
* 15:22 taavi: deleted shutoff deployment-mx02
* 14:12 hashar: added novaadmin to deployment-prep as a regular user. That lets MediaWiki OpenStack API list the instances [[phab:T171280|T171280]]
* 13:56 hashar: Created github mirror repo https://github.com/wikimedia/wikibase-wikiba.se  [[phab:T171160|T171160]]
* 10:46 hashar: Gerrit: created wikibase/wikibase.se repo for Amir1 / [[phab:T171160|T171160]]


== 2017-07-20 ==
== 2022-02-08 ==
* 16:42 hashar: How to fix ssh access on beta cluster instances: https://phabricator.wikimedia.org/T171174#3456966
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 15:30 hashar: deployment-prep : removing project wide puppet classes from https://horizon.wikimedia.org/project/puppet/   All are role::eventlogging::analytics::*
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 15:08 hashar: removed profile::recommendation_api  from deployment-sca01  to try to fix the ssh access for mobrovac  [[phab:T171173|T171173]]  [[phab:T171174|T171174]]
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
* 14:57 zeljkof: reloading Zuul to deploy {{Gerrit|80b9d85}}
* 14:31 hashar: deployment-prep: manually cleaned out the puppet master configuration. It was all screwed up.  Notably I removed bits about the puppetdb
* 10:20 zeljkof: Reloading Zuul to deploy {{Gerrit|80b9d855443a2f572d877b280783110684344c5d}}
* 09:17 hashar: Spawning and pooling integration-slave-docker-1003  as replacement to integration-slave-docker-1000 (broken)  - [[phab:T150502|T150502]]
* 09:03 hashar: Restoring castorby updating all jobs  to point to castor02 ( https://gerrit.wikimedia.org/r/366524 ) Starts with a cold cache :( - [[phab:T171148|T171148]]
* 08:53 hashar: Created castor02.integration.eqiad.wmflabs with puppet role role::ci::castor::server and adding it to Jenkins. Will then update the Jenkins jobs to point to it - [[phab:T171148|T171148]]
* 08:00 hashar: Disabled castor entirely via https://gerrit.wikimedia.org/r/366520 . The instance is broken - [[phab:T171148|T171148]]
* 07:55 hashar: Refreshing all Jenkins jobs defined in JJB in order to then disable castor entirely for [[phab:T171148|T171148]]
* 07:09 _joe_: rebooting castor, jobs are failing, and no one seems able to login
* 07:05 _joe_: adding myself to projectadmins for integration, trying to troubleshoot castor
* 01:38 thcipriani: scap on beta was failing because during the ldap downtime puppet created a shadow mwdeploy user, fixed using vipw and vigr


== 2017-07-19 ==
== 2022-02-07 ==
* 14:43 hashar: Jenkins: uploaded a patched android-emulator plugin for [[phab:T150623|T150623]] and restarting Jenkins
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 13:55 hashar: Jenkins: added JDK "Debian - OpenJdk 7" with  JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
* 12:54 hashar: Gerrit: created repo integration/jenkinsci/android-emulator-plugin.git owned by access group integration-jenkinsci-android-emulator-plugin  which has Mholloway - [[phab:T170904|T170904]]


== 2017-07-18 ==
== 2022-02-04 ==
* 16:26 halfak: manually restarted uwsgi-ores and celery-ores-worker on deployment-sca03
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
* 16:19 halfak: manually installed "aspell-el" on deployment-sca03 (work around for ongoing puppet issues)
* 09:04 hashar: deleted integration-slave-trusty-1006
* 03:57 twentyafterfour: Fixed deployment-imagescaler01 by cherry-picking https://gerrit.wikimedia.org/r/#/c/365891/ on deployment-puppetmaster02


== 2017-07-17 ==
== 2022-02-03 ==
* 18:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/365198
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]


== 2017-07-14 ==
== 2022-02-02 ==
* 20:16 Amir1: cpan[1]> install LWP::UserAgent on tin
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta


== 2017-07-13 ==
== 2022-02-01 ==
* 17:04 thcipriani: restarting jenkins for updates
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]


== 2017-07-12 ==
== 2022-01-31 ==
* 20:07 bearND: Update mobileapps to {{Gerrit|d30dae2}}
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 18:19 greg-g: where "things" == nodepool instance delete/creation
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]
* 18:18 greg-g: things are back to a bad state, chase etc investigating
* 17:52 greg-g: nodepool is back to making instances and running jobs, thanks Cloud team
* 17:22 greg-g: CI is backed up, only one nodepoll instance running for the last long while, many in building
* 00:35 legoktm: deploying https://gerrit.wikimedia.org/r/364628


== 2017-07-11 ==
== 2022-01-28 ==
* 21:30 legoktm: deploying https://gerrit.wikimedia.org/r/364601
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`


== 2017-07-09 ==
== 2022-01-27 ==
* 01:15 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki ([[phab:T170026|T170026]], [[phab:T165716|T165716]])
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2  # [[phab:T299431|T299431]]
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300214|T300214]]
* 00:24 thcipriani: restarting jenkins


== 2017-07-07 ==
== 2022-01-26 ==
* 14:53 hashar: deployment-prep: change webproxy http://recommendation-api-beta.wmflabs.org/ to deployment-sca02 (has the proper security rule) - [[phab:T148129|T148129]]
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 14:53 hashar: deployment-prep: add port 9632 to security group "sca" https://horizon.wikimedia.org/project/access_and_security/security_groups/593/ - [[phab:T148129|T148129]]
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 14:03 hashar: Image snapshot-ci-trusty-1499435837 in wmflabs-eqiad is ready
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 13:57 hashar: Nodepool: updating snapshot-ci-trusty
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 13:56 hashar: Nodepool: uploaded new Ubuntu Trusty image
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2017-07-06 ==
== 2022-01-25 ==
* 17:28 thcipriani: committed changes to modules/kafkatee on deployment-puppetmaster02 since having them uncommitted broke git-sync-upstream
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 16:20 hashar: Deleting Nodepool snapshot snapshot-ci-jessie-1499350442 - faulty php7.0-sqlite package that breaks phan jobs - [[phab:T169904|T169904]]
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 15:29 hashar: deployment-cache-upload04 manually ran apt-get upgrade to downgrade ldap-utils and libldap-2.4-(caused puppet failure)
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 14:14 hashar: regenerating mediawiki-core-qunit-selenium-jessie jenkins job
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 12:05 hashar: deployment-prep created Web proxy for recommendation-api-beta.wmflabs.org -> http://10.68.20.183:9632  (deployment-sca01) for schana
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 02:38 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/363519
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2017-07-04 ==
== 2022-01-24 ==
* 14:10 hashar: manually upgraded apache2 on deployment-puppetmaster02  see [[phab:T159254|T159254]]
* 21:44 Reedy: unstick beta ci jobs
* 13:33 hashar: beta cluster puppet is broken: Error: Could not send report: Connection refused - connect(2) for "deployment-puppetmaster02.deployment-prep.eqiad.wmflabs" port 8140
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 09:28 hashar: gerrit: marking read-only  mediawiki/extensions/Nonlinear - [[phab:T169519|T169519]]
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2017-07-03 ==
== 2022-01-22 ==
* 11:34 hashar: jenkins: refreshing all jobs and updating the castor-save bit ( https://gerrit.wikimedia.org/r/#/c/361843/ )
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts


== 2017-06-30 ==
== 2022-01-21 ==
* 08:16 hashar: Gerrit: changing repos to read-only: analytics/kraken analytics/kraken/deploy  analytics/vagrant/kraken - [[phab:T169303|T169303]]
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`


== 2017-06-29 ==
== 2022-01-20 ==
* 23:17 legoktm: deploying https://gerrit.wikimedia.org/r/362314
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2017-06-28 ==
== 2022-01-19 ==
* 15:55 hashar: beta: git gc mediawiki repos in /srv/mediawiki-staging
* 18:20 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 15:47 hashar: beta: git -C /srv/deployment/ores/deploy/submodules/editquality gc  (saving 380MBytes)
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 15:33 hashar: running git gc under /srv/mediawiki-staging
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]
* 14:43 hashar: pypi.python.org is back again - [[phab:T169091|T169091]]
* 14:33 elukey: running alter tables on the EL database in deployment-eventlogging03.deployment-prep.eqiad.wmflabs
* 14:06 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates and [[phab:T169091|T169091]]
* 14:04 hashar: pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python  - See https://status.python.org for updates
* 10:06 hashar: Unblocked beta cluster jenkins job. Have been stalled for a while


== 2017-06-27 ==
== 2022-01-18 ==
* 22:58 Amir1: cherry-picking gerrit:360891/3
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 22:42 Amir1: cherry-picking gerrit:360891/2
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 21:58 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'string' --property-id P34
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0  with proper PHP version in the images # [[phab:T299389|T299389]]
* 18:31 hashar: Image snapshot-ci-jessie-1498587497 in wmflabs-eqiad is ready - [[phab:T169004|T169004]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # [[phab:T299389|T299389]]
* 18:18 hashar: Regenerating Jessie nodepool image to hopefulyl bring back hhvm-tidy package - [[phab:T169004|T169004]]
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 17:39 Amir1: running mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'external-id' --property-id P34
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2017-06-26 ==
== 2022-01-17 ==
* 22:24 halfak: deploying ores-prod-deploy:82dfd56 to beta (note: [[phab:T168099|T168099]])
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 22:20 halfak: deploying ores-prod-deploy:82dfd56 to beta
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 20:33 bearND: Update mobileapps to {{Gerrit|0b05026}}
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
* 18:44 hashar: nodepool image-delete 1636  # Deletes snapshot-ci-trusty-1498491445 which lack nodejs when we still need it.
* 18:23 twentyafterfour: renamed previously active image to 'image-ci-trusty_bad_20170626'
* 18:22 twentyafterfour: reverted nodepool image-ci-trusty to previous version 'image-ci-trusty-old_20170626'
* 15:41 hashar: Image snapshot-ci-trusty-1498491445 in wmflabs-eqiad is ready
* 15:34 hashar: Rebuilding nodepool image for trusty and regenerating snapshots
* 09:19 hashar: gerrit: marked wikimedia/bugzilla/* repos read-only


== 2017-06-24 ==
== 2022-01-14 ==
* 06:02 legoktm: deployment-flourine02 /srv partition is alerting on low disk space but once logs get automatically gzip'd it should be fine
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta


== 2017-06-23 ==
== 2022-01-13 ==
* 20:59 hasharAway: deployment-db03 reinstall ldap-utils, libldap-2.4-2  2.4.44+dfsg-4~bpo8+1 > 2.4.41+dfsg-1+wmf1
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 20:54 hasharAway: apt-get upgrade deployment-elastic06
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2017-06-22 ==
== 2022-01-12 ==
* 19:02 Amir1: cherry-picking gerrit:360891/1 ([[phab:T163922|T163922]])
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group .  That also makes him an owner to wikimedia/discovery/* - [[phab:T168588|T168588]]
* 13:35 hashar: Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group . That also makes him an owner to wikimedia/discovery/*
* 08:18 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x! - [[phab:T168511|T168511]]
* 08:16 hashar: deployment-prep: removed /etc/apt/preferences.d/puppet.pref which was pinning puppet packages to jessie-backports and hence 4.8.x!
* 08:12 hashar: deployment-prep: upgraded puppet to 3.8.5  on all instances


== 2017-06-21 ==
== 2022-01-11 ==
* 20:03 bearND: Update mobileapps to {{Gerrit|21f771d}}
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 19:54 hashar: deployment-tin stopped keyholder and armed it
* 19:25 hashar: hard rebooting deployment-db04
* 19:20 hashar: hard rebooting deployment-db03
* 18:52 hashar: Removing /etc/apt/sources.list.d/wikimedia_mariadb.list  (content: deb http://apt.wikimedia.org/wikimedia precise-wikimedia mariadb  )
* 18:51 hashar: fixing up apt config on deployment-db03 and deployment-db04 /  upgrade packages and kernel / reboot
* 17:02 hashar: upgrading kernel and puppet on deployment-mcs01 deployment-restbase01 and deployment-restbase02 - [[phab:T168541|T168541]]
* 17:00 hashar: upgrading kernel and puppet on deployment-changeprop and deployment-conf03 - [[phab:T168541|T168541]]
* 16:56 hashar: upgrading kernel and puppet on deployment-aqs01 deployment-aqs02 and deployment-aqs03 - [[phab:T168541|T168541]]
* 16:38 hashar: rebooting deployment-cache-upload04 and deployment-cache-text-04 - [[phab:T168541|T168541]]
* 16:29 hashar: upgrading deployment-apertium02 and deployment-eventlogging04  - [[phab:T168541|T168541]]
* 16:23 hashar: upgrade and reboot deployment-prometheus01
* 16:11 hashar: rebooting deployment-ms-fe02
* 16:11 hashar: rebooting deployment-ms-be04
* 16:09 hashar: rebooting deployment-ms-be03
* 16:03 hashar: upgrading deployment-ms-fe02  deployment-ms-be03 and deployment-ms-be04
* 15:57 hashar: apt-get upgrade and reboot of deployment-memc04 and deployment-memc05
* 15:52 hashar: rebooting deployment-etcd-01
* 15:48 hashar: apt-get upgrade deployment-etcd-01
* 15:35 hashar: deployment-prep changing Varnish director for citoid from citoid.wmflabs.org to citoid-beta.wmflabs.org  ( via https://horizon.wikimedia.org/project/prefixpuppet/ ) - [[phab:T168519|T168519]]
* 14:41 hashar: deployment-tmh01 is down for some reason
* 14:21 hashar: deployment-prep: force running puppet on all instances
* 14:17 hashar: finally fixed puppet on deployment-prep !
* 14:02 hashar: deployment-puppmaster (cd /etc/puppet && ln -s /var/lib/git/operations/puppet/manifests && ln -s /var/lib/git/operations/puppet/modules)
* 13:26 hashar: deployment-prep: puppet master got erroneously upgrade to puppet* 4.8.  Roll it back to 3.8 which fail, and then back to 3.7!
* 12:47 hashar: broke deployment-prep puppet master while upgrading it :(
* 12:28 hashar: deployment-imagescaler01 removed puppetmaster and puppetmaster-common packages
* 12:04 hashar: apt-get dist-upgrade on deployment-mediawiki hosts
* 11:59 hashar: armed keyholder on deployment-tin and deployment-mira
* 11:15 hashar: deployment-cache-text04 : apt-get dist-upgrade
* 11:12 hashar: varnish fails on deployment-cache-text04
* 11:08 hashar: deployment-prep : rebooting deployment-tin deployment-mira deployment-cache-text04 deployment-cache-upload04
* 11:00 hashar: deployment-prep apt-get upgrade and reboot all hosts
* 10:21 hashar: deployment-zotero01  apt-get upgrade and rebooted
* 09:59 hashar: integration: removing swift / python-swift from integration-puppetmaster01
* 09:57 hashar: Upgrading puppet 3.7.2 .. 3.8.5 on integration-slave-docker-1001 and integration-slave-docker-1002
* 09:39 hashar: integration: deleting swift and and swift-storage-01  unused
* 09:38 hashar: upgrading/Rebooting all instances from integration project to catch up with Linux kernel upgrades


== 2017-06-20 ==
== 2022-01-10 ==
* 19:25 hashar: Nodepool rate being bumped from 1 query per 6 seconds to 1 query per 5 seconds ( https://gerrit.wikimedia.org/r/#/c/358601/ )
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* afk: deployment-tin stuck on post-merge queue for the past 13 hours, unstuck now
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]


== 2017-06-19 ==
== 2022-01-05 ==
* afk: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/360091/
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 08:29 hashar: Gerrit: added Ladsgroup to 'mediawiki' group - [[phab:T165860|T165860]]
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2017-06-18 ==
== 2022-01-04 ==
* 19:26 Reedy: Re-enabled beta-update-databases-eqiad as wikidatawiki takes < 10 minutes [[phab:T168036|T168036]] [[phab:T167981|T167981]]
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 19:25 Reedy: A lot of items on beta wikidatawiki deleted [[phab:T168036|T168036]] [[phab:T167981|T167981]]
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383


== 2017-06-16 ==
== 2022-01-03 ==
* 23:41 Reedy_: also deleting a lot of Property:P* pages on beta wikidatawiki [[phab:T168106|T168106]]
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001Restarted CI Jenkins.
* 22:55 Reedy: deleting Q100000-Q200000 on beta wikidatawiki [[phab:T168106|T168106]]
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
* 19:04 Reedy: disabled beta-update-databases-eqiad because it's not doing much useful atm
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|18a50a707eac0bcdd88f48f2321af78ee399a4eb}}
* 14:40 hashar: integration-slave-jessie-1001  apt-get upgrade to downgrade python-pbr to 0.8.2 as pinned since [[phab:T153877|T153877]]/usr/bin/unattended-upgrade magically upgraded it for some reason
* 06:49 Reedy: script upto `Processed up to page 336425 (Q235372)`... hopefully it's finished by morning
* 03:13 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 03:10 Reedy: running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen as root on deployment-tin for [[phab:T168036|T168036]]
* 02:23 Reedy: cherry-picked https://gerrit.wikimedia.org/r/#/c/354932/ onto beta puppetmaster


== 2017-06-15 ==
* 16:34 RainbowSprinkles: deployment-prep: Disabled database updates for awhile, running it by hand
* 10:39 hashar: apt-get upgrade on deployment-tin
* 00:52 thcipriani: deployment-tin jenkins agent borked for 4 hours, should be fixed now


== 2017-06-14 ==
{{SAL-archives/Release Engineering}}
* 12:24 hashar: gerrit: marked mediawiki/skins/Donate has read-only ( https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/skins/Donate ) - [[phab:T124519|T124519]]


== 2017-06-13 ==
* 22:05 hashar: Zuul resarted manually from a terminal on contint1001. It does not have any statsd configuration so we will miss metrics for a bit till it is restarted properly.
* 21:13 hashar: Gracefully restarting Zuul
* 20:37 hashar: Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4)
* 20:31 hashar: Nodepool: deleted a bunch of Trusty instances.  It scheduled lot of them that are taking slots in the pool. Better have jessie nodes to be spawned instead since there is high demand for them
* 20:19 hashar: deployment-prep: added Polishdeveloper to the "importer" global group. https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/Polishdeveloper  - [[phab:T167823|T167823]]
* 18:47 andrewbogott: root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter"
* 18:46 andrewbogott: using salt to "apt-get -y install facter" on all deployment-prep instances
* 18:38 andrewbogott: restarting apache2 on deployment-puppetmaster02
* 18:37 andrewbogott: doing a git fetch and rebase for deployment-puppetmaster02
* 17:00 elukey: hacking apache on mediawiki05 to test rewrite rules
* 16:04 Amir1: cherry-picked 357985/4 on puppetmaster
* 15:59 halfak: deployed ores-prod-deploy:862aea9
* 13:47 hashar: nodepool force running puppet for: lower min-ready for trusty [puppet] - https://gerrit.wikimedia.org/r/356466
* 10:53 elukey: rolling restart of all kafka brokers to pick up the new zookeper change (only deployment-zookeeper02 available)
* 10:36 elukey: delete deployment-zookeeper01 (old trusty instance, replaced with a jessie one)
* 09:50 elukey: big refactoring for zookeeper merged in operations/puppet - https://gerrit.wikimedia.org/r/#/c/354449 - ping the Analytics team for any issue
== 2017-06-12 ==
* 14:22 hashar: Image snapshot-ci-trusty-1497276913 in wmflabs-eqiad is ready
* 14:15 hashar: Nodepool: regenerating Trusty images to confirm that removal of keystone admin_token is a noop for nodepool - [[phab:T165211|T165211]]
* 12:44 hashar: Image snapshot-ci-jessie-1497270581 in wmflabs-eqiad is ready
* 12:30 hashar: nodepool: refreshing Jessie snapshot to upgrade HHVM from 3.12 to 3.18 - [[phab:T167493|T167493]] [[phab:T165074|T165074]]
* 08:47 hashar: deployment-prep : salt -v '*' cmd.run 'apt-get clean'
== 2017-06-09 ==
* 20:30 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358092/1
* 18:50 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/358067/3
== 2017-06-07 ==
* 17:49 elukey: forced /usr/local/bin/git-sync-upstream manually on puppetmaster02
* 17:30 elukey: manually fixed rebase issue for operations/puppet on puppetmaster02 (empty commit due to the change for scap3 and jobrunners)
* 09:33 elukey: restart kafka brokers to pick up the new zookeeper settings
* 09:00 elukey: adding deployment-zookeeper02.eqiad.wmflabs to Hiera:deployment-prep
* 08:43 gehel: upgrading kibana to v5.3.3 on deployment-logstash2
* 08:35 gehel: rolling back to kibana 5.3.2, incompatible elasticsearch version
* 08:28 gehel: upgrading kibana to v5.4.1 on deployment-logstash2
== 2017-06-06 ==
* 14:34 hashar: deleting buildlog.integration.eqiad.wmflabs  was mean to receive Jenkins logs in ElasticSearch. We are experimenting with relforge1001.eqiad.wmnet now - [[phab:T78705|T78705]]
* 12:37 hashar: Removing HHVM from permanent Trusty slaves
* 10:44 elukey: running eventlogging_cleaner.py (https://gerrit.wikimedia.org/r/#/c/356383/) on eventlogging to test the cleaning of old events
* 09:24 hashar: Deleting deployment-phab02 instance. Has been shut off since April 23rd -  [[phab:T167090|T167090]]
* 07:51 hashar_: Fixed puppet on deployment-aqs instances
== 2017-06-05 ==
* 15:38 elukey: manually hacking deployment-jobrunner02.deployment-prep.eqiad.wmflabs to test a new config
== 2017-06-02 ==
* 19:51 hashar: integration: granted ebernhardson sudo
* 12:12 hashar: jenkins: rebuild logstash plugin from HEAD of master for jenkins 2 back compat. logstash-1.2.0-4-gbcbc19e - [[phab:T78705|T78705]]
== 2017-06-01 ==
* 20:14 bearND: Update mobileapps to {{Gerrit|c4dc72d}}
* 20:12 mdholloway: killed the running emulator processes on integration-slave-jessie-android to get it booting again following yesterday's gerrit outage
* 13:39 hashar: Gerrit: change integration.git project to "Rebase if Necessary" with "Allow content merges" - [[phab:T131008|T131008]]
* 13:10 hashar: Gerrit allow content merge for integration/config ( https://gerrit.wikimedia.org/r/#/admin/projects/integration/config )  - [[phab:T131008|T131008]]
* 08:03 hashar: Purged all mysql bin files from deployment-db03 ( rm -fR /srv/sqldata/T166060 ) - [[phab:T166060|T166060]]
== 2017-05-31 ==
* 20:21 hashar: Jenkins: upgrading git-client-plugin 2.4.5..2.4.6 [[phab:T166557|T166557]]
* 07:50 hashar: deployment-db04:  mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
* 07:49 hashar: deployment-db03:  mysql> set global expire_logs_days = 7  -  to expire bin logs faster (instead of 30 days)  - [[phab:T166060|T166060]]
== 2017-05-30 ==
* 22:08 hasharAway: Changed integration/config.git submit type from "Fast forward only" to "Rebase if Necessary"  [[phab:T131008|T131008]]
== 2017-05-29 ==
* 14:44 elukey: reverted previous config on redis01
* 14:36 elukey: set redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 300 on redis01 as test (rollback: redis-cli -a "$(sudo grep -Po '(?<=masterauth ).*' /etc/redis/tcp_6379.conf)" -p 6381 config set tcp-keepalive 0)
* 10:22 hashar: force refreshed Nodepool Trusty images. Was stuck somehow
* 10:06 hashar: deployment-tin rm -fR /usr/src/hhvm  [[phab:T166492|T166492]]
* 09:51 hashar: deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json  [[phab:T166492|T166492]]
== 2017-05-26 ==
* 09:20 elukey: installing hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02
* 07:20 elukey: hacking on jobrunner02 in deployment-prep
* 01:28 bearND: Update mobileapps to {{Gerrit|db6493c}}
== 2017-05-25 ==
* 19:46 hashar: deployment-tin manually cleaning disk space
* 16:44 elukey: restored hhvm on jobrunner02
* 16:03 bearND: Update mobileapps to {{Gerrit|946fe1f}}
* 10:33 elukey: manual install of hhvm_3.18.2+dfsg-1+wmf4+exp1_amd64.deb on jobrunner02 to test a fix for the Redis.php lib
* 02:46 RainbowSprinkles: running `mwscript extensions/Flow/maintenance/FlowUpdateUserWiki.php --wiki=enwiki` in a screen on deployment-tin, probably going to take all night
== 2017-05-24 ==
* 16:04 hashar: rebooting integration-slave-trusty-1003 to catch up with kernel upgrade
* 12:22 hashar: deployment-prep: finished rebase of puppet.git
* 10:19 hashar: deployment-prep rebased  puppet repo with: git rebase -X theirs
* 10:10 hashar: deployment-prep : resetting puppet master to last known snapshot snapshot-20170523T0010  .  All cherry picks got deleted
* 10:09 hashar: deployment-etcd-01: fixed puppet run
* 08:38 moritzm: updated puppet on deployment-puppetmaster02 to 3.8.5-2~bpo8+2
== 2017-05-23 ==
* 16:55 RainbowSprinkles: there was no data
* 16:55 RainbowSprinkles: dropped flow_ext_ref from commonswiki on beta. schema migration is busted, going to let it recreate table
* 08:20 hashar: Updating Nodepool snapshot-ci-trusty
* 08:19 hashar: Regenerated Nodepool base image for Trusty. Got rid of hhvm from it
== 2017-05-22 ==
* 12:11 greg-g: ran git prune and rm'd the gc.log file
* 11:40 greg-g: gjg@deployment-tin:/srv/mediawiki/.git/gc.log has warning: There are too many unreachable loose objects; run 'git prune' to remove them.
== 2017-05-21 ==
* 12:05 Reedy: deployment-tin is back online
* 10:41 Reedy: disabled jerkins on deployment-tin again
* 09:10 greg-g: beta-update-database-eqiad has been hitting the timelimit since May 19th
* 09:02 Reedy: brought deployment-tin back online a while ago
== 2017-05-20 ==
* 09:10 greg-g: executers are running again
* 09:02 greg-g: All executers in Jenkins are "offline" including the permament ones
== 2017-05-19 ==
* 19:05 mutante: fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab.  remove again from instance level for phab-01
* 18:40 mutante: deployment-phab01 still has puppet error "Could not find class role::phabricator::main" and that should simply be removed from it, but i can NOT find it in Horizon, i checked instance config, project config, the "Other" section, the "All classes" tab. Because it's gone. But how do i fix the instance config then?
* 18:39 mutante: applying role::phabricator_server on instance deployment-phab01 (it had error, could not find role::phabricator::main and the name changed in role/profile conversion)
== 2017-05-15 ==
* 10:46 addshore: enabled beta-code-update-eqiad for some testing
* 10:38 addshore: temporarily disabled beta-code-update-eqiad for some testing
== 2017-05-13 ==
* 20:31 bd808: Deleted stuck mediawiki-core-doxygen-publish job. Jenkins had it marked for a particular nodepool instance that was offline.
== 2017-05-12 ==
* 13:12 hashar: Trying to refresh Nodepool Jessie image. Should get HHVM pinned to 'experimental' component => 3.12.x
== 2017-05-11 ==
* 20:43 hashar: nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit.  Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. [[phab:T165074|T165074]]
* 12:57 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/353282/
== 2017-05-10 ==
* 20:28 bearND: Update mobileapps to {{Gerrit|75b135e}}
* 18:32 mutante: deployment-tin/mira: the change of the role class name was because of https://gerrit.wikimedia.org/r/#/c/344728/ which moved deployment::server to profile/role structure. both instances configured accordingly now. the remaining issue with "id_rsa.bromine" should be all unrelated
* 18:28 mutante: deployment-mira: configure puppet config in horizon, remove "role::deployment::server", use correct new name "role::deployment_server" (moved to profile). (a bit tricky because then in Horizon it seems to disappear from the "others" section, but if you click the "all" tab you get to see the class names
* 18:12 mutante: deployment-tin: puppet run now ok, except ":Upload/File[/var/lib/releases/.ssh/id_rsa.bromine.eqiad.wmnet]: Could not evaluate:" this should be an unrelated issue
* 18:05 mutante: deployment-tin: configure to use role::deployment_server (instead of deployment::server), for some reason now Horizon shows _nothing_ under "other classes" where this was before
* 17:58 mutante: deployment-tin: deleting puppet lock file (claimed it was running but also didnt run since > 900 min), looking at fixing deployment::server role name change
* 15:26 elukey: refresh cherry pick gerrit/352582 on puppet master (rebase -i to remove, then cherry pick)
* 14:34 elukey: cherry pick gerrit/352582 to puppet master
* 12:35 hashar: deployment-prep: git -C /srv/mediawiki-staging/php-master/extensions rm --cached SemanticFormsInputs
* 08:04 hashar: merging 'composer test' into mwext-testextension-* jobs https://gerrit.wikimedia.org/r/#/c/352160/ - [[phab:T161895|T161895]]
== 2017-05-09 ==
* 12:44 hashar: deployment-ircd upgrading puppet 3.7.2 => 3.8.5
* 12:19 hashar: Unbroke puppet on deployment-irc and deployment-urldownloader . Both choked on a ruby one-liner, fixed via https://gerrit.wikimedia.org/r/#/c/336840/
== 2017-05-08 ==
* 21:42 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/351131/
* 00:57 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/351130
== 2017-05-06 ==
* 01:16 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/352154
== 2017-05-05 ==
* 20:11 hasharDisappear: Mass pushing addition of jakub-onderka/php-console-highlighter  to all mediawiki extensions having php-parallel-lint ( example: https://gerrit.wikimedia.org/r/#/c/352215/ )
* 09:20 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350817 on deployment-puppetmaster02
* 09:17 addshore: temporarily enabled beta-code-update-eqiad
* 09:12 addshore: temporarily disabled beta-code-update-eqiad
* 08:30 godog: cherry-pick https://gerrit.wikimedia.org/r/#/c/350220/ on deployment-puppetmaster02
== 2017-05-04 ==
* 10:47 hashar: puppet ca destroy deployment-zookeeper01.eqiad.wmflabs
* 10:46 hashar: puppet ca destroy deployment-ores-redis-02.deployment-prep.eqiad.wmflabs  (no such instance)
* 10:46 hashar: puppet ca sign deployment-ores-redis-02.deployment-prep.eqiad.wmflabs
* 10:39 hashar: Removing puppetmaster: puppetmaster.thumbor.eqiad.wmflabs  from deployment-imagescaler01 - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'  - [[phab:T153319|T153319]]
* 10:37 hashar: deployment-prep: force recompilation of puppet.conf : salt -v '*' cmd.run 'echo >> /etc/puppet/puppet.conf.d/10-main.conf'
* 10:31 hashar: deployment-phab01 / deployment-imagescaler01 rm /etc/puppet/puppet.conf.d/10-self.conf  - [[phab:T153319|T153319]]
* 10:29 hashar: Unbroke puppet on deployment-imagescaler01 and removing role::puppetmaster::self - [[phab:T153319|T153319]]
* 10:16 hashar: Unbroke puppet on deployment-phab01 - [[phab:T153319|T153319]]
* 07:30 hashar: deployment-prep: adding TTO (This, that and the other) as a project member to grant shell access - [[phab:T163887|T163887]]
== 2017-05-03 ==
* 17:39 mdholloway: (this concerns integration-slave-jessie-android)
* 17:37 mdholloway: enabled automatic Android component installation for the Android Gradle plugin, rebuilt the SDK, and deleted the old one
* 15:54 hashar: Granted sudo right for Niedzielski accounts on Android CI slave.  Already has it with the other labs account Sniedzielski - [[phab:T164388|T164388]]
* 15:38 hashar: Granted mdholloway (mobile team) full sudo access on integration labs project so he can reach integration-slave-jessie-android -  [[phab:T164388|T164388]]
== 2017-05-02 ==
* 21:14 hashar: Manually cancelled a few mediawiki-core-jsduck-publish and mediawiki-core-doxygen-publish job in Jenkins build queue. They seems to deadlock Jenkins somehow :(
* 19:59 hashar: Regenerate jobs selenium-GettingStarted from JJB - [[phab:T164296|T164296]]
* 19:51 hashar: Jenkins: rolling back Performance plugin from 2.2 to 2.0 due to an exception / failure to find a junit xml file.  [[phab:T164296|T164296]]
* 19:02 hashar: Added multichill ( https://github.com/multichill ) to the Wikimedia Github organization
* 10:21 godog: bounce varnish and varnish-frontend on deployment-cache-upload04
* 10:16 godog: upgrade scap on deployment-tin to overcome AttributeError: Lock instance has no attribute 'get_lock_excuse'
* 09:41 godog: flip deployment-cache-upload04 to deployment-ms-fe02 - [[phab:T162247|T162247]]
* 08:17 hashar: Reconfigured all Jenkins jobs via jjb
== 2017-05-01 ==
* 20:39 hashar: Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ )
* 20:26 hashar: nodepool: deleting alien instance ci-trusty-wikimedia-631443 4e66ad7e-b9d3-4af1-b559-{{Gerrit|3f54968d376e}}
* 02:49 TimStarling: on puppetmaster02 manually updating /etc/conftool/data-local
* 02:37 TimStarling: on puppetmaster02 updated cherry pick for https://gerrit.wikimedia.org/r/#/c/347360
== 2017-04-27 ==
* 18:18 urandom: deployment-prep: restarting cassandra-metrics-collector on deployment-restbase0[1-2]
* 07:26 Amir1: cherry-picking 348184/4 ([[phab:T161563|T161563]])
== 2017-04-26 ==
* 23:36 urandom: removing r/350485 from deployment-prep
* 21:53 urandom: cherry-picking r/350485 to deployment-prep
* 20:20 bearND: Update mobileapps to {{Gerrit|14bd4a5}}
* 15:24 godog: add new deployment-ms-be0[34] backends to swift in deployment-prep - [[phab:T162247|T162247]]
== 2017-04-25 ==
* 21:57 halfak: deployed ores {{Gerrit|cc12103}}
* 06:46 Amir1: uncherry-pick {{Gerrit|f6ce64e99a}} and {{Gerrit|225b8d4e82}} ([[phab:T161563|T161563]])
== 2017-04-22 ==
* 20:17 hashar: Added FlorianSW to Github organization "wikimedia" (no team though)
== 2017-04-21 ==
* 12:25 hashar: [[phab:T104048|T104048]]  zuul enqueue --trigger gerrit --pipeline postmerge --project AhoCorasick --change 345433,1
* 09:32 hashar: Zuul: deploying  "Decouple repos from mediawiki gate queue"  {{Gerrit|7a79f752363a}}  / [[phab:T107529|T107529]]
* 09:30 elukey: hack reverted on tin and scap pull performed on jobrunner02
== 2017-04-20 ==
* 17:09 elukey: reverted hack on deployment-tin (apparently no effects on the jobrunner)
* 16:41 elukey: temporary disable puppet on deployment-tin to remove jobrunner02 from scap dsh; manually enable persistent connection between it and rdb redis hosts
== 2017-04-19 ==
* 16:34 hashar: deleted nodepool alien ci-jessie-wikimedia-613597
* 09:20 hashar: apt-get upgrade deployment-tin  deployment-mira
* 09:16 hashar: apt-get upgrade on deployment-mx deployment-redis01 deployment-redis02 deployment-cache-text04
* 02:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/348896
== 2017-04-18 ==
* 14:29 hashar: unbreaking integration puppetmaster. Broke it when upgrading the puppet package :(
* 14:09 hashar: integration: upgrade puppet on Jessie permanent slaves  3.7.2 -> 3.8.5  (and add ruby-rgen).  Done via: salt -v '*' pkg.upgrade
* 13:17 elukey: upgrade deployment-jobrunner02 to hhvm 3.18.2+wmf2 - [[phab:T162354|T162354]]
* 10:07 godog: upgrade swift to 2.2.0 on deployment-ms*
== 2017-04-14 ==
* 12:29 hashar: Delete integration-c1 instance (32GB RAM) on labvirt1004. It was used as a workaround for [[phab:T161006|T161006]]
* 08:17 hashar: beta: cherry picking again 348184/4 'service: use gzip for logging in uwsgi' for [[phab:T161563|T161563]]
* 08:03 hashar: beta: resetting puppetmaster to last good tag snapshot-20170414T0030 A cherry pick for [[phab:T161563|T161563]] end up dropping three patches which broke other parts of the infrastructure
* 07:52 hashar_: Puppet failing on deployment-tin and deployment-mira . Some patches have been dropped from the puppet master :-((
* 00:59 Amir1: three cherry-picks failed to merge, skipped them {{Gerrit|93dad5bec8e937ef93bdd63046b0bbbf14ad9722}} {{Gerrit|92c7d0b002a02ff46a00e79d6d89fe83d5f65c17}} {{Gerrit|21d60a478ffb21160049679eea235bdb1010a489}}
* 00:45 Amir1: cherry-picking 348184/1 ([[phab:T161563|T161563]])
== 2017-04-13 ==
* 15:37 hashar: deployment-mediawiki04 clearing /var/cache/hhvm/fcgi.hhbc.sq3
* 15:15 hashar: Deployed  mediawiki-core-qunit-selenium-jessie job (runs qunit + selenium with webdriverio)  https://gerrit.wikimedia.org/r/#/c/347587/ - [[phab:T139740|T139740]]
== 2017-04-12 ==
* 15:14 hashar: rm -fR  /mnt/home/jenkins-deploy/.android/build-cache/*  # [[phab:T162635|T162635]]
* 14:56 hashar: integration-slave-jessie-1001 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 14:54 hashar: integration-slave-jessie-1002 : mv /mnt/home/jenkins-deploy/.android-sdk /mnt/home/jenkins-deploy/.android-sdk.[[phab:T162635|T162635]].back  for [[phab:T162635|T162635]]
* 10:37 hashar: Jenkins email-ext plugin got upgraded.  Some groovy templating might be prevented and would have to be reviewed/approved via https://integration.wikimedia.org/ci/scriptApproval/
* 08:52 hashar: Cancelled bunch of mediawiki-core-doxygen-publish jobs that were keeping the queue busy/deadlocked builds.  Should be moved to poll scm instead ( [[phab:T115755|T115755]] )
== 2017-04-11 ==
* 15:59 hashar: integration-config-tox-jessie job is broken due to the JJB upgrade
* 15:40 hashar: Upgraded JJB to latest master {{Gerrit|4f77324f}}  with a couple cherrypicks on top of that. 022738f8...edebce7f  [[phab:T162674|T162674]]
* 15:36 hashar: Updating selenium-* jobs configuration for the performance plugin due to JJB upgrade [[phab:T162674|T162674]]
* 15:24 hashar: Adding parameter ZUUL_VOTING to all Jenkins jobs due to JJB upgrade [[phab:T162674|T162674]]
* 15:13 hashar: Forced updated jenkins-job-builder 86478421...022738f8 - [[phab:T162674|T162674]]
* 13:44 hashar: Forced updated jenkins-job-builder 1639a86e...86478421 - [[phab:T162674|T162674]]
* 13:44 hashar: Updating all Jenkins jobs using the git plugin due to JJB change  {{Gerrit|cdfeb7bf66b0eacfed3eaf2a77813d65ab0e29f2}} -  [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e  - [[phab:T162674|T162674]]
* 12:35 hashar: Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e
* 10:41 hashar: Enable webdriver.io  browser tests for MediaWiki core - https://gerrit.wikimedia.org/r/#/c/324719/ - [[phab:T139740|T139740]]
* 09:50 hashar: Regenerating MediaWiki doxygen documentations for all 1.23.x releases.
* 08:55 hashar: Retriggering MediaWiki doxygen publishing job for 1.26.0 - [[phab:T162506|T162506]] : zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.26.0 --newrev {{Gerrit|981ec62244e4806a16bda804dda1d14cb5d7f193}}
== 2017-04-10 ==
* 21:17 hashar: marked a nodepool node online manually. The instance was up but Jenkins failed to reach it due to some SEVERE: I/O error in channel
* 20:52 hashar: integration-slave-jessie-1001 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete # [[phab:T162635|T162635]]
* 20:49 hashar: integration-slave-jessie-1002 : cleaning up /tmp: sudo find /tmp -path '/tmp/android-tmp-robo*' -delete  # [[phab:T162635|T162635]]
* 20:08 bearND: Update mobileapps to {{Gerrit|1695900}}
== 2017-04-06 ==
* 16:36 halfak: staging ores:554ea12
* 12:23 hashar: Image snapshot-ci-trusty-1491480759 in wmflabs-eqiad is ready
* 12:13 hashar: Updating Nodepool Trusty image to let Linux overcommit memory ( https://gerrit.wikimedia.org/r/#/c/346634/ )
== 2017-04-05 ==
* 13:34 ema: testing possible fix for [[phab:T162035|T162035]] on deployment-ms-fe01
== 2017-04-04 ==
* 21:29 hashar: contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 21:26 hashar: contint2001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy  due to [[phab:T157818|T157818]]
* 20:58 hashar: integration: purging precise cow images from integration-slave-jessie-1001 and integration-slave-jessie-1002  ( https://gerrit.wikimedia.org/r/#/c/345836/ )
* 20:58 hashar: rebased integration puppet master
* 20:02 legoktm: deploying https://gerrit.wikimedia.org/r/346348
== 2017-04-03 ==
* 20:43 bearND: Update mobileapps to {{Gerrit|fdd4e31}}
* 20:39 hashar: Nodepool: holding instance ci-trusty-wikimedia-597386  in an attempt debug Wikibase/Scribunto memory usage exploding [[phab:T125050|T125050]]
* 20:37 hashar: jenkins: disabled/reenabled gearman plugin to unlock the beta cluster related jobs
* 09:17 hashar: deployment-jobrunner02 : cherry picked a monkey patch for Redis::close() to prevent it from sending QUIT command ( https://gerrit.wikimedia.org/r/#/c/346117/ ) - [[phab:T125735|T125735]]
== 2017-04-01 ==
* 09:48 Sagan: puppet on deployment-tin looks like it is not running properly
== 2017-03-29 ==
* 23:51 Krinkle: Free up space on integration-slave-jessie-1001 by removing old /srv/jenkins-workspace and /srv/pbuilder dirs
* 19:57 thcipriani: added --force flag for scap in beta-scap-eqiad temporarily
* 18:41 ebernhardson: upgrading elasticsearch and kibana to 5.1.2 on deployment-logstash2 to test puppet+integration prior to prod deployment
* 15:18 hashar: Delete a 32GB instance integration-ci  - [[phab:T161006|T161006]]
== 2017-03-28 ==
* 19:53 hashar: Populating package manager cache of oojs-ui-npm-run-jenkins-node-6-jessie  by manually triggering a build with ZUUL_PIPELINE=postmerge  [[phab:T155483|T155483]]
* 19:34 hashar: Migrate oojs/ui to just run 'npm jenkins'  https://gerrit.wikimedia.org/r/345203 /  [[phab:T155483|T155483]]
* 16:05 halfak: deployed ores:18beebf ([[phab:T160638|T160638]])
* 13:22 gehel: restarting elasticsearch on deployment-elastic05 to reload log4j configuration
* 10:28 hashar: Jenkins: installing Android Lint plugin 2.4 - [[phab:T161305|T161305]]
* 07:42 hashar: nodepool cleared a couple alien instances
== 2017-03-27 ==
* 17:02 ebernhardson: cherry pick https://gerrit.wikimedia.org/r/344964 to puppetmaster to test upgrade to logstash 5.x
* 11:10 hashar: Image snapshot-ci-jessie-1490612363 in wmflabs-eqiad is ready
* 10:59 hashar: Updating Nodepool Jessie image to include PhantomJS (take two) - [[phab:T137112|T137112]]
* 10:58 hashar: Image snapshot-ci-jessie-1490611594 in wmflabs-eqiad is ready
* 10:47 hashar: Updating Nodepool Jessie image to include PhantomJS - [[phab:T137112|T137112]]
* 10:20 hashar: Restarting Jenkins to drop the Throttle Concurrent Builds plugin - [[phab:T158596|T158596]]
== 2017-03-25 ==
* 10:46 Amir1: deleting deployment-ores-redis ([[phab:T160762|T160762]])
* 10:39 Amir1: changing ores redis address to deployment-ores-redis-01 ([[phab:T160762|T160762]])
* 10:02 Amir1: deleted deployment-ores-redis-02
== 2017-03-24 ==
* 21:34 Amir1: launching deployment-ores-redis-02 ([[phab:T160762|T160762]])
== 2017-03-23 ==
* 16:07 mobrovac: restbase deploying {{Gerrit|752ca4b7}}
* 15:52 hashar: Deleting integration-slave-trusty-1011 m1.large. One less perm slave to take care about
* 14:02 hashar: deployment-ms-be01 and deployment-ms-be02 : Lower Swift replicator on, upgrade package, reboot hosts.  [[phab:T160990|T160990]]
== 2017-03-22 ==
* 09:45 hashar: beta: purging all Linux kernel from Swift instances
* 08:48 hashar: deployment-ms-be01: swift-init reload all - [[phab:T160990|T160990]]
* 08:45 hashar: deployment-ms-be01: swift-init reload container - [[phab:T160990|T160990]]
* 08:43 hashar: deployment-ms-be01: swift-init reload object - [[phab:T160990|T160990]]
== 2017-03-21 ==
* 16:47 halfak: halfak@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 16:07 Amir1: ladsgroup@deployment-ores-redis:~$ redis-cli -h deployment-ores-redis.deployment-prep.eqiad.wmflabs -p 6380 -a areallysecretpassword flushall ([[phab:T160762|T160762]])
* 11:27 hashar: integration: purging old packages on permanent slaves, mostly old kernels:  apt-get autoremove --purge
* 09:06 hashar: CI deploying config hack "High priority test pipeline"  :  https://gerrit.wikimedia.org/r/343318  - [[phab:T160667|T160667]]
== 2017-03-20 ==
* 20:51 andrewbogott: migrating deployment-urldownloader to labvirt1013
* 20:45 andrewbogott: migrating deployment-pdf01 to labvirt1011
* 20:14 andrewbogott: migrating deployment-puppetmaster02 to a different labvirt
* 20:09 bearND: Update mobileapps to {{Gerrit|c0ab01d}}
* 08:51 hashar: Jenkins: depooling / deleting Precise instances.
== 2017-03-17 ==
* 14:08 hashar: salt -v '*precise*' cmd.run 'puppet agent --disable "Pending shutdown on March 20th - [[phab:T158652|T158652]]"'
== 2017-03-16 ==
* 21:48 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/343113/
* 15:03 hashar: deployment-prep setting role::logging::mediawiki::udp2log::rotate: 15  in project wide hiera configuration
== 2017-03-15 ==
* 20:29 bearND: Update mobileapps to {{Gerrit|bb8fcf2}}
* 19:02 niedzielski: Reloading Zuul to deploy {{Gerrit|f1c9073c6bda0bc83329c5178cb53613479ba200}}
* 15:55 Reedy: Removed hhvm statcache cherrypick from beta puppetmaster
* 11:09 elukey: Restore prod version of memcached on deployment-memc04 after experiment (I installed a new version a while ago)
* 10:22 elukey: created instances deployment-aqs0[23] to have better testing for the AQS beta environment
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=hewiktionary
* 09:10 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=dewiktionary
* 09:08 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary
* 08:56 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary // (ParameterTypeException, [[phab:T160503|T160503]])
* 08:50 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (3 sites added)
* 08:49 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https --load-from=https://deployment.wikimedia.beta.wmflabs.org/w/api.php
* 08:49 addshore: addshore@deployment-tin mwscript sql.php --wiki=enwiktionary "TRUNCATE sites; TRUNCATE site_identifiers;"
* 08:44 addshore: addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=dewiktionary --site-group=wiktionary // (0 sites added)
* 08:43 addshore: addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (1 site added)
== 2017-03-14 ==
* 19:22 thcipriani: removed alien nodepool instance via: openstack server delete ci-jessie-wikimedia-566503
* 10:15 hashar: Added Niedzielski to integration.
* 09:54 hashar: Jenkins: dropping Sniedzielski  more specific permissions. Account is already in wmf ldap group
== 2017-03-13 ==
* 13:19 hashar: Depooled Precise instances from Jenkins [[phab:T158652|T158652]]  leaving the instances up for now.
* 11:38 hashar: Deleting php53lint jobs. Replacing them with php55 equivalents
* 09:39 hashar: upgrading puppet on deployment-pdf01
* 09:30 hashar: Removing old kernel packages from deployment-pdf01 to free up disk space
* 08:55 hashar: Deleting deployment-copper  Fails puppet due to broken OpenStack metadata  http://169.254.169.254/openstack/2015-10-15/meta_data.json  (fails)  and  no more needed (per elukey )
== 2017-03-10 ==
* 12:29 kart_: Beta: [[phab:T159800|T159800]]: Update DB index for [[phab:T146450|T146450]]
== 2017-03-09 ==
* 16:20 gehel: upgrading elasticsearch on deployment-prep to v5.1.2
* 09:39 hashar: deployment-prep: rebasing puppet master. Got stall due to a submodule update apparently
== 2017-03-08 ==
* 22:45 Reedy: https://gerrit.wikimedia.org/r/#/c/341916/ cherry picked onto deployment-puppetmaster02
== 2017-03-07 ==
* 22:39 hashar: upgrading jenkins02.ci-staging to jenkins 2.x
* 15:26 hashar: ci-staging, enabling puppet master auto signing ( puppetmaster::autosigner: true )
* 08:25 hashar: Image snapshot-ci-jessie-1488874660 in wmflabs-eqiad is ready (Chromium 55->56 among others) - [[phab:T153038|T153038]]
* 08:16 hashar: Pushing new Jessie image: image-jessie-20170306T224719Z.qcow2
== 2017-03-06 ==
* 19:03 addshore: mwscript sql.php --wiki=aawiki "CREATE DATABASE cognate_wiktionary"
* 16:03 hashar: Jenkins upgrading  "Git client plugin" 1.19.6 to 2.3.0
== 2017-03-02 ==
* 20:47 hashar: deployment-prep: restarted apache/puppet master. Maybe that will fix ssh_known_hosts being emptied from time to time [[phab:T159332|T159332]]
* 19:32 thcipriani: snapshot-ci-jessie updated for nodepool
* 19:15 thcipriani: running: nodepool image-update wmflabs-eqiad snapshot-ci-jessie to manually update the ci-jessie snapshot for nodepool
* 18:26 godog: integration update composer on '*slave*'
* 11:52 hashar: gerrit: killed a stalled connection: {{Gerrit|dd511e52}} Feb-27 07:11  git-receive-pack '/mediawiki/services/zotero/translators'
* 09:53 hashar: Image snapshot-ci-jessie-1488447340 in wmflabs-eqiad is ready
* 09:29 hashar: Image snapshot-ci-trusty-1488446586 in wmflabs-eqiad is ready
* 09:18 hashar: upgrading composer on permanent slaves for [[phab:T125343|T125343]] : salt -v '*slave*' cmd.run 'cd /srv/deployment/integration/composer && git pull'
* 09:16 hashar: upgrade composer to 1.1.0 https://gerrit.wikimedia.org/r/#/c/339645/
* 08:40 elukey: upgrading apache2 on deployment-mediawiki* - latest debian DSA, introduces https://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions (risk of HTTP 400 responses regression, contact elukey or moritzm if you see any issue)
== 2017-03-01 ==
* 19:09 addshore: "mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki he wiktionary hewiktionary he.wiktionary.beta.wmflabs.org" [[phab:T158628|T158628]]
* 17:11 hashar: cleaned out Jenkins security matrix to drop users that are no more used/inexistent -- [[phab:T69027|T69027]]
* 14:13 hashar: deployment-prep : on deployment-tin removed empty dir /etc/ssh/userkeys/root.d  . Causes puppet noise
* 12:21 hashar: deployment-prep cleaning out git repos on deployment-tin
* 10:00 legoktm: deployed https://gerrit.wikimedia.org/r/340280 to slaves
* 04:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/340465
* 01:03 Reedy: beta-scap-eqiad giving Host key verification failed
== 2017-02-28 ==
* 19:43 thcipriani: deployment-puppetmaster02 puppetmaster running again, apache2 was refusing to start with: Invalid command 'SSLOpenSSLConfCmd' -- installed apache from wmf repo instead of debian fixed it
* 08:36 hashar: nodepool deleted alien instances 541585 541586 and 541587
== 2017-02-27 ==
* 21:36 bearND: Update mobileapps to {{Gerrit|c924126}}
== 2017-02-25 ==
* 03:50 MaxSem: deployment-prep Deleted January logs from deployment-fluorine02, was running out of space
== 2017-02-24 ==
* 13:56 hashar: Log refresh Nodepool instances to deploy slave script update to be able to merge mediawiki/composer.json into vendor/composer.json  6527f49..a7728a5  https://gerrit.wikimedia.org/r/#/c/339202/ [[phab:T158674|T158674]]
* 13:52 hashar: deployed slave script update to be able to merge mediawiki/composer.json into vendor/composer.json  6527f49..a7728a5  https://gerrit.wikimedia.org/r/#/c/339202/ [[phab:T158674|T158674]]
== 2017-02-23 ==
* 18:35 greg-g: 18:29 <  chasemp> !log labnodepool1001:~# service nodepool restart
* 09:27 hashar: Clearing skins from testextension jobs [[phab:T117710|T117710]]  salt -v '*slave*' cmd.run 'rm -fR /srv/jenkins-workspace/workspace/mwext-testextension*/src/skins/*'
== 2017-02-22 ==
* 20:58 hashar: Deleted jenkins job pplint-HEAD. Fully replaced by rake / puppet-syntax gem - [[phab:T154894|T154894]]
* 20:54 hashar: Deleted jenkins job erblint-HEAD. Fully replaced by rake / puppet-syntax gem - [[phab:T154894|T154894]]
== 2017-02-20 ==
* 14:53 hashar: integration: applying role::ci::slave::saucelabs to saucelabs-01
* 12:50 hashar: integration-slave-jessie-1001  downgraded cowbuilder to 0.73 from jessie to match integration-slave-jessie-1002
== 2017-02-17 ==
* 14:07 hashar: integration: deleting "repository" instance. No time to figure out how to ship Sonatype Nexus to it. [[phab:T147635|T147635]]
== 2017-02-16 ==
* 18:34 greg-g: chase restarted nodepool, the daemon crashed
* 18:32 greg-g: no active nodepool instances listed in Jenkin's view: https://integration.wikimedia.org/ci/ but zuul has plenty to do https://integration.wikimedia.org/zuul/
* 16:56 hashar: integration: provisioned browsertests-1001 with role::ci::slaves::browsertests . Added it to Jenkins with label  BrowserTests
* 16:33 halfak: deploying ores:e9bbda3
* 16:30 hashar: integration: created browsertests-1001  intended to run the daily browser tests later on
== 2017-02-15 ==
* 15:47 hashar: Zuul reducing gate-and-submit minimum amount of changes to process from the wrong 12 down to 2.  In case of repeating failures it would  end up running jobs for only two jobs which would prevent cancelling jobs for up to 11 changes!
== 2017-02-14 ==
* 14:38 hashar: Updating castor-save publish job to properly capture composer cache on Jessie ( it is in ~/.composer/cache for some reason)  [[phab:T156359|T156359]]
== 2017-02-13 ==
* 21:25 bearND: Update mobileapps to {{Gerrit|3af473f}}
* 20:15 hashar: Image snapshot-ci-jessie-1487016035 in wmflabs-eqiad is ready
* 20:01 hashar: Updating Nodepool Jessie snapshot to update the Parsoid zuul-cloner map ( https://gerrit.wikimedia.org/r/#/c/337430/ )
* 09:25 hashar: Changing Jenkins slave contint1001 working dir  from /srv/ssd/jenkins-slave to /srv/jenkins-slave ( https://gerrit.wikimedia.org/r/#/c/337286/ )
== 2017-02-10 ==
* 22:11 halfak: deployed ores:a15ec90
* 21:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/337079
* 16:18 thcipriani: deployment-puppetmaster02:/var/lib/git/operations/puppet removed untracked file "how", updated submodules
* 14:49 hashar: rebase beta puppet master. Fixed conflicts with https://gerrit.wikimedia.org/r/#/c/321096/ and https://gerrit.wikimedia.org/r/#/c/312523/
* 11:36 hashar: Pruning some old caches from castor.integration.eqiad.wmflabs  (eg node-4  jobs are gone)
== 2017-02-09 ==
* 22:22 greg-g: manually kicked off a bunch of selenium tests after tgr and Reedy fixed [[phab:T157636|T157636]]
* 18:36 Reedy: someone should kill the ee_prototypewiki db from beta
* 16:40 Amir1: deploying {{Gerrit|030c269}} ores to sca03
* 11:11 legoktm[NE]: deploying https://gerrit.wikimedia.org/r/336615 and https://gerrit.wikimedia.org/r/336779
== 2017-02-08 ==
* 22:26 mdholloway: mobileapps deployed {{Gerrit|0efa7b8}} in the beta cluster
* 14:14 hashar: integration-slave-jessie-1001 upgrading cowbuilder
* 09:20 hashar: deployment-fluorine02  upgraded packages, deleted old files from /srv/mw-log/archive
== 2017-02-07 ==
* 17:49 halfak: deploying ores {{Gerrit|7c80636}}
* 09:02 hashar: Hard rebooting integration-slave-jessie-1001 .  I messed up with the DHCP client :(
== 2017-02-06 ==
* 21:31 bearND: Update mobileapps to {{Gerrit|034a391}}
== 2017-02-04 ==
* 21:37 halfak: deploying ores {{Gerrit|7c80636}}
* 21:24 halfak: deploying ores {{Gerrit|691b340}}
== 2017-02-03 ==
* 11:09 hashar: beta: removed old kernels from deployment-redis02  to free up disk space
* 10:42 hashar: Image ci-jessie-wikimedia-1486115643 in wmflabs-eqiad is ready  [[phab:T156923|T156923]]
* 10:12 hashar: Image ci-jessie-wikimedia-1486115643 in wmflabs-eqiad is ready  [[phab:T156923|T156923]]
* 09:54 hashar: Regenerate Nodepool Jessie snapshot.  Would get a new HHVM version [[phab:T156923|T156923]]
== 2017-02-02 ==
* 21:56 hashar: integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-amd64.cow  it was not properly provisioned causing build to fail (eg lack of /etc/hosts) Running puppet to reprocvision it (poke [[phab:T156651|T156651]])
* 16:26 Amir1: deploying {{Gerrit|9fd75a1}} ores in beta
* 16:17 hashar: integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-i386.cow/  it was not properly provisioned causing build to fail (eg lack of /etc/hosts)  Running puppet to reprocvision it (poke [[phab:T156651|T156651]])
* 14:15 hashar: Nodepool: delete the image building of Jessie (image id 1322) to prevent a faulty HHVM version from being added. [[phab:T156923|T156923]]
* 00:52 tgr: added mhurd as member
== 2017-02-01 ==
* 21:43 bearND: Update mobileapps to {{Gerrit|e48a88c}}
* 18:51 thcipriani: nodepool delete-image 1320 per [[phab:T156923|T156923]]
* 14:53 gehel: deployment-elastic* fully migrated to Jessie and /srv as data partition - [[phab:T151326|T151326]]
* 14:52 gehel: killing test node deployment-elastic08 - [[phab:T151326|T151326]]
* 14:32 gehel: shutting down and reimaging deployment-elastic07 - [[phab:T151326|T151326]]
* 14:06 gehel: shutting down and reimaging deployment-elastic06 - [[phab:T151326|T151326]]
* 13:34 gehel: shutting down and reimaging deployment-elastic05 - [[phab:T151326|T151326]]
* 13:29 gehel: starting deployment-elastic* migration to jessie and moving data partition to /srv ([[phab:T151326|T151326]] / [[phab:T151328|T151328]])
* 13:18 moritzm: upgraded deployment-prep to hhvm 3.12.12
== 2017-01-31 ==
* 22:12 thcipriani: started mysql on all integration precise instances via salt -- was stopped for some reason
* 01:59 bd808: nodepool is full of instance stuck in "delete"
* 01:53 bd808: https://integration.wikimedia.org/zuul/ showing huge backlogs but https://integration.wikimedia.org/ci/ looks mostly idle
== 2017-01-26 ==
* 14:25 hashar: Created Github repo for Gerrit replication https://github.com/wikimedia/mediawiki-libs-phpstorm-stubs  [[phab:T153252|T153252]]
* 13:49 hashar: Gerrit creating  mediawiki/libs/phpstorm-stubs to fork https://github.com/JetBrains/phpstorm-stubs for [[phab:T153252|T153252]]
== 2017-01-24 ==
* 11:04 hashar: Deleting integration-publisher (Precise) replaced by integration-publishing (Jessie).  [[phab:T156064|T156064]] [[phab:T143349|T143349]]
== 2017-01-23 ==
* 23:41 bearND: Update mobileapps to {{Gerrit|66ef3c2}}
* 21:05 hashar: Created integration-publishing Jessie instance 10.68.23.254 with puppet class role::ci::publisher::labs .  Meant to replace Precise instance integration-publisher [[phab:T156064|T156064]]
* 12:45 hashar: Image ci-jessie-wikimedia-1485174573 in wmflabs-eqiad is ready  {{!}} should no more spawn varnish on boot
* 09:02 hashar: Archiving Gerrit project wikidata/gremlin marking it read-only [[phab:T155829|T155829]]
* 07:15 _joe_: cherry-picking the move of base to profile::base
== 2017-01-21 ==
* 21:20 hashar: integration: updating slave scripts for https://gerrit.wikimedia.org/r/#/c/333389/
* 21:08 bd808: Puppet failures on deployment-restbase0[12] seem to be some sort of hang of the Puppet process itself. Run prints "Finished catalog run in 2n.nn seconds" but Puppet doesn't terminate for about a minute longer. The only state change logged is cassandra-metrics-collector service start.
== 2017-01-20 ==
* 10:14 hashar: puppet fails on "integration" labs instances due to an attempt to unmount the non existing NFS /home.  Filled [[phab:T155820|T155820]]
* 09:18 hashar: beta: reset workspace of /srv/mediawiki-staging/php-master/extensions/reCaptcha  it had a .gitignore local hack for some reason
* 09:05 hashar: integration restarted mysql on trusty permanent slaves [[phab:T141450|T141450]] [[phab:T155815|T155815]]  salt -v '*trusty*' cmd.run 'service mysql start'
== 2017-01-19 ==
* 22:11 Krenair: added bunch of others to the same group per request. we should figure out how to make this process sane somehow
* 22:06 Krenair: added nuria to deploy-service group on deployment-tin
* 16:56 hashar: rebased puppet master on integration and deployment-prep Trivial conflict between https://gerrit.wikimedia.org/r/#/c/312523/  and a lint change
* 09:36 hashar: Nuking workspaces of all mwext-testextension-hhvm-composer* jobs. Lame attempt for [[phab:T155600|T155600]].  salt -v '*slave*' cmd.run 'rm -fR /srv/jenkins-workspace/workspace/mwext-testextension-hhvm-composer*'
== 2017-01-18 ==
* 10:49 hashar: Disconnected/connected Jenkins Gearman client.  The beta cluster builds had a deadlock.
* 10:39 hashar: Image ci-jessie-wikimedia-1484735445 in wmflabs-eqiad is ready (add python-conftool to hopefully have puppet rspec pass on https://gerrit.wikimedia.org/r/#/c/332475/ )
== 2017-01-17 ==
* 21:47 urandom: deployment-prep restarting Cassandra on deployment-restbase02
* 21:46 urandom: deployment-prep restarting Cassandra on deployment-restbase01
* 19:02 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/332534/
* 18:25 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/332521/
* 18:07 urandom: deployment-prep restarting Cassandra on deployment-restbase01
* 17:50 urandom: re-enabling puppet on deployment-restbase02
* 17:47 urandom: re-enabling puppet on deployment-restbase01
* 10:32 hashar: Refreshing all jobs in Jenkins 'jenkins-jobs --conf jenkins_jobs.ini update config/jjb'
== 2017-01-16 ==
* 09:33 hashar: integration  nuked the Zuul merger path for SelectTag mw extension ( on scandium /srv/ssd/zuul/git/mediawiki/extensions/SelectTag )  Failed to merge https://gerrit.wikimedia.org/r/#/c/331974/
== 2017-01-12 ==
* 00:33 legoktm: deploying https://gerrit.wikimedia.org/r/331796 and https://gerrit.wikimedia.org/r/331795
== 2017-01-11 ==
* 18:07 urandom: restarting restbase cassandra nodes
* 18:01 urandom: disabling puppet on restbase cassandra nodes to experiment with prometheus exporter
== 2017-01-10 ==
* 23:07 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/331099
== 2017-01-08 ==
* 05:20 Krenair: deployment-stream: live hacked /usr/lib/python2.7/dist-packages/socketio/handler.py a bit (added apostrophes) to try to make rcstream work
== 2017-01-07 ==
* 10:17 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki ([[phab:T139110|T139110]])
== 2017-01-06 ==
* 16:31 hashar: Nodepool Image ci-jessie-wikimedia-1483719758 in wmflabs-eqiad is ready
* 16:24 hashar: Nodepool Image ci-trusty-wikimedia-1483719370 in wmflabs-eqiad is ready
* 04:56 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/330843
== 2017-01-05 ==
* 17:20 hashar: Dropping puppet source from https://doc.wikimedia.org/puppetsource/ . contint1001: sudo rm -fR /srv/org/wikimedia/doc/puppetsource  ([[phab:T143233|T143233]])
== 2017-01-04 ==
* 21:29 mutante: deployment-cache-text-04 - running acme-setup command to debug  .. Creating CSR /etc/acme/csr/beta_wmflabs_org.pem
* 21:26 Krenair: trying to troubleshoot puppet by stopping nginx then letting puppet start it
* 21:05 mutante: deployment-cache-text04 stopping nginx service, running puppet to debug dependency issue
* 09:41 hashar: integration: pruning /srv/pbuilder/aptcache/  on Jessie perm slaves
== 2017-01-02 ==
* 11:22 hashar: Nodepool Image ci-jessie-wikimedia-1483355768 in wmflabs-eqiad is ready
* 11:17 hashar: Jessie images have the wrong python-pbr version ( [[phab:T153877|T153877]] ) causing zuul-cloner to fail. Refreshing image
* 10:02 hashar: Nodepool Image ci-jessie-wikimedia-1483350885 in wmflabs-eqiad is ready
* 09:57 hashar: Nodepool Image ci-trusty-wikimedia-1483350368 in wmflabs-eqiad is ready
== 2016-12-27 ==
* 05:00 Amir1: deploying {{Gerrit|5230e7d}} in ores beta node ([[phab:T154168|T154168]])
== 2016-12-26 ==
* 12:09 hashar: beta: restarted varnish.service and varnish-frontend.service on deployment-cache-text04
== 2016-12-24 ==
* 09:02 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/329038
== 2016-12-23 ==
* 12:18 legoktm: deploying https://gerrit.wikimedia.org/r/328886
== 2016-12-22 ==
* 22:11 thcipriani: disable production l10nupdate for deployment freeze
== 2016-12-21 ==
* 05:57 Krinkle: Jenkins "Collapsing Console Sections" for PHPUnit was broken since "-d zend.enable_gc=0" was added to phpunit.php invocation. Updated pattern in Jenkins system configuration.
== 2016-12-19 ==
* 21:21 andrewbogott: and also python-functools32_3.2.3.2-3~bpo8+1_all.deb
* 21:20 andrewbogott: upgrading to python-jsonschema_2.5.1-5~bpo8+1_all.deb on deployment-eventlogging03
* 20:51 andrewbogott: upgrading to python-requests_2.12.3-1_all.deb ./python-urllib3_1.19.1-1_all.deb on deployment-mediawiki04 and deployment-tin
* 09:35 legoktm: deploying https://gerrit.wikimedia.org/r/328145
* 08:00 legoktm: deploying https://gerrit.wikimedia.org/r/288819 https://gerrit.wikimedia.org/r/276065 https://gerrit.wikimedia.org/r/328136
* 02:25 legoktm: deploying https://gerrit.wikimedia.org/r/327692
== 2016-12-16 ==
* 22:34 legoktm: deploying https://gerrit.wikimedia.org/r/327202
* 14:33 hashar: Nodepool Image ci-jessie-wikimedia-1481897950 in wmflabs-eqiad is ready
* 14:25 hashar: Nodepool Image ci-trusty-wikimedia-1481897961 in wmflabs-eqiad is ready
* 14:19 hashar: Refreshing Nodepool images. The snapshots were broken due to mariadb-client failing to upgrade
* 13:45 hashar: integration / contintcloud : remove security rules of labs projects that allowed gallium (phased out) [[phab:T95757|T95757]]
* 13:44 hashar: integration / contintcloud : update security rules of labs projects to allow contint2001
* 13:15 hashar: integration: update sudo policy for debian-glue to keep the env variable SHELL_ON_FAILURE (for https://gerrit.wikimedia.org/r/#/c/327720/ )
* 10:15 hashar: integration: apt-get upgrade on all permanent slaves
* 10:13 hashar: integration-slave-docker-1000  changed docker::version from no more existent '1.12.3-0~jessie' to simply 'present'. Will have to manually upgrade it from now on.  [[phab:T153419|T153419]]
* 10:04 hashar: deployment-puppetmaster02  updated puppet repo. Was stall due to a bump of the mariadb submodule
== 2016-12-15 ==
* 21:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/324368
* 19:23 marxarelli: Manually rebasing and re-applying cherry picks for operations/puppet on integration-puppetmaster01.eqiad.wmflabs
* 16:08 hashar: deployment-phab02 : apt-get upgrade  [[phab:T147818|T147818]]
* 14:48 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki ([[phab:T139110|T139110]])
* 11:41 zeljkof: Reloading Zuul to deploy 327473
== 2016-12-14 ==
* 12:38 elukey: created deployment-copper on deployment-prep as temporary test
== 2016-12-13 ==
* 22:52 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/327119
* 21:15 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/327048
* 09:42 hashar: Updating MediaWiki Jenkins jobs to support injecting skin dependencies [[phab:T151593|T151593]]
* 02:17 legoktm: deploying https://gerrit.wikimedia.org/r/326880
* 02:10 legoktm: deploying https://gerrit.wikimedia.org/r/326877
== 2016-12-09 ==
* 04:01 legoktm: deploying https://gerrit.wikimedia.org/r/326070
* 03:45 legoktm: deploying https://gerrit.wikimedia.org/r/326069
== 2016-12-08 ==
* 23:35 legoktm: deploying https://gerrit.wikimedia.org/r/326048 https://gerrit.wikimedia.org/r/326050
* 22:32 legoktm: deploying https://gerrit.wikimedia.org/r/325930
* 21:14 legoktm: deploying https://gerrit.wikimedia.org/r/326032
* 21:08 legoktm: deploying https://gerrit.wikimedia.org/r/326020
* 20:27 legoktm: deploying https://gerrit.wikimedia.org/r/325974
* 20:19 legoktm: deploying https://gerrit.wikimedia.org/r/326016
* 20:11 legoktm: deploying https://gerrit.wikimedia.org/r/326015
* 19:51 legoktm: deploying https://gerrit.wikimedia.org/r/326009
* 19:44 legoktm: deploying https://gerrit.wikimedia.org/r/325912 https://gerrit.wikimedia.org/r/326006
* 15:33 hashar: Image ci-jessie-wikimedia-1481210905 in wmflabs-eqiad is ready : Notice: /Stage[main]/Main/Package[netcat-openbsd]/ensure: ensure changed 'purged' to 'present'
* 15:28 hashar: Updating Nodepool Jessie image to ship `netcat`  [[phab:T151469|T151469]] [[phab:T152684|T152684]]
* 10:31 hashar: Image ci-trusty-wikimedia-1481192772 in wmflabs-eqiad is ready
* 10:21 hashar: Refreshing Nodepool base image for Trusty. Was blocked on a mariadb upgrade, should also acquire network faster [[phab:T113342|T113342]]
* 09:45 legoktm: deploying https://gerrit.wikimedia.org/r/325903
* 08:48 hashar: Image ci-jessie-wikimedia-1481186016 in wmflabs-eqiad is ready  [[phab:T113342|T113342]]
* 05:31 legoktm: legoktm@integration-saltmaster:~$ sudo salt '*jessie*' cmd.run 'puppet agent -tv'
* 05:26 legoktm: cherry-picked https://gerrit.wikimedia.org/r/#/c/325877/ onto integration-puppetmaster01
* 03:26 legoktm: deploying https://gerrit.wikimedia.org/r/325873
== 2016-12-07 ==
* 15:04 hashar: Image ci-trusty-wikimedia-1481122712 in wmflabs-eqiad is ready  [[phab:T117418|T117418]]
* 02:29 matt_flaschen: foreachwikiindblist FlowFixInconsistentBoards complete
* 02:27 matt_flaschen: Started (foreachwikiindblist flow.dblist extensions/Flow/maintenance/FlowFixInconsistentBoards.php) 2>&1 {{!}} tee FlowFixInconsistentBoards_2016-12-06.txt on deployment-tin
== 2016-12-06 ==
* 21:20 hashar: Image ci-jessie-wikimedia-1481058839 in wmflabs-eqiad is ready [[phab:T113342|T113342]]
* 21:13 hashar: Refresh Nodepool Jessie snapshot which boot 3 times faster. Will help get nodes available faster [[phab:T113342|T113342]]
* 16:33 hashar: Nodepool imported a new Jessie image 'jessie-[[phab:T113342|T113342]]' with some network configuration hotfix. Will use for debugging. [[phab:T113342|T113342]]
* 09:08 Reedy: running foreachwiki update.php on beta
== 2016-12-05 ==
* 20:43 hashar: Image ci-jessie-wikimedia-1480969940 in wmflabs-eqiad is ready (include trendingedits::packages  which explicitly define the installation of librdkafka-dev' )
* 09:52 elukey: add https://gerrit.wikimedia.org/r/#/c/324642/ to the deployment-prep's puppet master to test nutcracker
* 09:39 hashar: beta-update-databases-eqiad fails due to CONTENT_MODEL_FLOW_BOARD not registered on the wiki. [[phab:T152379|T152379]]
* 08:44 hashar: Image ci-jessie-wikimedia-1480926961 in wmflabs-eqiad is ready  [[phab:T113342|T113342]]
* 08:35 hashar: Pushing new Jessie image to Nodepool that is supposedly boot 3x times faster [[phab:T113342|T113342]]
== 2016-12-04 ==
* 15:25 Krenair: Found a git-sync-upstream cron on deployment-mx for some reason... commented for now, but wtf was this doing on a MX server?
== 2016-12-03 ==
* 23:07 legoktm: deploying https://gerrit.wikimedia.org/r/325132
* 10:48 legoktm: deploying https://gerrit.wikimedia.org/r/325093 and https://gerrit.wikimedia.org/r/325094
== 2016-12-02 ==
* 14:40 hashar: added Tobias Gritschacher to Gerrit "integration" group so he can +2 patches on integration/* repositories \O/
== 2016-12-01 ==
* 18:20 elukey: removing https://gerrit.wikimedia.org/r/#/c/305536 from the puppet master via rebase -i (no-op for beta)
* 18:11 elukey: adding https://gerrit.wikimedia.org/r/#/c/305536/3 to the puppet master
* 14:16 hashar: Image ci-jessie-wikimedia-1480601060 in wmflabs-eqiad is ready  {{!}} [[phab:T152096|T152096]]
== 2016-11-30 ==
* 17:22 gehel: restart of logstash on deployment-logstash2 - upgrade to Java 8 - [[phab:T151325|T151325]]
* 17:11 gehel: rolling restart of deployment-elastic0* - upgrade to Java 8 - [[phab:T151325|T151325]]
* 11:22 hashar: Gerrit hide mediawiki/extensions/JsonData/JsonSchema Empty since 2013
* 11:20 hashar: Gerrit made mediawiki/extensions/GuidedTour/guiders read-only (per README.md, no more used)
* 11:18 hashar: Gerrit  mediawiki/extensions/CentralNotice/BannerProxy.git  Empty since 2014
== 2016-11-29 ==
* 15:23 hashar: Image ci-jessie-wikimedia-1480432368 in wmflabs-eqiad is ready
* 14:30 hashar: Image ci-trusty-wikimedia-1480429423 in wmflabs-eqiad is ready  [[phab:T151879|T151879]]
* 14:24 hashar: Refreshing Nodepool Trusty snapshot to get php5-xsl installed [[phab:T151879|T151879]]
== 2016-11-28 ==
* 09:48 hashar: Image ci-trusty-wikimedia-1480326016 in wmflabs-eqiad is ready
* 09:39 hashar: Regenerated Nodepool image for Trusty. It no more includes apache::mod::php5 which broke the build and is not needed on Trusty ( https://gerrit.wikimedia.org/r/323803  )
* 09:15 elukey: cherry-pick of https://gerrit.wikimedia.org/r/#/c/323517 to deployment-puppetmaster02 to test
== 2016-11-26 ==
* 16:15 Reedy: killed /srv/jenkins-workspace/workspace/mediawiki-core-*/src and /srv/jenkins-workspace/workspace/mwext-*/src from integration slaves to get rid of borked MW dirs
* 15:51 Reedy: deleted /srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src on integration-slave-trusty-1006 to force a reclone
* 14:14 Reedy: moved old /srv/mediawiki-staging/php-master to /tmp/php-master, recloned MW Core, copied in LocalSettings, skins, vendor and extensions. [[phab:T151676|T151676]]. scap sync-dir running
* 13:05 Reedy: marked deployment-tin as offline due to [[phab:T151670|T151670]]
== 2016-11-24 ==
* 20:49 hashar: make contint1001 Jenkins slave to only builds jobs with a label matching the node  https://integration.wikimedia.org/ci/computer/contint1001/configure  [[phab:T86659|T86659]]
* 15:46 elukey: removing https://gerrit.wikimedia.org/r/#/c/322268/ from the list of cherry picks on puppet master since it is not the right way to go
* 08:58 elukey: rebased puppet operations git repo on  deployment-puppetmaster to refresh https://gerrit.wikimedia.org/r/#/c/322268/
== 2016-11-23 ==
* 15:04 Krenair: fixed puppet on deployment-cache-text04 by manually enabling experimental apt repo, see [[phab:T150660|T150660]]
* 10:57 hashar: Terminating deployment-apertium01 again [[phab:T147210|T147210]]
== 2016-11-22 ==
* 19:31 hashar: beta: rebased puppet master
* 19:30 hashar: beta: dropping cherry pick for the PDF render by mobrovac ( https://gerrit.wikimedia.org/r/#/c/305256/ ). Got merged
* {{SAL entry|1=08:29 hashar: Deleting shut off instances: integration-puppetmaster , deployment-puppetmaster , deployment-pdf02 , deployment-conftool  - T150339}}
== 2016-11-21 ==
* {{SAL entry|1=12:46 hashar: beta: Cherry picked puppet fix for udp2log https://gerrit.wikimedia.org/r/#/c/322639/  T151169}}
== 2016-11-19 ==
* {{SAL entry|1=00:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/322370}}
== 2016-11-18 ==
* {{SAL entry|1=15:42 elukey: cherry picked https://gerrit.wikimedia.org/r/#/c/322268 on puppet master}}
== 2016-11-17 ==
* {{SAL entry|1=22:07 mutante: re-enabled puppet on contint1001 after live Apache fix}}
* {{SAL entry|1=11:34 hasharLunch: Deleted instance deployment-apertium01 . Was Trusty and lacked packages, replaced by a Jessie one ages ago. T147210}}
== 2016-11-16 ==
* {{SAL entry|1=20:53 elukey: restored apache2 config on deployment-mediawiki06}}
* {{SAL entry|1=20:28 elukey: temporary increasing verbosity of mod_rewrite on deployment-mediawiki06 as test}}
* {{SAL entry|1=20:02 Krenair: mysql master back up, root identity is now unix socket based rather than password}}
* {{SAL entry|1=19:57 Krenair: taking mysql master down to fix perms}}
* {{SAL entry|1=13:02 hashar: Restarted HHVM on deployment-mediawiki05 was not honoring requests T150849}}
* {{SAL entry|1=12:24 hashar: beta: created dewiktionary table on the Database slave. Restarted replication with START SLAVE;    T150834  T150764}}
* {{SAL entry|1=10:39 hashar: Removing revert b47ce21cec3a4340dd37c773210a514350f10297 from deployment-tin and reenabling jenkins job.  https://gerrit.wikimedia.org/r/321857 will get it fixed}}
* {{SAL entry|1=10:26 hashar: Reverting mediawiki/core b47ce21cec3a4340dd37c773210a514350f10297 on beta cluster T150833}}
* {{SAL entry|1=09:51 hashar: marking deployment-tin offline so I can live hack mediawiki code / scap for T150833 and T15034}}
* {{SAL entry|1=09:12 hashar: deployment-mediawiki04 stopping hhvm}}
* {{SAL entry|1=09:12 hashar: deployment-mediawiki04 stopping hhv}}
* {{SAL entry|1=08:59 hashar: beta database update broken with: MediaWiki 1.29.0-alpha Updater\n\nYour composer.lock file is up to date with current dependencies!}}
* {{SAL entry|1=07:52 Krenair: the new mysql root password for -db04 is at /tmp/newmysqlpass as well as in a new file in the puppetmaster's labs/private.git}}
* {{SAL entry|1=06:34 twentyafterfour: restarting hhvm on deployment-mediawiki04}}
* {{SAL entry|1=06:33 Amir1: ladsgroup@deployment-mediawiki05:~$ sudo service hhvm restart}}
* {{SAL entry|1=06:30 mutante: restarting hhvm on deployment-mediawiki06}}
== 2016-11-15 ==
* {{SAL entry|1=16:03 hasharAway: adding thcipriani to the labs "git" project maintained by paladox}}
== 2016-11-14 ==
* {{SAL entry|1=08:16 Amir1: cherry-picking 321096/3 in beta puppetmaster}}
== 2016-11-12 ==
* 14:02 Amir1: cherry-picked gerrit change 321096/2 in puppetmaster
== 2016-11-11 ==
* 23:48 bd808: Updated _template/logstash on deployment-logstash2 to include change from https://gerrit.wikimedia.org/r/#/c/320441/
* 23:44 bd808: Cherry-picked https://gerrit.wikimedia.org/r/#/c/320441/ for testing on deployment-logstash2
* 21:27 hashar: deployment-tin  deleted /var/lock/scap . Was left over after beta-scap-eqiad job got abruptly aborted
== 2016-11-10 ==
* 09:33 hashar: Image ci-jessie-wikimedia-1478770026 in wmflabs-eqiad is ready
* 09:26 hashar: Regenerate Nodepool base image for Jessie and refreshing snapshot image
== 2016-11-09 ==
* 20:27 Krenair: removed default SSH access from production host 208.80.154.135, the old gallium IP
* 16:34 Reedy: deployment-tin no longer offline, jenkins running jobs now
* 16:11 Reedy: marking deployment-tin.eqiad  as offline to test -labs -> beta config rename
== 2016-11-08 ==
* 10:23 hashar: refreshing all jenkins jobs to clear out potential live hack I made but can't remember on which jobs I did
== 2016-11-07 ==
* 14:01 gilles: Pointing deployment-imagescaler01.eqiad.wmflabs' puppet to puppetmaster.thumbor.eqiad.wmflabs
== 2016-11-04 ==
* 13:20 hashar: gerrit: created mediawiki/extensions/PageViewInfo.git  and renamed user group extension-WikimediaPageViewInfo to extension-PageViewInfo T148775
* 12:57 hashar: Image ci-jessie-wikimedia-1478263647 in wmflabs-eqiad is ready (bring in java for maven projects)
* 12:49 dcausse: deployment-prep reloading nginx on deployment-elastic0[5-7] to fix ssl cert issue
* 09:28 hashar: Delete integration-slave-jessie-1003 , only have a few jobs running on permanent Jessie slaves - T148183
* 09:26 hashar: Delete zuul-dev-jessie.integration.eqiad.wmflabs  was for testing Zuul on Jessie and it works just fine on contint1001 :]  T148183
* 09:25 hashar: Delete integration-slave-trusty-1012 one less permanent slave since some load has been moved to Nodepool  T148183
* 09:24 hashar: Delete integration-slave-trusty-1016 not pooled in Jenkins anymore T148183
== 2016-11-03 ==
* 15:05 Amir1: deploy 0caa589 in ores to deployment-sca03
* 14:52 Amir1: deploying ores 0caa589 in deployment-sca03
* 11:32 hashar: deployment-apertium01 manually cleared puppet.conf
* 11:29 hashar: deployment-apertium01 fails puppet du to wrong certificate bah
* 07:22 Krenair: fiddled with jenkins jobs in mediawiki-core-doxygen-publish to try to get stuff moving in the postmerge queue again
* 05:04 Krenair: beginning to move the rest of beta to the new puppetmaster
* 01:53 mutante: followed instructions at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Gearman_deadlock
* 01:53 mutante: disabling and re-enabling gearman, zuul is not working and could be gearman deadlock
== 2016-11-02 ==
* 22:06 hashar: hello stashbot
* 18:51 Krenair: armed keyholder on -tin and -mira
* 18:50 Krenair: started mysql on -db boxes to bring beta back online
* 10:54 hashar: Image ci-jessie-wikimedia-1478083637 in wmflabs-eqiad is ready
* 10:47 hashar: Force refresh Nodepool snapshot for Jessie  so it get doxygen included T119140
== 2016-11-01 ==
* 22:22 Krenair: started mysql on -db03 to hopefully pull us out of read-only mode
* 22:21 Krenair: started mysql on -db04
* 22:19 Krenair: stopped and started udp2log-mw on -fluorine02
* 22:10 hashar: Armed keyholder on deployment-tin . Instance had 20 minutes uptime and apparently keyholder does not self arm
* 22:00 Krenair: started moving nodes back to the new puppetmaster
* 02:55 Krenair: Managed to mess up the deployment-puppetmaster02 cert, had to move those nodes back
== 2016-10-31 ==
* 20:57 Krenair: moving some nodes to deployment-puppetmaster02
* 16:57 bd808: Added Niharika29 as project member
== 2016-10-27 ==
* 20:51 hashar: reboot integration-puppetmaster01
* 18:50 bd808: stashbot has replaced qa-morebots in this channel as the sole bot handling !log messages
* 18:46 bd808: Testing dual page wiki logging by stashbot. (check #3)
* 18:36 bd808: !log deployment-prep Testing dual page wiki logging by stashbot. (second attempt)
* 18:14 bd808: !log deployment-prep Testing dual page wiki logging by stashbot.
* 10:30 hashar: integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing for T149294: salt -v '*slave-trusty*' cmd.run 'deluser jenkins-deploy kvm'
* 10:29 hashar: integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing:  salt -v '*slave-trusty*' cmd.run 'groupdeluser jenkins-deploy kvm'
* 10:25 hashar: integration: purge Android packages from Trusty slaves for T149294 : salt -v '*slave-trusty*' cmd.run 'apt-get --yes remove --purge gcc-multilib lib32z1 lib32stdc++6 qemu'
== 2016-10-25 ==
* 19:21 hasharAway: Python PyPi mirror has some issue. Impacts all CI jobs relying on tox  https://status.python.org/
* 10:39 elukey: cherry picked https://gerrit.wikimedia.org/r/#/c/314519/ and https://gerrit.wikimedia.org/r/#/c/306943/ to deployment-puppetmaster
== 2016-10-24 ==
* 16:19 andrewbogott: upgrading deployment-puppetmaster to puppet 3.8.5 packages
* 09:14 hashar: rebasing integration puppet master
== 2016-10-21 ==
* 09:42 gehel: decommission of deployment-elastic08 - T147777
== 2016-10-20 ==
* 23:37 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/317083
* 20:53 legoktm: deploying https://gerrit.wikimedia.org/r/317022
== 2016-10-14 ==
* 21:13 matt_flaschen: Ran START SLAVE to restart replication after columns created directly on replica were deleted.
* 20:53 bd808: Dropped lu_local_id, lu_global_id from replica db which were added improperly
* 20:37 matt_flaschen: Applied CentralAuth's patch-lu_local_id.sql migration for T148111, to sql --write
* 20:09 bd808: Applied CentralAuth's patch-lu_local_id.sql migration for T148111
* 11:30 dcausse: deployment-prep running sudo update-ca-certificates --fresh on deployment-ton to fix curl error code 60 in cirrus maint script (T145609)
== 2016-10-13 ==
* 21:21 hashar: Deleted CI slaves integration-slave-jessie-1004 integration-slave-jessie-1005 integration-slave-trusty-1013 integration-slave-trusty-1014 integration-slave-trusty-1017 integration-slave-trusty-1018
* 20:12 hashar: Switching composer-hhvm / composer-php55 to Nodepool  https://gerrit.wikimedia.org/r/#/c/306727/  T143938
* 16:23 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 16:06 godog: add settings to duplicate traffic to thumbor in beta and restart swift-proxy
* 16:03 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315648/ on deployment-puppetmaster
* 15:35 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:38 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/5 on deployment-puppetmaster
* 14:34 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:32 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/4 on deployment-puppetmaster
* 14:32 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 14:27 gilles: Cherry-picking https://gerrit.wikimedia.org/r/#/c/315234/ on deployment-puppetmaster
* 14:22 gilles: Resetting to 61a9cd1f47c5aec8ded92f2486ce43309b9e3e03 on deployment-puppetmaster
* 13:42 gilles: Cherry picking https://gerrit.wikimedia.org/r/#/c/315248/ on deployment-puppetmaster
== 2016-10-12 ==
* 13:37 elukey: upgraded memcached on deployment-memc04 to 1.4.28-1.1+wmf1 as part of a perf experiment (T129963) - rollback: wipe https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep/host/deployment-memc04, apt-get remove memcached on deployment-memc04, puppet run
== 2016-10-11 ==
* 21:35 hasharAway: Force pushed Zuul patchqueue  5628f95...fc6a118 HEAD -> patch-queue/debian/precise-wikimedia
* 14:37 hashar: Mysql was down on Precise slaves. Apparently rebooted 17 days ago and I guess mysql does not spawn on boot. Restarted mysql on all Precise via: salt -v '*slave-precise*' cmd.run 'start mysql'
* 09:35 godog: reboot deployment-imagescaler01 to enable memory cgroup
* 08:29 hashar: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/313387/ Filter out refs/meta/config from all pipelines  T52389
== 2016-10-10 ==
* 15:45 dcausse: deployment-prep deployment-elastic0[5-8]: reduce the number of replicas to 1 max for all indices
== 2016-10-07 ==
* 20:10 hashar: Created repository.integration.eqiad.wmflabs to play/Test Sonatype Nexus
* 20:10 hashar: rebooting integration-puppetmaster01
* 07:55 hashar: Upgrading Nodepool image for Jessie
== 2016-10-06 ==
* 14:45 hashar: deployment-mira disarmed/rearmed keyholder in an attempt to clear a Shinken alarm
* 12:16 hashar: Jenkins slave deployment-tin.eqiad , removing label "deployment-tin.eqiad"  it has "BetaClusterBastion" and all jobs are bound to it already
== 2016-10-05 ==
* 19:33 andrewbogott: removing mediawiki::conftool from deployment-mediawiki04, deployment-mediawiki06, deployment-mediawiki05
== 2016-10-04 ==
* 19:43 andrewbogott: removed contint::slave_scripts and associated files from deployment-sca01 and  deployment-sca02
* 16:22 bd808: Restarted puppetmaster process on deployment-puppetmaster
* 16:20 bd808: deployment-puppetmaster: removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/305256/; conflicts with upstream changes
* 15:01 godog: shutdown deployment-poolcounter02, replaced by deployment-poolcounter04 - T123734
* 09:03 hashar: Regenerating configuration of all Jenkins job due to https://gerrit.wikimedia.org/r/#/c/313306/
* 01:14 twentyafterfour: New scap command line autocompletions are now installed on deployment-tin and deployment-mira refs T142880
== 2016-10-03 ==
* 22:40 thcipriani: manual rebase on deployment-puppetmaster:/var/lib/git/operations/puppet
* 22:05 thcipriani: reapplied beta::deployaccess to mediawiki servers
* 21:42 cscott: updated OCG to version 0bf27e3452dfdc770317f15793e93e6e89c7865a
* 21:36 cscott: starting OCG deploy
* 13:43 hashar: Added integration-slave-trusty-1014  back in the pool
* 13:41 hashar: Tip of the day: to reboot an instance and bypass molly-guard: /sbin/reboot
* 13:39 hashar: integration-slave-trusty-1014  upgrading packages, clean up and rebooting it