You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(Krinkle: Re-enable beta-scap-eqiad job)
imported>Stashbot
(TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye)
(857 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== 2018-10-09 ==
== 2022-06-28 ==
* 19:00 Krinkle: Re-enable beta-scap-eqiad job
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 18:19 Krinkle: Messing with scap in beta to test [[phab:T121597|T121597]] / D1114
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members {{!}} [[phab:T311370|T311370]]
* 15:32 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # [[phab:T311390|T311390]]


== 2018-10-08 ==
== 2022-06-27 ==
* 03:31 kart_: Update(d) cxserver to {{Gerrit|47a864b}}
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 03:02 legoktm: onlined integration-slave-jessie-1002
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
* 02:20 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *


== 2018-10-06 ==
== 2022-06-24 ==
* 13:59 Reedy: cleared some large folders out of /tmp on deployment-deploy01
* 20:52 taavi: added `denisse` as a member


== 2018-10-05 ==
== 2022-06-23 ==
* 19:47 marxarelli: bringing integration-slave-docker-1040 back online
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
* 19:05 marxarelli: taking integration-docker-slave-1040 offline for docker daemon restart
* 19:04 marxarelli: bringing integration-slave-docker-1038/1041/1043 back online
* 19:02 marxarelli: taking integration-docker-slave-1038/1041/1043 offline for docker daemon restart
* 19:01 marxarelli: bringing integration-slave-docker-1033/1037 back online
* 18:58 marxarelli: taking integration-docker-slave-1033/1037 offline for docker daemon restart
* 18:56 marxarelli: bringing integration-slave-docker-1034 back online
* 18:56 marxarelli: integration-puppetmaster01:/var/lib/git/operations/puppet is up-to-date again after manually updating submodules and subsequent automated git-sync-upstream
* 18:33 marxarelli: taking integration-slave-docker-1034 offline for docker daemon restart
* 11:18 Krenair: rm -rf /tmp/scap_l10n_* on deployment-deploy01
* 06:00 legoktm: deployed https://gerrit.wikimedia.org/r/464757
* 04:35 legoktm: deploying https://gerrit.wikimedia.org/r/464747
* 01:17 legoktm: triggering php71 quibble jobs manually via contint1001


== 2018-10-04 ==
== 2022-06-22 ==
* 22:23 thcipriani: updated jenkins jobs: https://phabricator.wikimedia.org/P7634
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in [[phab:T308278|T308278]]
* 22:19 thcipriani: updating 103 jenkins jobs that use ci-src-setup-simple
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in [[phab:T303147|T303147]]
* 21:45 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/464719/
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 21:13 marxarelli: bringing integration-slave-jessie-1003 back online
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 21:12 marxarelli: deleted /mnt/home/jenkins-deploy/{.m2,.gradle} on integration-slave-jessie-1003
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # [[phab:T307620|T307620]]
* 21:10 marxarelli: deleting cache directories in /mnt/home/jenkins-deploy on integration-slave-jessie-1003 to free up disk space
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # [[phab:T307620|T307620]]
* 20:31 marxarelli: terminating long-running (> 3 hours) CI docker containers ([[phab:T198517|T198517]])
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180


== 2018-10-03 ==
== 2022-06-21 ==
* 18:45 marxarelli: deploying {{Gerrit|I0fcd95427a6468213feb0be2ab7ab52f0d719f90}} for all 313 affected jobs
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 18:16 marxarelli: deploying {{Gerrit|I0fcd95427a6468213feb0be2ab7ab52f0d719f90}} for 12 mediawiki-quibble-vendor-* jobs
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
* 17:46 marxarelli: removing old workspaces on integration-slave-docker-1034 (`rm -rf /srv/jenkins-workspace/workspace/*`) and bringing back online
* 16:52 thcipriani: reloading zuul to deploy Change gate-and-submit-l10n to low precedence
* 15:46 marxarelli: bringing integration-slave-docker-1040/1043 nodes back online after killing long running docker job and freeing up /var/lib/docker space ([[phab:T206134|T206134]])
* 15:40 marxarelli: killing long running docker jobs on integration-slave-docker-1040/1043 which are filling up /var/lib/docker with log output
* 15:12 thcipriani: reloading zuul to deploy l10 pipeline
* 14:53 thcipriani: integration-slave-docker-1038 removed workspaces, brought back online integration-slave-docker-104{0,3} need more investigation since docker is full(?)
* 07:07 mdholloway: deployment-maps04 deployed [kartotherian/deploy@27062b4]: Specify WDQS endpoint in the service config ([[phab:T205607|T205607]])
* 06:22 _joe_: cherry-picking 455154 (php-fpm installation) on deployment-prep


== 2018-10-02 ==
== 2022-06-20 ==
* 21:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/464048/
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 18:22 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 15:50 ori: On deployment-cache-<nowiki>{</nowiki>text,upload<nowiki>}</nowiki>06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service ([[phab:T310957|T310957]])
* 18:15 legoktm: deploying https://gerrit.wikimedia.org/r/464000
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
* 16:45 thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*
* 16:20 thcipriani: investigating integration-slave-docker-1037
* 08:56 godog: bounce logstash
* 07:59 legoktm: deployed https://gerrit.wikimedia.org/r/463905
* 07:38 legoktm: building quibble-stretch-php71 docker image
* 05:32 legoktm: rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883
* 03:26 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886


== 2018-10-01 ==
== 2022-06-17 ==
* 20:09 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 17:13 marxarelli: bringing integration-slave-docker-1041 back online following source directory clean up ([[phab:T205902|T205902]])
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
* 16:52 marxarelli: removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041
* 07:10 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest
* 05:24 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to {{Gerrit|a1fa41b}}


== 2018-09-29 ==
== 2022-06-16 ==
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463615
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 01:22 Krinkle: Marking integration-slave-docker-1041 as offline (again). Why/How did it come back?
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 00:24 Krinkle: Marking integration-slave-docker-1041 as offline. Various odd build failures, including https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/6913/console
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]


== 2018-09-28 ==
== 2022-06-15 ==
* 15:31 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki  --prefix ([[phab:T201009|T201009]])
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 ([[phab:T310742|T310742]])
* 14:50 thcipriani: investigating integration-slave-docker-1041
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 07:35 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@bf09080]: Update mobileapps to {{Gerrit|7878ffc}}
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster


== 2018-09-27 ==
== 2022-06-14 ==
* 17:53 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@a0054ba]: Update mobileapps to {{Gerrit|0d6c2b7}}
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 07:38 mdholloway: deployment-maps04 updated tilerator and kartotherian node modules ([[phab:T195513|T195513]], [[phab:T200594|T200594]])
* 17:18 brennen: starting 1.39.0-wmf.16 ([[phab:T308069|T308069]]) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production


== 2018-09-26 ==
== 2022-06-13 ==
* 15:13 thcipriani: integration-slave-docker-1034:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online -- https://phabricator.wikimedia.org/P7592
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 15:05 thcipriani: integration-slave-docker-1033:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
* 14:47 thcipriani: investigating integration-slave-docker-103{3,4}
* 11:57 Amir1: [[gerrit:462927]] (ores) is going to beta
* 08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
* 08:14 hashar: Restarting CI Jenkins on contint1001


== 2018-09-25 ==
== 2022-06-12 ==
* 23:01 marxarelli: configured new jenkins node integration-slave-docker-1043 with 6 executors
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
* 23:01 marxarelli: replaced integration-slave-docker-1042 with new integration-slave-docker-1043 instance
* 22:39 marxarelli: launching new integration-slave-docker-1042 bigram instance
* 22:33 marxarelli: deleting remaining m1.medium instances used as m4executors ([[phab:T205362|T205362]])
* 22:15 marxarelli: taking remaining m1.medium m4executor jenkins nodes offline ([[phab:T205362|T205362]])
* 18:16 marxarelli: reconfiguring bigram jenkins nodes to use 6 executors. 7 were configured by mistake ([[phab:T205362|T205362]])
* 18:00 marxarelli: configuring new integration-slave-docker-1041 jenkins node with 7 executors ([[phab:T205362|T205362]])
* 17:42 marxarelli: configuring new jenkins node integration-slave-docker-1040 with 7 executors ([[phab:T205362|T205362]])
* 17:38 marxarelli: launching integration-slave-docker-1041 bigram instance ([[phab:T205362|T205362]])
* 17:30 marxarelli: the puppet parameter for docker_lvm_volume specified in horizon was not applied correctly on the first puppet run for some reason. tearing down integration-slave-docker-1039...
* 17:25 marxarelli: launching integration-slave-docker-1040 bigram instance ([[phab:T205362|T205362]])
* 17:24 marxarelli: deleting instances integration-slave-docker-1007/1008 ([[phab:T205362|T205362]])
* 17:13 marxarelli: launching new integration-slave-docker-1039 bigram instance
* 17:12 marxarelli: taking integration-slave-docker-1007/1008 offline for replacement ([[phab:T205362|T205362]])
* 17:09 marxarelli: deleting integration-slave-docker-1030/1031 instances ([[phab:T205362|T205362]])
* 17:05 marxarelli: taking integration-slave-docker-1030/1031 offline for replacement
* 16:47 marxarelli: increasing executors to 7 for jenkins nodes integration-slave-docker-1033/1034
* 16:46 marxarelli: new instance creation delayed due to quota
* 16:45 marxarelli: launching new integration-slave-docker-1039/1040 bigram instances
* 01:21 legoktm: deployed https://gerrit.wikimedia.org/r/450508
* 00:36 legoktm: deploying https://gerrit.wikimedia.org/r/462609
* 00:22 legoktm: deploying https://gerrit.wikimedia.org/r/453447


== 2018-09-24 ==
== 2022-06-10 ==
* 20:21 bearND: (beta): Update mobileapps to {{Gerrit|badb463}}
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for [[phab:T309932|T309932]]
* 10:55 hashar: gerrit: granting labs/tools/* project owners the ability to submit changes {{!}} https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 09:51 hashar: deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo  and created a new. Its size of 69632 bytes might cause slow writes?? {{!}} [[phab:T204762|T204762]]
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for [[phab:T295578|T295578]]
* 09:24 hashar: Live hacked scap code on deployment-deploy01 for [[phab:T204762|T204762]] and reverted hack changes
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for [[phab:T295578|T295578]] sanity check
* 08:32 hashar: deployment-deploy01 rm -fR /tmp/scap_l10n_*
* 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/462341
* 03:45 kart_: Update cxserver to {{Gerrit|d913793}}


== 2018-09-23 ==
== 2022-06-09 ==
* 14:03 Krenair: rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 01:05 andrewbogott: rebooted deployment-maps03; OOM and also [[phab:T205195|T205195]]
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration


== 2018-09-22 ==
== 2022-06-08 ==
* 20:51 Hauskatze: github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for [[phab:T183891|T183891]]
* 17:14 hashar: Reloaded Zuul for {{Gerrit|I39342265033e82ae13998f53defe6612dc6819b4}}
* 20:05 Hauskatze: github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer {{!}} [[phab:T183891|T183891]]; moving to the next one
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for [[phab:T237033|T237033]]
* 18:21 Krenair: went to do the same with deployment-maps03 and accidentally broke SSH access to the server
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134
* 18:21 Krenair: removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced [[phab:T153468|T153468]], this just explains why puppet hadn't purged stuff)
* 18:01 Krenair: rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
* 18:00 Krenair: rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)


== 2018-09-21 ==
== 2022-06-06 ==
* 17:26 marxarelli: adding jenkins node integration-slave-docker-1038 with 7 executors
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for [[phab:T302711|T302711]]
* 16:47 marxarelli: added new jenkins node integration-slave-docker-1037 with 7 executors
* 15:49 marxarelli: replacing integration-slave-docker-1036 with new bigram instance
* 15:48 marxarelli: taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations
* 15:17 marxarelli: integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations
* 15:15 marxarelli: taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend
* 15:13 marxarelli: launching integration-slave-docker-1037 bigram instance
* 13:03 Amir1: ores:7b987a7 is going beta
* 05:32 legoktm: deployed https://gerrit.wikimedia.org/r/461510


== 2018-09-20 ==
== 2022-06-02 ==
* 23:48 marxarelli: adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 23:23 marxarelli: launching integration-slave-docker-1035/1036 bigram instances
* 11:26 hashar: Restarting Jenkins on contint2001
* 23:20 marxarelli: taking integration-slave-docker-1004/1005 offline for replacement ([[phab:T202160|T202160]])
* 11:19 hashar: Restarting Jenkins on releases1002
* 16:52 Amir1: deploy ores:ee2d28b
* 11:21 hashar: Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. {{!}} [[phab:T203181|T203181]] and [[phab:T204919|T204919]]
* 11:17 hashar: deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes)
* 11:07 hashar: deployment-deploy01 is out of disk space (again)


== 2018-09-19 ==
== 2022-05-31 ==
* 21:18 Hauskatze: github: deleted https://github.com/wikimedia/mediawiki-services-ocg-collection {{!}} [[phab:T183891|T183891]]
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|a224e99}}
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 18:56 Amir1: ores:76fe25a goes to beta ([[phab:T204862|T204862]])
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster


== 2018-09-18 ==
== 2022-05-30 ==
* 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
* 11:47 jelto: apply gitlab-settings to gitlab1004 - [[phab:T307142|T307142]]
* 09:46 hashar: updating mwselenium-quibble-docker to Quibble 0.0.26
* 11:46 jelto: apply gitlab-settings to gitlab1003 - [[phab:T307142|T307142]]
* 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
* 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins  # untouched since 2016
* 07:31 hashar: cleaning disk on deployment-sca01
* 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors


== 2018-09-17 ==
== 2022-05-28 ==
* 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything [[phab:T309437|T309437]]
* 22:00 Reedy: update.php in screen is done
* 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
* 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
* 20:17 bearND: (beta): Update mobileapps to {{Gerrit|d56e4cf}}
* 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy {{Gerrit|I5df2b72fa411adf05027c8f6745284e7e1dedf7a}}
* 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
* 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest ([[phab:T109776|T109776]])
* 13:24 hashar: Building docker images for Quibble 0.0.25 {{!}} https://gerrit.wikimedia.org/r/460536
* 12:58 hashar: mwselenium-quibble-docker to Chromium 69 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ {{!}} [[phab:T204214|T204214]]
* 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
* 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
* 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
* 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775


== 2018-09-16 ==
== 2022-05-27 ==
* 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism {{!}} [[phab:T200856|T200856]]
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> {{!}} [[phab:T204372|T204372]]
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` ([[phab:T309419|T309419]])
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` ([[phab:T309415|T309415]])
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep


== 2018-09-14 ==
== 2022-05-26 ==
* 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
* 18:33 dancy: Updated Jenkins beta-* job configs
* 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of {{Gerrit|2c7b5825}}
* 14:16 hashar: deployment-deploy01: sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch
* 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
* 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
* 13:48 hashar: Tagged Quibble 0.0.25 {{Gerrit|01663f5d9bcc9e77473732a9fd230271bbd43db4}}
* 11:49 hashar: Cleaned /tmp on deployment-deploy01
* 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ {{!}} [[phab:T204340|T204340]]
* 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
* 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473


== 2018-09-13 ==
== 2022-05-25 ==
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230{{!}}T309230]]
* 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
* 15:46 dancy: Restarted apache2 on gerrit1001
* 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor  mediawiki/vendor fails composer --ansi validate --no-check-publish
* 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 {{!}} [[phab:T203902|T203902]]
* 08:42 hashar: aborted maintenance-disconnect-full-disks job {{!}} [[phab:T204077|T204077]]
* 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
* 05:13 legoktm: onlined integration-slave-docker-1021
* 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
* 05:08 legoktm: deployed
* 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184


== 2018-09-12 ==
== 2022-05-24 ==
* 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for [[phab:T309099|T309099]]
* 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
* 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
* 20:51 marxarelli: deleting integration-slave-docker-1002 instance
* 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
* 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
* 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
* 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
* 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
* 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
* 18:21 marxarelli: deleting integration-slave-docker-1027 instance
* 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
* 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
* 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
* 17:59 marxarelli: deleting integration-slave-docker-1001 instance
* 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
* 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
* 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
* 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
* 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
* 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
* 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker  for Chromium 69 and Firefox 60 - [[phab:T203902|T203902]]
* 00:14 marxarelli: deleting instance integration-slave-docker-1029


== 2018-09-11 ==
== 2022-05-23 ==
* 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts [[phab:T299797|T299797]]
* 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*'  # [[phab:T188742|T188742]]
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - [[phab:T296982|T296982]]


== 2018-09-10 ==
== 2022-05-22 ==
* 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
* 08:34 Amir1: ores {{Gerrit|ae96071}} is going to beta


== 2018-09-09 ==
== 2022-05-21 ==
* 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as [[phab:T308908|T308908]]
* 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265


== 2018-09-08 ==
== 2022-05-20 ==
* 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
* 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
* 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
* 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
* 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
* 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882


== 2018-09-07 ==
== 2022-05-19 ==
* 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 {{!}} Don't re-trigger the test pipeline on patches with C+2 already
* 18:30 legoktm: started gear_client on contint1001
* 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
* 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
* 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4.  8 CPUS can not sustain 5 concurrent Quibble builds {{!}} [[phab:T201972|T201972]]


== 2018-09-06 ==
== 2022-05-18 ==
* 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill {{Gerrit|eae9ba3a1459}} -> stopped a container that had been running for 5 hours
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar`  # [[phab:T307339|T307339]]#7937268
* 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # [[phab:T252071|T252071]]
* 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026


== 2018-09-05 ==
== 2022-05-17 ==
* 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for [[phab:T308558|T308558]]
* 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
* 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
* 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095


== 2018-09-04 ==
== 2022-05-16 ==
* 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for [[phab:T203431|T203431]]
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday [[phab:T299797|T299797]]
* 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809


== 2018-09-03 ==
== 2022-05-14 ==
* 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
* 07:54 hashar: deployment-deploy01 scap sync-l10n master
* 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
* 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
* 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
* 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
* 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278


== 2018-09-02 ==
== 2022-05-13 ==
* 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv {{!}} attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ {{!}} more permissions fixes for beta-scap-eqiad
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap {{!}} fixing beta-scap-update-equiad failures
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk [[phab:T299797|T299797]]
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group [[phab:T305317|T305317]]


== 2018-09-01 ==
== 2022-05-12 ==
* 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom [[phab:T299797|T299797]]
* 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test [[phab:T307137|T307137]]
* 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day  and `docker system prune --force` on Sunday {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
* 00:46 brennen: gitlab: disabling container registries on all existing projects ([[phab:T307537|T307537]])
* 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
* 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase


== 2018-08-31 ==
== 2022-05-11 ==
* 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
* 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
* 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
* 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
* 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
* 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528


== 2018-08-30 ==
== 2022-05-10 ==
* 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/790641 {{!}} [[phab:T300340|T300340]]
* 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for [[phab:T199417|T199417]]
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for [[phab:T199422|T199422]]
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 19:38 thcipriani: bring integration-slave-docker-1025 back online
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per [[phab:T197918|T197918]]
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377
* 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
* 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
* 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
* 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
* 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
* 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073


== 2018-08-29 ==
== 2022-05-09 ==
* 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ ([[phab:T203093|T203093]])
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for [[phab:T295375|T295375]]
* 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189  ([[phab:T203110|T203110]])
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # [[phab:T307963|T307963]]
* 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
* 12:35 hashar: fixed keyholder on deployment-deploy01
* 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002


== 2018-08-28 ==
== 2022-05-08 ==
* 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ {{!}} [[phab:T203009|T203009]]
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # [[phab:T304057|T304057]]
* 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 {{!}} [[phab:T203009|T203009]]
* 17:30 legoktm: reverted back to previous quibble version ([[phab:T203009|T203009]])
* 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 {{!}} https://gerrit.wikimedia.org/r/455861
* 16:13 hashar: building docker containers for quibble 0.0.22
* 16:06 hashar: tagged quibble 0.0.22 {{Gerrit|5093492821693918da4fecdcb15a8ca831ce560f}}


== 2018-08-27 ==
== 2022-05-06 ==
* 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # [[phab:T252071|T252071]]
* 18:49 Amir1: ores:0e8cc73 is going to beta cluster
* 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ {{!}} [[phab:T179963|T179963]]
* 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian {{!}} [[phab:T174338|T174338]]


== 2018-08-25 ==
== 2022-05-05 ==
* 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins  builds from https://integration.wikimedia.org/ci/view/Beta/


== 2018-08-24 ==
== 2022-05-04 ==
* 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # [[phab:T307590|T307590]]
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [[phab:T307507|T307507]]


== 2018-08-23 ==
== 2022-05-03 ==
* 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 11:06 _joe_: reverting my cherry-pick
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682


== 2018-08-22 ==
== 2022-05-02 ==
* 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 20:05 bearND: (beta): Update mobileapps to {{Gerrit|141ff20}} ([[phab:T202105|T202105]] [[phab:T202237|T202237]])
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # [[phab:T301766|T301766]]
* 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
* 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport [[phab:T202220|T202220]]
* 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
* 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
* 05:37 TimStarling: on deployment-db03 installing pt-heartbeat


== 2018-08-21 ==
== 2022-04-29 ==
* 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants ([[phab:T305301|T305301]])
* 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms ([[phab:T202260|T202260]])
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for [[phab:T297687|T297687]]
* 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used)  rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* {{!}} [[phab:T202341|T202341]]
* 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
* 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
* 01:13 legoktm: triggering seccheck jobs manually on contint1001


== 2018-08-20 ==
== 2022-04-28 ==
* 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for [[phab:T293509|T293509]]
* 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
* 18:57 brennen: finished editing mediawiki-new-errors
* 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 20:16 mateusbs17: (beta): Update mobileapps to {{Gerrit|95e976d}} ([[phab:T202105|T202105]])
* 09:03 hashar: Gerrit upgraded to 3.4.4  at roughly 8:00 UTC
* 19:53 marxarelli: Deleting instance integration-slave-docker-1016
* 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
* 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
* 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
* 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
* 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced {{!}} [[phab:T192561|T192561]]
* 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
* 01:42 legoktm: running jobs manually on contint1001 with gear_client
* 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
* 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
* 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884


== 2018-08-18 ==
== 2022-04-27 ==
* 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # [[phab:T292759|T292759]]
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 {{!}} [[phab:T303867|T303867]] {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/777029


== 2018-08-17 ==
== 2022-04-26 ==
* 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
* 15:40 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for [[phab:T306068|T306068]]
* 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for [[phab:T306897|T306897]]
* 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204
* 08:46 addshore: reloaded zuul
* 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618


== 2018-08-16 ==
== 2022-04-25 ==
* 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to {{Gerrit|a808c9d}} ([[phab:T201979|T201979]])
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI


== 2018-08-15 ==
== 2022-04-20 ==
* 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for [[phab:T201782|T201782]].
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service
* 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins


== 2018-08-14 ==
== 2022-04-18 ==
* 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for [[phab:T306329|T306329]]
* 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per [[phab:T306054|T306054]]; it's apparently unused, so this shouldn't break anything.
* 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
* 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
* 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2


== 2018-08-13 ==
== 2022-04-14 ==
* {{safesubst:SAL entry|1=23:05 Krenair: deactivated and cleaned puppet node entries for deployment-{tin,mira} T192561}}
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
* 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
* 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......


== 2018-08-12 ==
== 2022-04-12 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes ([[phab:T304290|T304290]], [[phab:T305143|T305143]])
* 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141
* 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179


== 2018-08-10 ==
== 2022-04-08 ==
* 17:58 marxarelli: Reloading zuul to deploy {{Gerrit|I32007b56d05c80aadefa686c505e1da23c343dc1}}
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
* 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins


== 2018-08-09 ==
== 2022-04-07 ==
* 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # [[phab:T304461|T304461]], output is at P24204
* 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=<nowiki>{</nowiki>enwiki,cswiki<nowiki>}</nowiki> --delete # [[phab:T304461|T304461]]
* 17:13 bearND: (beta): Update mobileapps to {{Gerrit|616ffef}}
* 01:42 awight: [[phab:T201518|T201518]]: ORES, fawiki wp10, misc updates


== 2018-08-08 ==
== 2022-04-06 ==
* 20:39 James_F: Updated mobileapps to {{Gerrit|162ebcf}} in Beta Cluster
* 20:03 thcipriani: rebooting phabricator
* 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for [[phab:T304596|T304596]]
* 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055  / [[phab:T199850|T199850]]


== 2018-08-07 ==
== 2022-04-04 ==
* 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for [[phab:T294260|T294260]]
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / [[phab:T201107|T201107]])
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 19:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450644
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul


== 2018-08-06 ==
== 2022-04-02 ==
* 16:11 addshore: reload zuul for  https://gerrit.wikimedia.org/r/450604
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
* 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
* 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514


== 2018-08-04 ==
== 2022-03-31 ==
* 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957


== 2018-08-02 ==
== 2022-03-29 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for [[phab:T304948|T304948]]
* 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*`  [[phab:T304932|T304932]] [[phab:T302477|T302477]]
* 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it ([[phab:T201077|T201077]])
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
* 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
* 14:07 gehel: re-imaging depoyment-elastic* to stretch
* 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py


== 2018-08-01 ==
== 2022-03-28 ==
* 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # [[phab:T252071|T252071]]
* 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones {{!}} [[phab:T197469|T197469]]
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 [[phab:T252071|T252071]]
* 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 ([[phab:T199801|T199801]])
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` [[phab:T220606|T220606]]
* 10:08 dcausse: deployment-prep [[phab:T199270|T199270]]: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`.  They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 09:53 hashar: Tag Quibble 1.4.5 @ {{Gerrit|abe16d574}} {{!}} [[phab:T291549|T291549]]


== 2018-07-31 ==
== 2022-03-27 ==
* 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting [[phab:T304736|T304736]]
* 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
* 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor .  We use 'es6' branch instead.  (made for dcausse )
* 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399


== 2018-07-30 ==
== 2022-03-26 ==
* 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
* 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
* 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:03 Krinkle: Delete and recreate deployment-webperf13 ([[phab:T195312|T195312]] / [[phab:T180761|T180761]])
* 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
* 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value ([[phab:T200634|T200634]])
* 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/  {{!}} [[phab:T195918|T195918]]
* 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, [[phab:T195709|T195709]]
* 17:04 hashar: github: granted @thcipriani owner rights
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087


== 2018-07-29 ==
== 2022-03-25 ==
* 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord


== 2018-07-28 ==
== 2022-03-24 ==
* 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
* 20:00 hashar: reloading Zuul for {{Gerrit|Id844e1723a38eed627af03397cf0ad90c7b09a32}} # [[phab:T299320|T299320]]
* 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # [[phab:T197698|T197698]]
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for [[phab:T304652|T304652]]
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for [[phab:T303899|T303899]]
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck


== 2018-07-27 ==
== 2022-03-23 ==
* 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
* 17:35 brennen: restarting phabricator for [[phab:T304540|T304540]], brief downtime expected
* 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container {{!}} [[phab:T195918|T195918]]
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
* 14:43 hashar: Building tox-certcentral container for vgutierrez !
* 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
* 12:05 hashar: Building releng/quibble images 0.0.21
* 11:40 hashar: Tagged quibble 0.0.21  {{Gerrit|e681dce021e}}
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ [[phab:T200508|T200508]]
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
* 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337


== 2018-07-26 ==
== 2022-03-22 ==
* 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` [[phab:T304226|T304226]]
* 15:03 zeljkof: Reloading Zuul to deploy {{Gerrit|208a505b5d2d864e57924325ca7665cf1231c8dc}}
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945
* 04:00 kart_: Updated cxserver to {{Gerrit|e98c81b}}


== 2018-07-24 ==
== 2022-03-21 ==
* 05:22 kart_: Updated cxserver to {{Gerrit|d3c9d15}}
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # [[phab:T300203|T300203]]


== 2018-07-23 ==
== 2022-03-18 ==
* 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to {{Gerrit|254cef5}}
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 [[phab:T304147|T304147]]
* 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, [[phab:T195312|T195312]]
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
* 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 [[phab:T195312|T195312]]
* 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / [[phab:T195312|T195312]]
* 15:34 Krinkle: Creating deployment-webperf13 - [[phab:T195312|T195312]]
* 15:34 Krinkle: Deleting deployment-webperf12 - [[phab:T195312|T195312]]
* 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
* 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB.  The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable


== 2018-07-19 ==
== 2022-03-17 ==
* 21:47 hashar: CI: Force npm loglevel to 'info'  via a Zuul parameter {{!}} [[phab:T198348|T198348]]
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
* 19:06 hashar: Tag Quibble 1.4.4 @ {{Gerrit|56b2c9ba52c}} # [[phab:T300340|T300340]]
* 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/#/c/446863 {{!}} [[phab:T198348|T198348]]
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/770468 {{!}} [[phab:T300340|T300340]]
* 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/442989 {{!}} [[phab:T198348|T198348]]
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]] {{!}} [[phab:T252071|T252071]]
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` {{!}} https://gerrit.wikimedia.org/r/770921 {{!}} [[phab:T216244|T216244]]


== 2018-07-17 ==
== 2022-03-16 ==
* 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - [[phab:T195312|T195312]], [[phab:T180761|T180761]].
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for [[phab:T303958|T303958]]
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for [[phab:T302607|T302607]] and [[phab:T302608|T302608]]
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep


== 2018-07-16 ==
== 2022-03-15 ==
* 21:10 awight: ran namespaceDupes.php on beta enwiki
* 18:26 brennen: gitlab: removed most existing /people groups
* 19:56 bearND: (beta): Update mobileapps to {{Gerrit|bed7b29}}
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership ([[phab:T274461|T274461]], [[phab:T300935|T300935]])
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # [[phab:T216244|T216244]]
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # [[phab:T216244|T216244]]
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job


== 2018-07-14 ==
== 2022-03-14 ==
* 21:20 Krenair: redirected security_audit traffic (see [[phab:T72181|T72181]]) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in [[phab:T192996|T192996]])
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds -  [[phab:T179963|T179963]]
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for [[phab:T267890|T267890]]
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for [[phab:T267890|T267890]]
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for [[phab:T267890|T267890]]
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 {{!}} [[phab:T300340|T300340]]
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079


== 2018-07-13 ==
== 2022-03-11 ==
* 17:31 Reedy: Reloading Zuul to deploy stuff
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki


== 2018-07-12 ==
== 2022-03-10 ==
* 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 20:03 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for [[phab:T250496|T250496]] / [[phab:T303417|T303417]].
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]


== 2018-07-11 ==
== 2022-03-09 ==
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|b5e152d}}
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 {{!}} https://phabricator.wikimedia.org/T288312
* 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
* 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch


== 2018-07-10 ==
== 2022-03-08 ==
* 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002  ( apt-get clean , apt-get autoremove --purge ,  nuked /srv/jenkins-workspace/workspace/*lint
* 20:31 brennen: requiring 2fa for all users under /repos
* 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator {{!}} [[phab:T199103|T199103]]
* 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884  as well)
* 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in [[phab:T72792|T72792]])
* 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 {{!}} [[phab:T198171|T198171]]


== 2018-07-09 ==
== 2022-03-07 ==
* 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs ([[phab:T142152|T142152]])
* 10:53 zabe: restarted apache on deployment-mediawiki11 # [[phab:T302699|T302699]]
* 22:09 Krinkle: Deleting instance deployment-apertium02 - [[phab:T142152|T142152]]
* 19:10 Krenair: deleted deployment-mediawiki06 [[phab:T192996|T192996]]
* 17:35 zeljkof: Reloading Zuul to deploy {{Gerrit|7967dad02843d3692abaf3acfd548047f192c9f6}}
* 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation {{!}} [[phab:T198952|T198952]]
* 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport {{!}} [[phab:T198957|T198957]]
* 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat {{!}} [[phab:T198955|T198955]]
* 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon {{!}} [[phab:T198864|T198864]]


== 2018-07-08 ==
== 2022-03-04 ==
* 20:17 Krinkle: Shutting off deployment-apertium02 ([[phab:T142152|T142152]])
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 18:31 Krenair: deleted deployment-mx [[phab:T184244|T184244]]
* 19:13 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/768068
* 16:55 Krenair: deleted deployment-redis01 [[phab:T179371|T179371]]
* 16:54 Krenair: deleted deployment-redis02 [[phab:T179371|T179371]]
* 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/


== 2018-07-07 ==
== 2022-03-03 ==
* 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for [[phab:T278203|T278203]]
* 14:43 hashar: Reloading Zuul for {{Gerrit|Iae45cae8ec209a3e795fe4fd7dd92290565277db}}
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ {{Gerrit|cf5cd1a0a07}}
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ {{Gerrit|63d2855a1e}} # [[phab:T302226|T302226]] [[phab:T302707|T302707]]


== 2018-07-06 ==
== 2022-03-02 ==
* 03:53 kart_: Updated cxserver to {{Gerrit|bfc9c84}}
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for [[phab:T302908|T302908]]
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 15:27 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/767493
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04


== 2018-07-05 ==
== 2022-02-28 ==
* 06:27 kart_: Updated cxserver to {{Gerrit|f8c71a1}}
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin


== 2018-07-04 ==
== 2022-02-26 ==
* 20:15 Reedy: beta unbroke beta code autodeploy [[phab:T72597|T72597]]
* 20:05 zabe: apply [[phab:T302658|T302658]] on deployment-prep centralauth databases
* 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ [[phab:T192996|T192996]]
* 13:24 zabe: apply [[phab:T302660|T302660]] on deployment-prep centralauth databases
* 05:31 Krinkle: beta-scap-eqiad  failing due to mediawiki06 still being a scap target  (despite being shutdown per [[phab:T192996|T192996]])
* 13:19 zabe: apply [[phab:T302659|T302659]] on deployment-prep centralauth databases


== 2018-07-03 ==
== 2022-02-24 ==
* 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev {{!}} [[phab:T198076|T198076]]
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 ([[phab:T195312|T195312]])
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod  ([[phab:T195312|T195312]], [[phab:T194390|T194390]])
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - [[phab:T192996|T192996]]
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1


== 2018-07-02 ==
== 2022-02-23 ==
* 20:28 hashar: building docker image releng/tox:0.1.1  , that adds libssl-dev {{!}} [[phab:T198076|T198076]]
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub {{!}} [[phab:T198322|T198322]]
* 20:10 dancy: Updating scap in beta
* 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # [[phab:T252071|T252071]]
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/  # [[phab:T252071|T252071]]
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # [[phab:T284774|T284774]]


== 2018-06-30 ==
== 2022-02-22 ==
* 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # [[phab:T112147|T112147]]
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki ([[phab:T210492|T210492]])


== 2018-06-28 ==
== 2022-02-21 ==
* 21:17 hasharAway: castor02: nuking cache of npm/node jobs via  rm -fR /srv/jenkins-workspace/caches/*/*/*node*  (note: other jobs might still have a npm cache) {{!}} [[phab:T198348|T198348]]
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # [[phab:T302226|T302226]]
* 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm  and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm {{!}} [[phab:T198348|T198348]]
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker {{!}} [[phab:T198348|T198348]]
* 07:27 hashar: Refreshing all Jenkins jobs
* 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.


== 2018-06-27 ==
== 2022-02-20 ==
* 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 {{!}} [[phab:T196346|T196346]] [[phab:T198336|T198336]]
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
* 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to {{Gerrit|d7221ba}}


== 2018-06-26 ==
== 2022-02-19 ==
* 22:17 Krenair: arming keyholder on deployment-deploy01
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 20:16 Krenair: done the same on -sca02 and -apertium02
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for [[phab:T293705|T293705]]
* 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for [[phab:T293705|T293705]]
* 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015  (converted to m1.medium instances)
* 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
* 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
* 13:18 hashar: cleaned containers on integration-slave-docker-1006


== 2018-06-25 ==
== 2022-02-18 ==
* 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to {{Gerrit|8c76d52}}
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for [[phab:T293705|T293705]]
* 17:45 Krenair: fixed puppet repo rebasing
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for [[phab:T293705|T293705]]
* 16:20 hashar: deployment-prep:  git gc  on a few repositories under /srv/mediawiki-staging/php-master
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 [[phab:T301453|T301453]]
* 12:19 kart_: "Beta: Update cxserver to ece5e7a"
* 09:21 hashar: Reloading Zuul for {{Gerrit|I1494abb5e9e28da951ffb72154a074a16a0f8381}}
* 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice {{!}} [[phab:T196618|T196618]]


== 2018-06-23 ==
== 2022-02-17 ==
* 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # [[phab:T301995|T301995]]
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 11:12 hashar: Bringing deployment-deploy03 back
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome


== 2018-06-22 ==
== 2022-02-16 ==
* 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'    # some instances are causing disk warnings
* 18:20 hashar: Tag Quibble 1.4.1 @ {{Gerrit|d4bd2801de}} # [[phab:T300301|T300301]]
* 14:46 zeljkof: Reloading Zuul to deploy {{Gerrit|94c38c16c14c59bf6ff2de6c9b693b9a903da1c2}}
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 12:34 hashar: Building docker images for Quibble 0.0.19 {{!}} [[phab:T197687|T197687]]
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium {{!}} [[phab:T197687|T197687]]
* 10:09 hashar: Reloading Zuul for {{Gerrit|I997fee0f160ca3049b8085879831bfe175096ced}}
* 09:59 hashar: Reloading Zuul for {{Gerrit|I2ffa016563ad37f1e7c13dcce81deb8ab411c9e2}}


== 2018-06-21 ==
== 2022-02-15 ==
* 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor {{!}} [[phab:T197698|T197698]]
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2  https://gerrit.wikimedia.org/r/c/integration/config/+/441349/
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 15:26 hashar: Reloading Zuul for {{Gerrit|If80b4b4cfa5c1a869ceb220f5b11c272b384a721}}


== 2018-06-20 ==
== 2022-02-14 ==
* 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck {{!}} [[phab:T197806|T197806]]
* 16:16 Amir1: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image {{!}} [[phab:T197806|T197806]]
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 14:26 jnuche: Jenkins upgrade complete [[phab:T301361|T301361]]
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon


== 2018-06-19 ==
== 2022-02-12 ==
* 15:01 hashar: CI docker slaves:  docker container prune --force ; docker image prune --force
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 ([[phab:T289029|T289029]])
* 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php


== 2018-06-18 ==
== 2022-02-10 ==
* 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
* 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:07 Amir1: change_tag_def is back now
* 16:05 Amir1: making enwiki in beta clsuter readonly


== 2018-06-15 ==
== 2022-02-09 ==
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|529965bb8c8cfa2bb7539c7c79c9dd55de6c11ac}}
* 15:22 taavi: deleted shutoff deployment-mx02
* 08:30 hashar: github: deleted lot of archived/read-only extensions and skins {{!}} [[phab:T180864|T180864]]
* 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
* 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
* 07:51 hashar: cleaning disk space on integration-slave-jessie-1001


== 2018-06-14 ==
== 2022-02-08 ==
* 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 14:26 zeljkof: Reloading Zuul to deploy {{Gerrit|04b1c864f3b9d49e9b3882a1fc4c8fbf272c1bd7}}
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
* 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
* 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01{{!}} [[phab:T197207|T197207]]


== 2018-06-13 ==
== 2022-02-07 ==
* 19:47 Krenair: powered off deployment-mx [[phab:T184244|T184244]]
* 20:55 taavi: added Zabe as member of the deployment-prep project [[phab:T301179|T301179]]
* 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera {{!}} TT186096
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027


== 2018-06-12 ==
== 2022-02-04 ==
* 17:40 halfak: deploying ores {{Gerrit|36037b6}}
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
* 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829


== 2018-06-11 ==
== 2022-02-03 ==
* 21:47 halfak: deploying ores {{Gerrit|6ee8775}}
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
* 14:53 hashar: Building Docker images for Quibble 1.4.0  (prepared by kostajh)
* 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean {{!}} [[phab:T196430|T196430]]
* 13:51 kostajh: Tag Quibble 1.4.0 @ {{Gerrit|4231bc2832395d94e29a332fe8d863301a0cd441}} # [[phab:T300340|T300340]] [[phab:T291549|T291549]] [[phab:T225730|T225730]]
* 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *


== 2018-06-10 ==
== 2022-02-02 ==
* 17:03 Krenair: also fixed scap install on -zotero01
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta
* 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'


== 2018-06-09 ==
== 2022-02-01 ==
* 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from [[phab:T182927|T182927]] to test
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged [[phab:T300591|T300591]]
* 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster [[phab:T300591|T300591]]
* 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances [[phab:T179371|T179371]]


== 2018-06-08 ==
== 2022-01-31 ==
* 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to {{Gerrit|9e3ea96c548d7a84be763d38c2d118bc861cf189}} for [[phab:T222216|T222216]]
* 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn {{!}} [[phab:T196433|T196433]]
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # [[phab:T284774|T284774]]
* 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration {{!}} [[phab:T196442|T196442]]
* 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces


== 2018-06-07 ==
== 2022-01-28 ==
* 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 21:45 taavi: running recountCategories.php on all beta wikis per [[phab:T299823|T299823]]#7652496
* 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
* 14:27 hashar: taking heapdump  of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`
* 17:24 Reedy: unstuck jenkins deployments to beta
* 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
* 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
* 14:16 _joe_: killed "puppet master" process in depoloyment-prep
* 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex {{!}} [[phab:T193844|T193844]]
* 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
* 10:41 hashar: Building Docker images for Quibble 0.0.18
* 09:15 addshore: added wmde-leszek to deployment-prep


== 2018-06-06 ==
== 2022-01-27 ==
* 21:40 bearND: BC: Update mobileapps to {{Gerrit|5ea008c}}
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # [[phab:T299431|T299431]]
* 20:36 Amir1: deploying {{Gerrit|65ce165}} on ores
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 20:25 Amir1: deploying {{Gerrit|65e979f}} on ores
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # [[phab:T300214|T300214]]
* 19:59 bearND: Update mobileapps to {{Gerrit|3bf9be5}} on BC
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync'  # [[phab:T300236|T300236]]
* 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) {{!}} [[phab:T196108|T196108]]
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # [[phab:T300214|T300214]]
* 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore (archived repo) {{!}} [[phab:T196513|T196513]]
* 00:24 thcipriani: restarting jenkins
* 18:58 thcipriani: @deployment-tin: sudo keyholder arm
* 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow {{!}} [[phab:T186969|T186969]]
* 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ {{!}} [[phab:T196509|T196509]]
* 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS {{!}} [[phab:T196509|T196509]]


== 2018-06-05 ==
== 2022-01-26 ==
* 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to {{Gerrit|66727b7}}
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - [[phab:T252071|T252071]]
* 12:07 kart_: Update cxserver to {{Gerrit|7fb7671}}
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - [[phab:T234259|T234259]]
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for [[phab:T252071|T252071]]
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for [[phab:T252071|T252071]]
* 16:45 hashar: integration: creating  integration-agent-docker-1023  based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # [[phab:T290783|T290783]]


== 2018-06-04 ==
== 2022-01-25 ==
* 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to {{Gerrit|f579f0d}}
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force [[phab:T196401|T196401]]
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for [[phab:T232948|T232948]]
* 15:44 awight: ORES: Fix [[phab:T194322|T194322]]
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for [[phab:T232948|T232948]]
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for [[phab:T299995|T299995]]
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948  . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # [[phab:T299996|T299996]]
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # [[phab:T299996|T299996]]
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet  [[phab:T299996|T299996]]
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/


== 2018-06-02 ==
== 2022-01-24 ==
* 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
* 21:44 Reedy: unstick beta ci jobs
* 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>  # [[phab:T299933|T299933]]
* 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/<nowiki>{</nowiki>quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker<nowiki>}</nowiki>


== 2018-06-01 ==
== 2022-01-22 ==
* 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
* 13:40 taavi: apply [[phab:T299827|T299827]] on deployment-prep centralauth database
* 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01)  - [[phab:T195314|T195314]]
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
* 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, [[phab:T195314|T195314]]
* 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
* 20:51 legoktm: deleting old versions of docker images
* 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
* 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
* 20:26 mutante: contint1001 - apt-get clean  got a little bit more disk space
* 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
* 08:45 hashar: Bumping Quibble jobs to 0.0.16
* 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716


== 2018-05-31 ==
== 2022-01-21 ==
* 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ {{!}} [[phab:T187622|T187622]]
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
* 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
* 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - [[phab:T195314|T195314]]
* 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs [[phab:T196062|T196062]]
* 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
* 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
* 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
* 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
* 07:16 legoktm: repooled integration-slave-docker-1010
* 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
* 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - [[phab:T195314|T195314]]
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
* 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, [[phab:T195314|T195314]]
* 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722


== 2018-05-30 ==
== 2022-01-20 ==
* 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large ([[phab:T192561|T192561]])
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743  # [[phab:T292729|T292729]]
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents {{!}} https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org


== 2018-05-29 ==
== 2022-01-19 ==
* 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
* 18:20 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
* 17:31 hashar: Adding  https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # [[phab:T299542|T299542]]
* 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - [[phab:T193734|T193734]]
* 10:38 Reedy: kill some stuck jobs [[phab:T299485|T299485]]
* 12:42 zeljkof: Reloading Zuul to deploy {{Gerrit|bcd36d9a2cd2b2ebb1ec77bdf0fcdcc612aeae8d}}
* 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
* 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
* 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
* 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
* 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
* 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
* 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
* 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
* 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
* 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
* 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938


== 2018-05-28 ==
== 2022-01-18 ==
* 11:09 hashar: Building Quibble Docker images 0.0.13 {{!}} [[phab:T195634|T195634]]
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type {{!}} [[phab:T195634|T195634]]
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # [[phab:T299389|T299389]]
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0  # [[phab:T299389|T299389]]
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0


== 2018-05-27 ==
== 2022-01-17 ==
* 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673
* 16:16 hashar: Tagged Quibble 1.3.0 @ {{Gerrit|2b2c7f9a45}} # [[phab:T297480|T297480]] [[phab:T226869|T226869]] [[phab:T294931|T294931]]
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro


== 2018-05-26 ==
== 2022-01-14 ==
* 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # [[phab:T290783|T290783]]
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings [[phab:T265615|T265615]] [[phab:T299211|T299211]]
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta


== 2018-05-25 ==
== 2022-01-13 ==
* 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 21:41 hashar: cleaned up disk space on deployment-tin  . Reenabled jenkins slave.
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # [[phab:T298691|T298691]]
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins


== 2018-05-24 ==
== 2022-01-12 ==
* 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
* 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
* 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
* 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
* 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
* 15:56 zeljkof: Reloading Zuul to deploy {{Gerrit|2e02d97a03e9aaf6a01e287f381934b8a7b03c9a}}
* 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
* 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
* 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846


== 2018-05-23 ==
== 2022-01-11 ==
* 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
* 20:07 bearND: Beta Cluster: Update mobileapps to {{Gerrit|29ebe0f}}
* 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
* 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
* 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363


== 2018-05-22 ==
== 2022-01-10 ==
* 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for [[phab:T194926|T194926]], finished without errors on any of the hosts)
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done [[phab:T194926|T194926]]
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for [[phab:T222291|T222291]]
* 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
* 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref [[phab:T195312|T195312]]


== 2018-05-21 ==
== 2022-01-05 ==
* 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697  to get rid of the wmf-quibble-apache jobs # [[phab:T285649|T285649]]
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/


== 2018-05-20 ==
== 2022-01-04 ==
* 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms"  https://gerrit.wikimedia.org/r/751422
* 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383


== 2018-05-19 ==
== 2022-01-03 ==
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|9f3a185cb7d020757830e546e30e5d5e4ed67978}}
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
* 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
* 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
* 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
* 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845


== 2018-05-18 ==
* 18:31 zeljkof: Reloading Zuul to deploy {{Gerrit|f8e9625622aab96f0718f5fc9655117c8ebc3f55}}
* 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs [[phab:T194900|T194900]] [[phab:T191050|T191050]]
== 2018-05-17 ==
* 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image
== 2018-05-16 ==
* 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs [[phab:T177896|T177896]]
* 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
* 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308
== 2018-05-15 ==
* 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
* 23:22 legoktm: live hacking zuul config on contint1001
* 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020
== 2018-05-14 ==
* 19:41 bearND: BC: Update mobileapps to {{Gerrit|39c16e4}}
* 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
* 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
* 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0
== 2018-05-11 ==
* 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
* 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default
== 2018-05-10 ==
* 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
* 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
* 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
* 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
* 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
* 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
* 19:36 legoktm: kicked postmerge queue in jenkins
* 19:10 twentyafterfour: reset jenkins slave on deployment-tin
* 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' {{!}} [[phab:T194376|T194376]]
* 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'
== 2018-05-09 ==
* 22:53 awight: ORES: wheels fixups
* 21:31 awight: Bump ORES wheels
* 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
* 21:04 awight: ORES: drafttopic in beta
* 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space
== 2018-05-08 ==
* 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
* 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 ([[phab:T190717|T190717]])
== 2018-05-07 ==
* 20:19 bearND: Update mobileapps to {{Gerrit|c1f4de6}} in BC
* 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
* 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
* 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
* 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495
== 2018-05-06 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
* 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290
== 2018-05-05 ==
* afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
* 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
* 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
* 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055
== 2018-05-04 ==
* 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
* 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock
== 2018-05-03 ==
* 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 {{!}} [[phab:T193767|T193767]]
* 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159
== 2018-05-02 ==
* 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
* 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
* 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for [[phab:T193611|T193611]]
* 10:43 hashar: Building quibble docker iamges 0.0.11-2  [[phab:T118371|T118371]] [[phab:T193222|T193222]]
* 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
* 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
* 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291
== 2018-05-01 ==
* 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'
== 2018-04-30 ==
* 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble/  (thanks zeljkof )
* 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740
== 2018-04-27 ==
* 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
* 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins.  Triggered with a custom gear_client.py script from contint1001 . PID 13134
== 2018-04-26 ==
* 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
* 19:00 hashar: Building releng/quibble 0.0.11 docker images
* 17:41 mobrovac: stopped cpjobqueue and purging logs
* 16:51 thcipriani: mark integration-slave-jessie-1001 online
* 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline
== 2018-04-25 ==
* 21:27 hasharAway: building releng/quibble docker images @0.0.10
* 21:13 hasharAway: tagged quibble 0.0.10 {{Gerrit|a4e88ae}}
* 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
* 18:55 awight: ORES: Revscoring 2.2.2
* 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769
== 2018-04-24 ==
* 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859
== 2018-04-23 ==
* 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
* 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
* 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to {{Gerrit|b011b2a}}
* 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
* 07:27 hashar: Upgrading Blue Ocean on CI Jenkins  1.4.2 -> 1.5.0
== 2018-04-20 ==
* 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
* 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
* 08:38 hashar: Cleaned integration-slave-docker-1005  disk (deleting workspace and all docker images)
* 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
* 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
* 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
* 00:46 awight: roll back ORES beta to master
* 00:08 awight: Push ORES git-lfs to look at stuff
== 2018-04-19 ==
* 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
* 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
* 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 {{!}} [[phab:T192576|T192576]]
* 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
* 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
* 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
* 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
* 09:41 hashar: building releng/quibble-stretch:0.0.8-2
* 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done [[phab:T192476|T192476]]
== 2018-04-18 ==
* 22:53 eddiegp: started nutcracker on deployment-jobrunner03
* 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref [[phab:T178457|T178457]]
* 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
* 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
* 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
* 20:29 Reedy: reloaded zuul for 427469
* 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to {{Gerrit|fb161d7}}
* 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - [[phab:T192471|T192471]]
* 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
* 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
* 13:18 addshore: added WMDE-Fisch to deployment-prep
* 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
* 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/  [[phab:T192432|T192432]]
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
* 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
* 04:26 legoktm: try once more to refresh nodepool snapshot
* 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
== 2018-04-17 ==
* 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
* 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
* 20:28 hasharAway: building releng/quibble.*:0.0.8 images
* 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
* 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009  (m4executor label)
* 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only  use of that role in WMCS
* 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only  use of that role in WMCS
* 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)
== 2018-04-16 ==
* 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
* 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
* 22:25 legoktm: kick stuch beta-update jenkins jobs
* 20:35 mdholloway: restarted restbase and mobileapps services for testing ([[phab:T192287|T192287]])
* 18:58 awight: Update ORES editquality; [[phab:T185903|T185903]]
* 13:13 _joe_: also shutting down deployment-jobrunner02 [[phab:T192071|T192071]]
* 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
* 12:48 _joe_: turning off deployment-mediawiki{04,05} [[phab:T192071|T192071]], will be deleted by EOW
* 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837
== 2018-04-13 ==
* 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
* 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
* 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
* 08:48 eddiegp: debugging [[phab:T189493|T189493]] on beta
* 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state [[phab:T192071|T192071]]
* 07:22 legoktm: restarting jenkins
* 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
* 07:01 hashar: rebuilding quibble containers to use 0.0.7
* 00:53 awight: ORES: Test large file in LFS
== 2018-04-12 ==
* 23:01 awight: Try gerrit-based submodules for ORES, [[phab:T180627|T180627]]
* 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
* 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
* 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep ([[phab:T192071|T192071]])
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
* 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
* 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver [[phab:T192071|T192071]]
* 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/
== 2018-04-11 ==
* 21:04 awight: ORES experiment with git-lfs, [[phab:T180627|T180627]]
* 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446
== 2018-04-10 ==
* 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module ([[phab:T184854|T184854]])
* 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package {{!}} [[phab:T191771|T191771]]
== 2018-04-09 ==
* 19:50 awight: Redundant virtualenv for ORES
* 18:06 awight: Restore to ORES master branch
* 17:17 awight: Test git-lfs in ORES
* 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix [[phab:T191786|T191786]] (success: Notice: Applied catalog in 27.11 seconds)
== 2018-04-06 ==
* 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for [[phab:T173887|T173887]]
* 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 ([[phab:T190717|T190717]])
* 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep ([[phab:T191578|T191578]])
== 2018-04-05 ==
* 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
* 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt {{!}} mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
* 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak [[phab:T173887|T173887]]
* 17:08 bearND: Update mobileapps to {{Gerrit|dbc0687}}
* 17:05 hashar: docker: building releng/quibble:0.0.5
* 17:00 bearND: Update mobileapps to {{Gerrit|2d5ab5b}} on beta
* 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
* 07:52 moritzm: removed  unused/defunct  deployment-videoscaler01 from deployment-prep ([[phab:T191293|T191293]])
* 07:52 moritzm: removed  unused/defunct  deployment-tmh01 from deployment-prep ([[phab:T191293|T191293]])
* 07:47 hashar: deployment-prep: made Muehlenhoff an admin
== 2018-04-04 ==
* 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
* 21:24 awight: Roll back beta ORES
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to {{Gerrit|2d5ab5b}}
* 20:12 awight: Try dsh scap config for ORES
* 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
* 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
* 15:30 addshore: reload zuul for  https://gerrit.wikimedia.org/r/423652
* 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
* 11:58 hashar: Building releng/quibble-stretch:0.0.4
* 11:58 hashar: Building releng/quibble:0.0.4
* 07:11 hashar: deployment-prep: adding EddieGP as a member
== 2018-04-03 ==
* 09:49 hashar: building releng/quibble:0.0.3
* 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
== 2018-04-02 ==
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to {{Gerrit|58a0a88}}
== 2018-03-31 ==
* 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
== 2018-03-30 ==
* 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis ([[phab:T189694|T189694]])
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} [[phab:T191077|T191077]]
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} TT191077
== 2018-03-29 ==
* 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript  updateCollation.php --wiki=fawiki ([[phab:T190965|T190965]])
* 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
* 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
* 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1  https://gerrit.wikimedia.org/r/#/c/422959/
* 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
* 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
* 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
* 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host ([[phab:T191029|T191029]])
* 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug [[phab:T184234|T184234]]" due to git sync failures
== 2018-03-28 ==
* 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
* 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
* 20:45 bearND: Update mobileapps to {{Gerrit|a5833a0}} on BC
* 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
* 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
* 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)
== 2018-03-27 ==
* 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 {{!}} https://gerrit.wikimedia.org/r/#/c/422168/
* 08:30 kart_: Update cxserver to {{Gerrit|9e8ebda}}
* 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs
== 2018-03-26 ==
* 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to {{Gerrit|534f95d}}
* 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 {{!}} [[phab:T190393|T190393]]
* 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)
== 2018-03-23 ==
* 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin
== 2018-03-22 ==
* 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)
== 2018-03-21 ==
* 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
* 23:22 bd808: Raised security group quota from 20 to 40
* 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
* 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to {{Gerrit|e6b50a0}}
* 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
* 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
* 17:52 hashar: deployment-prep:  scap sync-file  docroot/wwwportal/portal '[[phab:T173887|T173887]]'
* 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins
== 2018-03-20 ==
* 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
* 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to {{Gerrit|634a15f}}
* 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
* 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
* 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
* 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
== 2018-03-19 ==
* 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
* 16:57 awight: ORES beta service is restored.
* 16:45 awight: Put ORES-beta back onto master branch
* 16:43 awight: ORES-beta has been down since January.
* 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
* 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
* 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
* 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1  {{!}} https://gerrit.wikimedia.org/r/#/c/420311/ {{!}} [[phab:T190032|T190032]]
* 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
* 07:43 kart_: Add cx_version to cx_translations table ([[phab:T190009|T190009]])
== 2018-03-18 ==
* 22:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/420230
* 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344
== 2018-03-16 ==
* 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
* 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
* 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
* 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/*  # some pack of 6GB belonged to root
* 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
* 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock [[phab:T189859|T189859]]
* 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance
== 2018-03-15 ==
* 17:15 bearND: Update mobileapps to {{Gerrit|c5e1522}}
* 16:43 Amir1: created deployment-ores01 to use stretch
* 16:36 Amir1: deleting deployment-ores01 to reimage
* 15:47 awight: ORES with git-lfs, scap config
* 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
* 00:21 awight: ORES with git-lfs
== 2018-03-14 ==
* 23:30 awight: new ORES submodule, pre-git-lfs
* 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to {{Gerrit|9f4a80c}}
* 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
* 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
* 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
* 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
* 12:16 kart_: update cxserver to {{Gerrit|c355d0c}}
* 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
* 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
* 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch  to Nodepool (they use too much disk space) {{!}} [[phab:T189365|T189365]]
== 2018-03-13 ==
* 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:08 awight: Enable Extension:JADE, [[phab:T176333|T176333]]
* 11:26 kart_: Update cxserver to {{Gerrit|bd2ccfc}}
* 10:39 kart_: Update cxserver to {{Gerrit|fd2c4be}}
* 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 {{!}} [[phab:T188112|T188112]]
* 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ {{!}} [[phab:T188112|T188112]]
* 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3  (HACK) {{!}} [[phab:T188112|T188112]]
* 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 {{!}} [[phab:T188112|T188112]]
== 2018-03-12 ==
* 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to {{Gerrit|5c90db7}}
* 20:09 bd808: Forced puppet run on deployment-logstash2 for {{gerrit{{!}}418986}}
* 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
* 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
* 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
* 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ {{!}} [[phab:T155840|T155840]]
* 12:24 hashar: Jenkins: uninstalled the "cvs" plugin
== 2018-03-10 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - [[phab:T189365|T189365]]
== 2018-03-09 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0  https://gerrit.wikimedia.org/r/#/c/417960/
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container {{!}} [[phab:T188686|T188686]]
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343
== 2018-03-08 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 .  Upgrade packages and reboot.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 18:36 bearND: Update mobileapps to {{Gerrit|afb0167}}
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc  # 1.4GBytes
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 10:27 hashar: Deploy docker images for /deploy repositories {{!}} https://gerrit.wikimedia.org/r/#/c/417217/
== 2018-03-07 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" {{!}} [[phab:T188686|T188686]]
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 02:28 legoktm: manually triggering jenkins jobs
== 2018-03-06 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets  parent permission group to scoring/ores
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic  scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from  scoring/ores itself inheriting from scoring/
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to {{Gerrit|afbe9af}}
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h
== 2018-03-05 ==
* 16:38 Reedy: deleted the stack traces too
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-03-04 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 02:12 Krenair: Regenerated captcha images for [[phab:T164047|T164047]]
== 2018-03-02 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs
== 2018-03-01 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 02:11 legoktm: manually queued jenkins jobs
== 2018-02-28 ==
* 16:31 legoktm: manually queued jenkins jobs
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 07:29 legoktm: mass queuing jenkins jobs again
== 2018-02-27 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 -  https://gerrit.wikimedia.org/r/#/c/415102/
* 10:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d9ed9d4dded7d646fc9c4b54155613eef99752a9}}
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.
== 2018-02-26 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx ([[phab:T188288|T188288]])
== 2018-02-24 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
== 2018-02-23 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 11:50 hashar: deployment-mediawiki04 :  apt-get clean && apt-get autoremove --purge
* 11:50 hashar: deployment-mediawiki04 :  rm /var/cache/hhvm/*.sq3  and restarting hhvm
* 11:48 hashar: deployment-mediawiki04 is out of disk space on /  causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only {{!}} [[phab:T187995|T187995]]
== 2018-02-22 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/413374/
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to {{Gerrit|a1339a9}}
== 2018-02-21 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0  and reloading Zuul {{!}} https://gerrit.wikimedia.org/r/413227
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 16:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d90d617e0794725280f9f2660effabea74533cbe}}
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis
== 2018-02-20 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
== 2018-02-19 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions
== 2018-02-18 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
== 2018-02-16 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely {{!}} https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-02-15 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/  (no jobs touched)
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only {{!}} [[phab:T187428|T187428]]
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255
== 2018-02-14 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to {{Gerrit|f23519f}}
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB
== 2018-02-13 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 10:51 moritzm: shutdown deployment-videoscaler01 ([[phab:T187063|T187063]])
* 10:49 moritzm: shutdown deployment-tmh01 ([[phab:T187063|T187063]])
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/  # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089
== 2018-02-12 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for [[phab:T186870|T186870]]
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. [[phab:T54919|T54919]]
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - [[phab:T186870|T186870]])
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - [[phab:T186870|T186870]])
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - [[phab:T186870|T186870]]
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/
== 2018-02-10 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - [[phab:T186870|T186870]])
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
== 2018-02-09 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 [[phab:T186856|T186856]]
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. {{!}} [[phab:T179378|T179378]] [[phab:T186856|T186856]]
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for {{Gerrit|I73a3142271d76f9640005752b1a336123b921b9f}}
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - [[phab:T186856|T186856]]
* 00:59 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
* 00:58 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
== 2018-02-08 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test {{!}} [[phab:T179552|T179552]]
== 2018-02-07 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230
== 2018-02-06 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix  [[phab:T186381|T186381]]
* 21:14 legoktm: restarted zuul due to patch being stuck ([[phab:T186381|T186381]])
* 19:25 hashar: Restarted Zuul due to [[phab:T186381|T186381]]
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480
== 2018-02-05 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 16:09 mdholloway: mobileapps deployment to BC failed with error ([[phab:T186532|T186532]])
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 {{!}} [[phab:T186494|T186494]]
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 {{!}} [[phab:T186494|T186494]]
== 2018-02-04 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
== 2018-02-03 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins
== 2018-02-01 ==
* 16:14 Amir1: deleting deployment-sca03 ([[phab:T184501|T184501]])
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-31 ==
* 22:01 mdholloway: updated mobileapps to {{Gerrit|3d717fa}} on beta cluster
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-30 ==
* 20:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/404778
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
== 2018-01-29 ==
* 23:24 awight: Experiment with versioned ORES venv, [[phab:T181071|T181071]]
== 2018-01-24 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
== 2018-01-23 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from  integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase {{!}} https://wikitech.wikimedia.org/wiki/Keyholder
== 2018-01-22 ==
* afk: restarting jenkins
== 2018-01-20 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
== 2018-01-19 ==
* 17:23 zeljkof: Reloading Zuul to deploy {{Gerrit|25cab6fdb53f1e5d284192159c43f0ce28f6e604}}
* 17:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7d6b4eecc6d7854de7db6714bcfd006d860f3e28}}
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
== 2018-01-18 ==
* 18:29 bearND: (beta): Update mobileapps to {{Gerrit|2690899}}
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state
== 2018-01-17 ==
* 16:24 zeljkof: Reloading Zuul to deploy {{Gerrit|5f757310f499a6a2cdf036dde3d258046377186f}}
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for [[phab:T185024|T185024]]  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
== 2018-01-16 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images{{!}}grep wmfreleng{{!}}awk '{print \$3}'{{!}}sort{{!}}uniq{{!}}xargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images {{!}} https://gerrit.wikimedia.org/r/#/c/404422/
== 2018-01-15 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy {{Gerrit|ff0a02decf1c75b2cef4f438c622d548c45fd270}}
* 17:14 zeljkof: Reloading Zuul to deploy {{Gerrit|fff6431628437cd9b3ccd5bafe5b58480b2a3714}}
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for [[phab:T180841|T180841]]
* 13:37 kart_: Ran update.php on metawiki betacluster ([[phab:T180841|T180841]])
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git {{!}} [[phab:T174338|T174338]]
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo
== 2018-01-12 ==
* 14:11 zeljkof: Reloading Zuul to deploy {{Gerrit|3816837e6d70ee7eb276c9b00034a5d2012a8afd}}
== 2018-01-11 ==
* 17:50 tgr: added Groovier1 to project members for [[phab:T158909|T158909]]
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403654
* 13:45 hashar: Migrate  composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403647
== 2018-01-10 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers {{!}} https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826
== 2018-01-09 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' {{!}} [[phab:T158243|T158243]] [[phab:T162191|T162191]]
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git {{!}} [[phab:T183569|T183569]]
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki ([[phab:T184276|T184276]])
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores ([[phab:T184282|T184282]])
* 08:48 Amir1: stopping ores services in deployment-sca03 ([[phab:T184282|T184282]])
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 ([[phab:T184282|T184282]])
== 2018-01-08 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
== 2018-01-05 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
== 2018-01-03 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3  the version in apt.wm.o
* 19:09 hashar: apt-get upgrade on deployment-tin  . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)
== 2018-01-01 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for [[phab:T183862|T183862]]
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki ([[phab:T183862|T183862]])
* 19:56 Amir1: restarting ores services in deployment-sca03 ([[phab:T183862|T183862]])


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Revision as of 14:45, 28 June 2022

2022-06-28

2022-06-27

2022-06-24

  • 20:52 taavi: added `denisse` as a member

2022-06-23

2022-06-22

  • 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
  • 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
  • 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
  • 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
  • 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
  • 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
  • 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180

2022-06-21

  • 22:01 brennen: gitlab-runners: re-registering all shared runners
  • 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster

2022-06-20

  • 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
  • 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
  • 14:07 ori: restarted acme-chief on deployment-acme-chief03

2022-06-17

  • 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
  • 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack

2022-06-16

  • 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
  • 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
  • 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]

2022-06-15

  • 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
  • 16:31 hashar: integration-agent-docker-1035: docker image prune
  • 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster

2022-06-14

  • 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
  • 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
  • 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
  • 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production

2022-06-13

  • 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
  • 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore

2022-06-12

2022-06-10

  • 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
  • 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
  • 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
  • 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check

2022-06-09

  • 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
  • 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration

2022-06-08

  • 17:14 hashar: Reloaded Zuul for I393422
  • 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for T237033
  • 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134

2022-06-06

  • 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711

2022-06-02

  • 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
  • 11:26 hashar: Restarting Jenkins on contint2001
  • 11:19 hashar: Restarting Jenkins on releases1002

2022-05-31

  • 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
  • 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
  • 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
  • 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster

2022-05-30

  • 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
  • 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142

2022-05-28

  • 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437

2022-05-27

  • 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
  • 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
  • 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
  • 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
  • 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
  • 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
  • 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
  • 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep

2022-05-26

  • 18:33 dancy: Updated Jenkins beta-* job configs
  • 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 2c7b5825
  • 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch

2022-05-25

  • 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] T309230
  • 15:46 dancy: Restarted apache2 on gerrit1001

2022-05-24

2022-05-23

  • 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
  • 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
  • 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
  • 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
  • 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
  • 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
  • 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982

2022-05-22

2022-05-21

2022-05-20

2022-05-19

2022-05-18

  • 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
  • 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
  • 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
  • 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071

2022-05-17

  • 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558

2022-05-16

2022-05-14

  • 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
  • 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
  • 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request

2022-05-13

2022-05-12

  • 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
  • 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
  • 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
  • 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
  • 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)

2022-05-11

  • 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
  • 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster

2022-05-10

2022-05-09

2022-05-08

  • 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057

2022-05-06

  • 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071

2022-05-05

2022-05-04

2022-05-03

2022-05-02

2022-04-29

2022-04-28

2022-04-27

2022-04-26

  • 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
  • 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
  • 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
  • 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
  • 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
  • 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204

2022-04-25

2022-04-20

  • 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

2022-04-18

  • 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
  • 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
  • 16:48 brennen: phabricator: adding self to acl*blog-admins
  • 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.

2022-04-14

2022-04-12

2022-04-08

2022-04-07

  • 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
  • 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461

2022-04-06

  • 20:03 thcipriani: rebooting phabricator
  • 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596

2022-04-04

2022-04-02

2022-03-31

2022-03-29

  • 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
  • 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
  • 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771

2022-03-28

2022-03-27

  • 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736

2022-03-26

  • 02:37 Reedy: beta-update-databases-eqiad is back to @hourly

2022-03-25

  • 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
  • 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
  • 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
  • 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
  • 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord

2022-03-24

  • 20:00 hashar: reloading Zuul for Id844e1 # T299320
  • 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
  • 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
  • 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck

2022-03-23

  • 17:35 brennen: restarting phabricator for T304540, brief downtime expected
  • 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster

2022-03-22

2022-03-21

  • 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
  • 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203

2022-03-18

  • 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
  • 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
  • 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)

2022-03-17

2022-03-16

2022-03-15

2022-03-14

  • 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
  • 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
  • 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
  • 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
  • 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
  • 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
  • 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
  • 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
  • 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
  • 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
  • 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
  • 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079

2022-03-11

  • 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki

2022-03-10

2022-03-09

2022-03-08

  • 20:31 brennen: requiring 2fa for all users under /repos

2022-03-07

  • 10:53 zabe: restarted apache on deployment-mediawiki11 # T302699

2022-03-04

2022-03-03

2022-03-02

  • 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
  • 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
  • 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
  • 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04

2022-02-28

  • 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
  • 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
  • 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin

2022-02-26

  • 20:05 zabe: apply T302658 on deployment-prep centralauth databases
  • 13:24 zabe: apply T302660 on deployment-prep centralauth databases
  • 13:19 zabe: apply T302659 on deployment-prep centralauth databases

2022-02-24

  • 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
  • 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
  • 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
  • 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
  • 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
  • 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
  • 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
  • 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1

2022-02-23

  • 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
  • 20:10 dancy: Updating scap in beta
  • 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
  • 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
  • 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
  • 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774

2022-02-22

  • 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
  • 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)

2022-02-21

  • 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
  • 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
  • 07:27 hashar: Refreshing all Jenkins jobs

2022-02-20

  • 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo

2022-02-19

  • 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
  • 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
  • 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705

2022-02-18

2022-02-17

  • 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
  • 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
  • 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
  • 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
  • 11:12 hashar: Bringing deployment-deploy03 back
  • 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome

2022-02-16

  • 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
  • 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
  • 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
  • 10:09 hashar: Reloading Zuul for I997fee
  • 09:59 hashar: Reloading Zuul for I2ffa01

2022-02-15

  • 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
  • 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
  • 16:24 hashar: Restarting CI Jenkins for plugins updates
  • 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
  • 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
  • 15:26 hashar: Reloading Zuul for If80b4b

2022-02-14

  • 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
  • 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
  • 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
  • 14:26 jnuche: Jenkins upgrade complete T301361
  • 13:54 jnuche: Jenkins contint instances are going to be restarted soon

2022-02-12

  • 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)

2022-02-10

2022-02-09

  • 15:22 taavi: deleted shutoff deployment-mx02

2022-02-08

  • 17:34 taavi: remove scap from deployment-kafka-main/jumbo
  • 16:23 taavi: hard reboot misbehaving deployment-echostore01
  • 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03

2022-02-07

2022-02-04

2022-02-03

  • 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
  • 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
  • 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730

2022-02-02

  • 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
  • 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta

2022-02-01

2022-01-31

  • 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
  • 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774

2022-01-28

  • 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
  • 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`

2022-01-27

  • 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
  • 19:34 Amir1: Reloading Zuul to deploy 757464
  • 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
  • 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
  • 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
  • 00:24 thcipriani: restarting jenkins

2022-01-26

  • 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
  • 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
  • 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
  • 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
  • 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783

2022-01-25

  • 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
  • 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
  • 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
  • 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
  • 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
  • 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
  • 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
  • 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
  • 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
  • 09:22 Reedy: unblocked beta again
  • 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/

2022-01-24

  • 21:44 Reedy: unstick beta ci jobs
  • 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
  • 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
  • 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}

2022-01-22

  • 13:40 taavi: apply T299827 on deployment-prep centralauth database
  • 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts

2022-01-21

  • 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
  • 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`

2022-01-20

  • 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
  • 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
  • 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
  • 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
  • 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
  • 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
  • 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
  • 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
  • 12:10 hashar: contint2001 : docker container prune && docker image prune
  • 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
  • 12:04 hashar: contint1001 `docker image prune`
  • 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org

2022-01-19

2022-01-18

  • 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
  • 18:01 taavi: added ryankemper as a member of the deployment-prep project
  • 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
  • 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
  • 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
  • 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0

2022-01-17

  • 16:28 hashar: Building Quibble 1.3.0 Docker images
  • 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
  • 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro

2022-01-14

  • 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
  • 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
  • 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
  • 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta

2022-01-13

  • 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
  • 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
  • 16:27 dancy: testing scap prep auto on deployment-deploy03
  • 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
  • 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
  • 10:52 hashar: Restarting Jenkins CI for plugins update
  • 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
  • 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins

2022-01-12

2022-01-11

  • 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings

2022-01-10

  • 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
  • 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291

2022-01-05

2022-01-04

2022-01-03

  • 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
  • 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=591650"