You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*)
imported>Stashbot
(Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old)
 
Line 1: Line 1:
== 2018-10-02 ==
== 2023-02-03 ==
* 16:45 thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*
* 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old
* 16:20 thcipriani: investigating integration-slave-docker-1037
* 08:56 godog: bounce logstash
* 07:59 legoktm: deployed https://gerrit.wikimedia.org/r/463905
* 07:38 legoktm: building quibble-stretch-php71 docker image
* 05:32 legoktm: rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883
* 03:26 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886


== 2018-10-01 ==
== 2023-02-02 ==
* 20:09 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space
* 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for [[phab:T321309|T321309]]
* 17:13 marxarelli: bringing integration-slave-docker-1041 back online following source directory clean up ([[phab:T205902|T205902]])
* 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete  "' research/navigation-vectors'"`
* 16:52 marxarelli: removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041
* 07:10 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest
* 05:24 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to {{Gerrit|a1fa41b}}


== 2018-09-29 ==
== 2023-01-31 ==
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463615
* 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
* 01:22 Krinkle: Marking integration-slave-docker-1041 as offline (again). Why/How did it come back?
* 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
* 00:24 Krinkle: Marking integration-slave-docker-1041 as offline. Various odd build failures, including https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/6913/console
* 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
* 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for [[phab:T326372|T326372]]


== 2018-09-28 ==
== 2023-01-30 ==
* 15:31 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki  --prefix ([[phab:T201009|T201009]])
* 23:23 James_F: Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' for [[phab:T327332|T327332]]
* 14:50 thcipriani: investigating integration-slave-docker-1041
* 18:35 James_F: Zuul: [mediawiki/extensions/PushAll] Add basic tests
* 07:35 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@bf09080]: Update mobileapps to {{Gerrit|7878ffc}}
* 14:39 hashar: Building Docker image for https://gerrit.wikimedia.org/r/884904
* 10:59 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/884835/


== 2018-09-27 ==
== 2023-01-27 ==
* 17:53 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@a0054ba]: Update mobileapps to {{Gerrit|0d6c2b7}}
* 20:42 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/884372
* 07:38 mdholloway: deployment-maps04 updated tilerator and kartotherian node modules ([[phab:T195513|T195513]], [[phab:T200594|T200594]])
* 19:35 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/884365/
* 14:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/884143 {{!}} [[phab:T315621|T315621]]
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884309
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884283
* 08:39 hashar: devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master {{!}} [[phab:T325128|T325128]]


== 2018-09-26 ==
== 2023-01-25 ==
* 15:13 thcipriani: integration-slave-docker-1034:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online -- https://phabricator.wikimedia.org/P7592
* 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner.  [[phab:T327949|T327949]]
* 15:05 thcipriani: integration-slave-docker-1033:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for [[phab:T327949|T327949]]
* 14:47 thcipriani: investigating integration-slave-docker-103{3,4}
* 11:57 Amir1: [[gerrit:462927]] (ores) is going to beta
* 08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
* 08:14 hashar: Restarting CI Jenkins on contint1001


== 2018-09-25 ==
== 2023-01-24 ==
* 23:01 marxarelli: configured new jenkins node integration-slave-docker-1043 with 6 executors
* 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [[phab:T326372|T326372]]
* 23:01 marxarelli: replaced integration-slave-docker-1042 with new integration-slave-docker-1043 instance
* 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for [[phab:T326372|T326372]]
* 22:39 marxarelli: launching new integration-slave-docker-1042 bigram instance
* 17:19 thcipriani: restarting ci jenkins for updates
* 22:33 marxarelli: deleting remaining m1.medium instances used as m4executors ([[phab:T205362|T205362]])
* 17:15 thcipriani: restarted releases-jenkins for plugin updates
* 22:15 marxarelli: taking remaining m1.medium m4executor jenkins nodes offline ([[phab:T205362|T205362]])
* 18:16 marxarelli: reconfiguring bigram jenkins nodes to use 6 executors. 7 were configured by mistake ([[phab:T205362|T205362]])
* 18:00 marxarelli: configuring new integration-slave-docker-1041 jenkins node with 7 executors ([[phab:T205362|T205362]])
* 17:42 marxarelli: configuring new jenkins node integration-slave-docker-1040 with 7 executors ([[phab:T205362|T205362]])
* 17:38 marxarelli: launching integration-slave-docker-1041 bigram instance ([[phab:T205362|T205362]])
* 17:30 marxarelli: the puppet parameter for docker_lvm_volume specified in horizon was not applied correctly on the first puppet run for some reason. tearing down integration-slave-docker-1039...
* 17:25 marxarelli: launching integration-slave-docker-1040 bigram instance ([[phab:T205362|T205362]])
* 17:24 marxarelli: deleting instances integration-slave-docker-1007/1008 ([[phab:T205362|T205362]])
* 17:13 marxarelli: launching new integration-slave-docker-1039 bigram instance
* 17:12 marxarelli: taking integration-slave-docker-1007/1008 offline for replacement ([[phab:T205362|T205362]])
* 17:09 marxarelli: deleting integration-slave-docker-1030/1031 instances ([[phab:T205362|T205362]])
* 17:05 marxarelli: taking integration-slave-docker-1030/1031 offline for replacement
* 16:47 marxarelli: increasing executors to 7 for jenkins nodes integration-slave-docker-1033/1034
* 16:46 marxarelli: new instance creation delayed due to quota
* 16:45 marxarelli: launching new integration-slave-docker-1039/1040 bigram instances
* 01:21 legoktm: deployed https://gerrit.wikimedia.org/r/450508
* 00:36 legoktm: deploying https://gerrit.wikimedia.org/r/462609
* 00:22 legoktm: deploying https://gerrit.wikimedia.org/r/453447


== 2018-09-24 ==
== 2023-01-23 ==
* 20:21 bearND: (beta): Update mobileapps to {{Gerrit|badb463}}
* 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
* 10:55 hashar: gerrit: granting labs/tools/* project owners the ability to submit changes {{!}} https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/
* 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for [[phab:T292818|T292818]]
* 09:51 hashar: deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo  and created a new. Its size of 69632 bytes might cause slow writes?? {{!}} [[phab:T204762|T204762]]
* 09:24 hashar: Live hacked scap code on deployment-deploy01 for [[phab:T204762|T204762]] and reverted hack changes
* 08:32 hashar: deployment-deploy01 rm -fR /tmp/scap_l10n_*
* 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/462341
* 03:45 kart_: Update cxserver to {{Gerrit|d913793}}


== 2018-09-23 ==
== 2023-01-20 ==
* 14:03 Krenair: rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining
* 06:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881665 {{!}} [[phab:T327301|T327301]]
* 01:05 andrewbogott: rebooted deployment-maps03; OOM and also [[phab:T205195|T205195]]


== 2018-09-22 ==
== 2023-01-19 ==
* 20:51 Hauskatze: github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for [[phab:T183891|T183891]]
* 20:01 James_F: Zuul: Add Izno to the CI allowlist
* 20:05 Hauskatze: github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer {{!}} [[phab:T183891|T183891]]; moving to the next one
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # [[phab:T299954|T299954]]
* 18:21 Krenair: went to do the same with deployment-maps03 and accidentally broke SSH access to the server
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
* 18:21 Krenair: removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced [[phab:T153468|T153468]], this just explains why puppet hadn't purged stuff)
* 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
* 18:01 Krenair: rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
* 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
* 18:00 Krenair: rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting ([[phab:T153468|T153468]]), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
* 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907


== 2018-09-21 ==
== 2023-01-18 ==
* 17:26 marxarelli: adding jenkins node integration-slave-docker-1038 with 7 executors
* 21:33 dancy: Rebooting deployment-jobrunner04 for [[phab:T327329|T327329]]
* 16:47 marxarelli: added new jenkins node integration-slave-docker-1037 with 7 executors
* 17:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881457 {{!}} [[phab:T327301|T327301]]
* 15:49 marxarelli: replacing integration-slave-docker-1036 with new bigram instance
* 15:42 thcipriani: deployment-jobrunner04: sudo rm -rf /srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~
* 15:48 marxarelli: taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations
* 15:31 andrewbogott: shutting down deployment-imagescaler03 as it is long-overdue for replacement.  See [[phab:T294148|T294148]] for details.
* 15:17 marxarelli: integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations
* 10:10 arturo: bump trove quotas ([[phab:T326674|T326674]])
* 15:15 marxarelli: taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend
* 15:13 marxarelli: launching integration-slave-docker-1037 bigram instance
* 13:03 Amir1: ores:7b987a7 is going beta
* 05:32 legoktm: deployed https://gerrit.wikimedia.org/r/461510


== 2018-09-20 ==
== 2023-01-17 ==
* 23:48 marxarelli: adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors
* 12:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880951/
* 23:23 marxarelli: launching integration-slave-docker-1035/1036 bigram instances
* 10:05 hashar: Updating sonar-scanner jobs https://gerrit.wikimedia.org/r/c/integration/config/+/880893/
* 23:20 marxarelli: taking integration-slave-docker-1004/1005 offline for replacement ([[phab:T202160|T202160]])
* 09:52 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-4
* 16:52 Amir1: deploy ores:ee2d28b
* 08:32 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880503 {{!}} [[phab:T321837|T321837]]
* 11:21 hashar: Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. {{!}} [[phab:T203181|T203181]] and [[phab:T204919|T204919]]
* 00:21 James_F: Zuul: [operations/debs/cadvisor] Add debian-glue-unstable CI for [[phab:T325557|T325557]]
* 11:17 hashar: deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes)
* 00:21 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] +AbuseFilter phan dep
* 11:07 hashar: deployment-deploy01 is out of disk space (again)


== 2018-09-19 ==
== 2023-01-16 ==
* 21:18 Hauskatze: github: deleted https://github.com/wikimedia/mediawiki-services-ocg-collection {{!}} [[phab:T183891|T183891]]
* 22:29 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/880541
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|a224e99}}
* 00:19 Krinkle: Reloading Zuul
* 18:56 Amir1: ores:76fe25a goes to beta ([[phab:T204862|T204862]])


== 2018-09-18 ==
== 2023-01-13 ==
* 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
* 10:41 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/879754
* 09:46 hashar: updating mwselenium-quibble-docker to Quibble 0.0.26
* 01:35 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879687
* 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
* 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins  # untouched since 2016
* 07:31 hashar: cleaning disk on deployment-sca01
* 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors


== 2018-09-17 ==
== 2023-01-12 ==
* 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
* 21:49 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879649
* 22:00 Reedy: update.php in screen is done
* 10:42 hashar: Deployed docroot change https://gerrit.wikimedia.org/r/878860 "zuul: Link to report_url if available"
* 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
* 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
* 20:17 bearND: (beta): Update mobileapps to {{Gerrit|d56e4cf}}
* 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy {{Gerrit|I5df2b72fa411adf05027c8f6745284e7e1dedf7a}}
* 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
* 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest ([[phab:T109776|T109776]])
* 13:24 hashar: Building docker images for Quibble 0.0.25 {{!}} https://gerrit.wikimedia.org/r/460536
* 12:58 hashar: mwselenium-quibble-docker to Chromium 69 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ {{!}} [[phab:T204214|T204214]]
* 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
* 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
* 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
* 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775


== 2018-09-16 ==
== 2023-01-10 ==
* 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
* 21:14 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/27
* 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism {{!}} [[phab:T200856|T200856]]
* 16:50 James_F: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf for [[phab:T326653|T326653]]
* 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> {{!}} [[phab:T204372|T204372]]
* 15:28 James_F: Zuul: Add experimental jobs for fundraising/REL1_39 for [[phab:T326372|T326372]]


== 2018-09-14 ==
== 2023-01-09 ==
* 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
* 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
* 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
* 16:30 wm-bot2: Increased quotas by 6 cores ([[phab:T326568|T326568]]) - cookbook ran by arturo@nostromo
* 14:16 hashar: deployment-deploy01:  sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
* 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests
* 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
* 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
* 13:48 hashar: Tagged Quibble 0.0.25 {{Gerrit|01663f5d9bcc9e77473732a9fd230271bbd43db4}}
* 11:49 hashar: Cleaned /tmp on deployment-deploy01
* 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ {{!}} [[phab:T204340|T204340]]
* 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
* 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473


== 2018-09-13 ==
== 2023-01-08 ==
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
* 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
* 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
* 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix
* 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor  mediawiki/vendor fails composer --ansi validate --no-check-publish
* 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 {{!}} [[phab:T203902|T203902]]
* 08:42 hashar: aborted maintenance-disconnect-full-disks job {{!}} [[phab:T204077|T204077]]
* 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
* 05:13 legoktm: onlined integration-slave-docker-1021
* 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
* 05:08 legoktm: deployed
* 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184


== 2018-09-12 ==
== 2023-01-07 ==
* 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
* 22:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876327
* 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
* 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
* 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
* 20:51 marxarelli: deleting integration-slave-docker-1002 instance
* 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
* 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
* 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
* 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
* 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
* 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
* 18:21 marxarelli: deleting integration-slave-docker-1027 instance
* 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
* 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
* 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
* 17:59 marxarelli: deleting integration-slave-docker-1001 instance
* 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
* 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
* 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
* 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
* 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
* 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
* 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker  for Chromium 69 and Firefox 60 - [[phab:T203902|T203902]]
* 00:14 marxarelli: deleting instance integration-slave-docker-1029


== 2018-09-11 ==
== 2023-01-06 ==
* 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
* 16:59 James_F: Docker: Publishing new CI images with composer 2.5.1 for [[phab:T319257|T319257]]
* 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
* 11:43 hashar: Refreshing Jenkins jobs to update Quibble jobs with php-excimer {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/876190
* 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
* 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
* 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
* 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*'  # [[phab:T188742|T188742]]


== 2018-09-10 ==
== 2023-01-05 ==
* 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
* 09:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868193 {{!}} Make CentralAuth a clone dependency for MassMessage {{!}} [[phab:T145430|T145430]]
* 08:34 Amir1: ores {{Gerrit|ae96071}} is going to beta


== 2018-09-09 ==
== 2023-01-04 ==
* 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
* 19:42 MatmaRex: Ran maintenansce script refreshGlobalimagelinks.php for [[phab:T322588|T322588]]
* 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
* 02:47 TimStarling: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/874959
* 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265


== 2018-09-08 ==
== 2023-01-03 ==
* 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
* 19:14 dduvall: restarting zuul
* 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
* 19:12 Krinkle: Disable publishing of rPMSM, ref [[phab:T143162|T143162]], [[phab:T326112|T326112]]
* 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
* 17:38 hashar: gerrit: reactivated account for `jmando` using `gerrit set-account 9431 --active` {{!}} [[phab:T326159|T326159]]
* 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
* 14:20 hashar: integration: restarting all instances for Linux kernel upgrade
* 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
* 07:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # [[phab:T322368|T322368]]
* 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882


== 2018-09-07 ==
* 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
* 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
* 18:30 legoktm: started gear_client on contint1001
* 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
* 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
* 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4.  8 CPUS can not sustain 5 concurrent Quibble builds {{!}} [[phab:T201972|T201972]]
== 2018-09-06 ==
* 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
* 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
* 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill {{Gerrit|eae9ba3a1459}} -> stopped a container that had been running for 5 hours
* 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
* 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026
== 2018-09-05 ==
* 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
* 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
* 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
* 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095
== 2018-09-04 ==
* 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for [[phab:T203431|T203431]]
* 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
* 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808
== 2018-09-03 ==
* 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
* 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
* 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
* 07:54 hashar: deployment-deploy01 scap sync-l10n master
* 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
* 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
* 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
* 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
* 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278
== 2018-09-02 ==
* 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv {{!}} attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
* 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ {{!}} more permissions fixes for beta-scap-eqiad
* 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
* 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap {{!}} fixing beta-scap-update-equiad failures
== 2018-09-01 ==
* 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
* 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
* 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
* 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
* 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
* 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
* 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
== 2018-08-31 ==
* 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
* 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
* 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
* 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
* 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
* 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
* 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528
== 2018-08-30 ==
* 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
* 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for [[phab:T199417|T199417]]
* 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for [[phab:T199422|T199422]]
* 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
* 19:38 thcipriani: bring integration-slave-docker-1025 back online
* 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per [[phab:T197918|T197918]]
* 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
* 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
* 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
* 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
* 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
* 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073
== 2018-08-29 ==
* 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ ([[phab:T203093|T203093]])
* 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189  ([[phab:T203110|T203110]])
* 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
* 12:35 hashar: fixed keyholder on deployment-deploy01
* 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002
== 2018-08-28 ==
* 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ {{!}} [[phab:T203009|T203009]]
* 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 {{!}} [[phab:T203009|T203009]]
* 17:30 legoktm: reverted back to previous quibble version ([[phab:T203009|T203009]])
* 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 {{!}} https://gerrit.wikimedia.org/r/455861
* 16:13 hashar: building docker containers for quibble 0.0.22
* 16:06 hashar: tagged quibble 0.0.22 {{Gerrit|5093492821693918da4fecdcb15a8ca831ce560f}}
== 2018-08-27 ==
* 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
* 18:49 Amir1: ores:0e8cc73 is going to beta cluster
* 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ {{!}} [[phab:T179963|T179963]]
* 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian {{!}} [[phab:T174338|T174338]]
== 2018-08-25 ==
* 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
* 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
* 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107
== 2018-08-24 ==
* 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
* 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
* 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO
== 2018-08-23 ==
* 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
* 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
* 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
* 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
* 11:06 _joe_: reverting my cherry-pick
* 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
* 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge
== 2018-08-22 ==
* 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
* 20:05 bearND: (beta): Update mobileapps to {{Gerrit|141ff20}} ([[phab:T202105|T202105]] [[phab:T202237|T202237]])
* 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
* 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport [[phab:T202220|T202220]]
* 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
* 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
* 05:37 TimStarling: on deployment-db03 installing pt-heartbeat
== 2018-08-21 ==
* 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
* 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms ([[phab:T202260|T202260]])
* 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used)  rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* {{!}} [[phab:T202341|T202341]]
* 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
* 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
* 01:13 legoktm: triggering seccheck jobs manually on contint1001
== 2018-08-20 ==
* 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
* 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
* 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
* 20:16 mateusbs17: (beta): Update mobileapps to {{Gerrit|95e976d}} ([[phab:T202105|T202105]])
* 19:53 marxarelli: Deleting instance integration-slave-docker-1016
* 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
* 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
* 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
* 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
* 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced {{!}} [[phab:T192561|T192561]]
* 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
* 01:42 legoktm: running jobs manually on contint1001 with gear_client
* 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
* 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
* 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884
== 2018-08-18 ==
* 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04
== 2018-08-17 ==
* 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
* 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
* 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
* 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
* 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
* 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
* 08:46 addshore: reloaded zuul
* 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618
== 2018-08-16 ==
* 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to {{Gerrit|a808c9d}} ([[phab:T201979|T201979]])
== 2018-08-15 ==
* 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for [[phab:T201782|T201782]].
* 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins
== 2018-08-14 ==
* 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
* 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
* 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
* 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
* 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2
== 2018-08-13 ==
* {{safesubst:SAL entry|1=23:05 Krenair: deactivated and cleaned puppet node entries for deployment-{tin,mira} T192561}}
* 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
* 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
* 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......
== 2018-08-12 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
* 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
* 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
* 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
* 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
* 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179
== 2018-08-10 ==
* 17:58 marxarelli: Reloading zuul to deploy {{Gerrit|I32007b56d05c80aadefa686c505e1da23c343dc1}}
* 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins
== 2018-08-09 ==
* 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
* 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
* 17:13 bearND: (beta): Update mobileapps to {{Gerrit|616ffef}}
* 01:42 awight: [[phab:T201518|T201518]]: ORES, fawiki wp10, misc updates
== 2018-08-08 ==
* 20:39 James_F: Updated mobileapps to {{Gerrit|162ebcf}} in Beta Cluster
* 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
* 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055  / [[phab:T199850|T199850]]
== 2018-08-07 ==
* 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / [[phab:T201107|T201107]])
* 19:52 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/450644
* 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005
== 2018-08-06 ==
* 16:11 addshore: reload zuul for  https://gerrit.wikimedia.org/r/450604
* 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
* 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514
== 2018-08-04 ==
* 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296
== 2018-08-02 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
* 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
* 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it ([[phab:T201077|T201077]])
* 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
* 14:07 gehel: re-imaging depoyment-elastic* to stretch
* 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py
== 2018-08-01 ==
* 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
* 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones {{!}} [[phab:T197469|T197469]]
* 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 ([[phab:T199801|T199801]])
* 10:08 dcausse: deployment-prep [[phab:T199270|T199270]]: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01
== 2018-07-31 ==
* 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
* 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
* 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor .  We use 'es6' branch instead.  (made for dcausse )
* 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399
== 2018-07-30 ==
* 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
* 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - [[phab:T195312|T195312]] / [[phab:T180761|T180761]]
* 23:03 Krinkle: Delete and recreate deployment-webperf13 ([[phab:T195312|T195312]] / [[phab:T180761|T180761]])
* 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
* 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value ([[phab:T200634|T200634]])
* 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/  {{!}} [[phab:T195918|T195918]]
* 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, [[phab:T195709|T195709]]
* 17:04 hashar: github: granted @thcipriani owner rights
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087
== 2018-07-29 ==
* 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002
== 2018-07-28 ==
* 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
* 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # [[phab:T197698|T197698]]
== 2018-07-27 ==
* 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
* 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container {{!}} [[phab:T195918|T195918]]
* 14:43 hashar: Building tox-certcentral container for vgutierrez !
* 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 {{!}} https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
* 12:05 hashar: Building releng/quibble images 0.0.21
* 11:40 hashar: Tagged quibble 0.0.21  {{Gerrit|e681dce021e}}
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ [[phab:T200508|T200508]]
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
* 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337
== 2018-07-26 ==
* 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
* 15:03 zeljkof: Reloading Zuul to deploy {{Gerrit|208a505b5d2d864e57924325ca7665cf1231c8dc}}
* 04:00 kart_: Updated cxserver to {{Gerrit|e98c81b}}
== 2018-07-24 ==
* 05:22 kart_: Updated cxserver to {{Gerrit|d3c9d15}}
== 2018-07-23 ==
* 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to {{Gerrit|254cef5}}
* 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
* 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, [[phab:T195312|T195312]]
* 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 [[phab:T195312|T195312]]
* 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / [[phab:T195312|T195312]]
* 15:34 Krinkle: Creating deployment-webperf13 - [[phab:T195312|T195312]]
* 15:34 Krinkle: Deleting deployment-webperf12 - [[phab:T195312|T195312]]
* 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
* 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB.  The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable
== 2018-07-19 ==
* 21:47 hashar: CI:  Force npm loglevel to 'info'  via a Zuul parameter {{!}} [[phab:T198348|T198348]]
* 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
* 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/#/c/446863 {{!}} [[phab:T198348|T198348]]
* 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes {{!}} https://gerrit.wikimedia.org/r/442989 {{!}} [[phab:T198348|T198348]]
== 2018-07-17 ==
* 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - [[phab:T195312|T195312]], [[phab:T180761|T180761]].
== 2018-07-16 ==
* 21:10 awight: ran namespaceDupes.php on beta enwiki
* 19:56 bearND: (beta): Update mobileapps to {{Gerrit|bed7b29}}
== 2018-07-14 ==
* 21:20 Krenair: redirected security_audit traffic (see [[phab:T72181|T72181]]) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in [[phab:T192996|T192996]])
* 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds -  [[phab:T179963|T179963]]
== 2018-07-13 ==
* 17:31 Reedy: Reloading Zuul to deploy stuff
== 2018-07-12 ==
* 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
* 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05
== 2018-07-11 ==
* 20:07 bearND: (beta): Update mobileapps to {{Gerrit|b5e152d}}
* 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
* 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch
== 2018-07-10 ==
* 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002  ( apt-get clean , apt-get autoremove --purge ,  nuked /srv/jenkins-workspace/workspace/*lint
* 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator {{!}} [[phab:T199103|T199103]]
* 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884  as well)
* 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in [[phab:T72792|T72792]])
* 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 {{!}} [[phab:T198171|T198171]]
== 2018-07-09 ==
* 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs ([[phab:T142152|T142152]])
* 22:09 Krinkle: Deleting instance deployment-apertium02 - [[phab:T142152|T142152]]
* 19:10 Krenair: deleted deployment-mediawiki06 [[phab:T192996|T192996]]
* 17:35 zeljkof: Reloading Zuul to deploy {{Gerrit|7967dad02843d3692abaf3acfd548047f192c9f6}}
* 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation {{!}} [[phab:T198952|T198952]]
* 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport {{!}} [[phab:T198957|T198957]]
* 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat {{!}} [[phab:T198955|T198955]]
* 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon {{!}} [[phab:T198864|T198864]]
== 2018-07-08 ==
* 20:17 Krinkle: Shutting off deployment-apertium02 ([[phab:T142152|T142152]])
* 18:31 Krenair: deleted deployment-mx [[phab:T184244|T184244]]
* 16:55 Krenair: deleted deployment-redis01 [[phab:T179371|T179371]]
* 16:54 Krenair: deleted deployment-redis02 [[phab:T179371|T179371]]
* 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/
== 2018-07-07 ==
* 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git
== 2018-07-06 ==
* 03:53 kart_: Updated cxserver to {{Gerrit|bfc9c84}}
== 2018-07-05 ==
* 06:27 kart_: Updated cxserver to {{Gerrit|f8c71a1}}
== 2018-07-04 ==
* 20:15 Reedy: beta unbroke beta code autodeploy [[phab:T72597|T72597]]
* 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ [[phab:T192996|T192996]]
* 05:31 Krinkle: beta-scap-eqiad  failing due to mediawiki06 still being a scap target  (despite being shutdown per [[phab:T192996|T192996]])
== 2018-07-03 ==
* 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
* 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
* 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev {{!}} [[phab:T198076|T198076]]
* 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 ([[phab:T195312|T195312]])
* 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod  ([[phab:T195312|T195312]], [[phab:T194390|T194390]])
* 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - [[phab:T192996|T192996]]
== 2018-07-02 ==
* 20:28 hashar: building docker image releng/tox:0.1.1  , that adds libssl-dev {{!}} [[phab:T198076|T198076]]
* 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub {{!}} [[phab:T198322|T198322]]
* 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023
== 2018-06-30 ==
* 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space
== 2018-06-28 ==
* 21:17 hasharAway: castor02: nuking cache of npm/node jobs via  rm -fR /srv/jenkins-workspace/caches/*/*/*node*  (note: other jobs might still have a npm cache) {{!}} [[phab:T198348|T198348]]
* 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm  and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm {{!}} [[phab:T198348|T198348]]
* 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker {{!}} [[phab:T198348|T198348]]
* 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.
== 2018-06-27 ==
* 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 {{!}} [[phab:T196346|T196346]] [[phab:T198336|T198336]]
* 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to {{Gerrit|d7221ba}}
== 2018-06-26 ==
* 22:17 Krenair: arming keyholder on deployment-deploy01
* 20:16 Krenair: done the same on -sca02 and -apertium02
* 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
* 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015  (converted to m1.medium instances)
* 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
* 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
* 13:18 hashar: cleaned containers on integration-slave-docker-1006
== 2018-06-25 ==
* 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to {{Gerrit|8c76d52}}
* 17:45 Krenair: fixed puppet repo rebasing
* 16:20 hashar: deployment-prep:  git gc  on a few repositories under /srv/mediawiki-staging/php-master
* 12:19 kart_: "Beta: Update cxserver to ece5e7a"
* 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice {{!}} [[phab:T196618|T196618]]
== 2018-06-23 ==
* 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up
== 2018-06-22 ==
* 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean'    # some instances are causing disk warnings
* 14:46 zeljkof: Reloading Zuul to deploy {{Gerrit|94c38c16c14c59bf6ff2de6c9b693b9a903da1c2}}
* 12:34 hashar: Building docker images for Quibble 0.0.19 {{!}} [[phab:T197687|T197687]]
* 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium {{!}} [[phab:T197687|T197687]]
== 2018-06-21 ==
* 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor {{!}} [[phab:T197698|T197698]]
* 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2  https://gerrit.wikimedia.org/r/c/integration/config/+/441349/
== 2018-06-20 ==
* 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
* 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck {{!}} [[phab:T197806|T197806]]
* 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image {{!}} [[phab:T197806|T197806]]
== 2018-06-19 ==
* 15:01 hashar: CI docker slaves:  docker container prune --force ; docker image prune --force
* 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php
== 2018-06-18 ==
* 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:07 Amir1: change_tag_def is back now
* 16:05 Amir1: making enwiki in beta clsuter readonly
== 2018-06-15 ==
* 12:19 zeljkof: Reloading Zuul to deploy {{Gerrit|529965bb8c8cfa2bb7539c7c79c9dd55de6c11ac}}
* 08:30 hashar: github: deleted lot of archived/read-only extensions and skins {{!}} [[phab:T180864|T180864]]
* 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
* 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
* 07:51 hashar: cleaning disk space on integration-slave-jessie-1001
== 2018-06-14 ==
* 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
* 14:26 zeljkof: Reloading Zuul to deploy {{Gerrit|04b1c864f3b9d49e9b3882a1fc4c8fbf272c1bd7}}
* 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
* 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
* 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01{{!}} [[phab:T197207|T197207]]
== 2018-06-13 ==
* 19:47 Krenair: powered off deployment-mx [[phab:T184244|T184244]]
* 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera {{!}} TT186096
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027
== 2018-06-12 ==
* 17:40 halfak: deploying ores {{Gerrit|36037b6}}
* 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829
== 2018-06-11 ==
* 21:47 halfak: deploying ores {{Gerrit|6ee8775}}
* 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
* 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean {{!}} [[phab:T196430|T196430]]
* 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-06-10 ==
* 17:03 Krenair: also fixed scap install on -zotero01
* 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
* 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'
== 2018-06-09 ==
* 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from [[phab:T182927|T182927]] to test
* 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
* 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
* 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances [[phab:T179371|T179371]]
== 2018-06-08 ==
* 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
* 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn {{!}} [[phab:T196433|T196433]]
* 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration {{!}} [[phab:T196442|T196442]]
* 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces
== 2018-06-07 ==
* 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
* 17:24 Reedy: unstuck jenkins deployments to beta
* 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
* 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
* 14:16 _joe_: killed "puppet master" process in depoloyment-prep
* 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex {{!}} [[phab:T193844|T193844]]
* 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
* 10:41 hashar: Building Docker images for Quibble 0.0.18
* 09:15 addshore: added wmde-leszek to deployment-prep
== 2018-06-06 ==
* 21:40 bearND: BC: Update mobileapps to {{Gerrit|5ea008c}}
* 20:36 Amir1: deploying {{Gerrit|65ce165}} on ores
* 20:25 Amir1: deploying {{Gerrit|65e979f}} on ores
* 19:59 bearND: Update mobileapps to {{Gerrit|3bf9be5}} on BC
* 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) {{!}} [[phab:T196108|T196108]]
* 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore  (archived repo) {{!}} [[phab:T196513|T196513]]
* 18:58 thcipriani: @deployment-tin: sudo keyholder arm
* 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow {{!}} [[phab:T186969|T186969]]
* 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ {{!}} [[phab:T196509|T196509]]
* 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS  {{!}} [[phab:T196509|T196509]]
== 2018-06-05 ==
* 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to {{Gerrit|66727b7}}
* 12:07 kart_: Update cxserver to {{Gerrit|7fb7671}}
== 2018-06-04 ==
* 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to {{Gerrit|f579f0d}}
* 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force [[phab:T196401|T196401]]
* 15:44 awight: ORES: Fix [[phab:T194322|T194322]]
== 2018-06-02 ==
* 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
* 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
* 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
* 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky
== 2018-06-01 ==
* 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
* 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01)  - [[phab:T195314|T195314]]
* 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, [[phab:T195314|T195314]]
* 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
* 20:51 legoktm: deleting old versions of docker images
* 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
* 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
* 20:26 mutante: contint1001 - apt-get clean  got a little bit more disk space
* 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
* 08:45 hashar: Bumping Quibble jobs to 0.0.16
* 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716
== 2018-05-31 ==
* 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
* 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ {{!}} [[phab:T187622|T187622]]
* 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
* 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - [[phab:T195314|T195314]]
* 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs [[phab:T196062|T196062]]
* 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
* 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
* 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
* 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
* 07:16 legoktm: repooled integration-slave-docker-1010
* 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
* 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - [[phab:T195314|T195314]]
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
* 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, [[phab:T195314|T195314]]
* 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722
== 2018-05-30 ==
* 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
* 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
* 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
* 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
* 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large ([[phab:T192561|T192561]])
== 2018-05-29 ==
* 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
* 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
* 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - [[phab:T193734|T193734]]
* 12:42 zeljkof: Reloading Zuul to deploy {{Gerrit|bcd36d9a2cd2b2ebb1ec77bdf0fcdcc612aeae8d}}
* 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
* 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
* 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
* 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
* 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
* 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
* 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
* 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
* 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
* 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
* 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938
== 2018-05-28 ==
* 11:09 hashar: Building Quibble Docker images 0.0.13 {{!}} [[phab:T195634|T195634]]
* 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type {{!}} [[phab:T195634|T195634]]
* 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems
== 2018-05-27 ==
* 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
* 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673
== 2018-05-26 ==
* 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
* 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours
== 2018-05-25 ==
* 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
* 21:41 hashar: cleaned up disk space on deployment-tin  . Reenabled jenkins slave.
== 2018-05-24 ==
* 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
* 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
* 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
* 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
* 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
* 15:56 zeljkof: Reloading Zuul to deploy {{Gerrit|2e02d97a03e9aaf6a01e287f381934b8a7b03c9a}}
* 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
* 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
* 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846
== 2018-05-23 ==
* 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
* 20:07 bearND: Beta Cluster: Update mobileapps to {{Gerrit|29ebe0f}}
* 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
* 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
* 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363
== 2018-05-22 ==
* 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for [[phab:T194926|T194926]], finished without errors on any of the hosts)
* 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done [[phab:T194926|T194926]]
* 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
* 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref [[phab:T195312|T195312]]
== 2018-05-21 ==
* 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.
== 2018-05-20 ==
* 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
* 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-05-19 ==
* 14:56 zeljkof: Reloading Zuul to deploy {{Gerrit|9f3a185cb7d020757830e546e30e5d5e4ed67978}}
* 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
* 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
* 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
* 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845
== 2018-05-18 ==
* 18:31 zeljkof: Reloading Zuul to deploy {{Gerrit|f8e9625622aab96f0718f5fc9655117c8ebc3f55}}
* 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs [[phab:T194900|T194900]] [[phab:T191050|T191050]]
== 2018-05-17 ==
* 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image
== 2018-05-16 ==
* 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs [[phab:T177896|T177896]]
* 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
* 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308
== 2018-05-15 ==
* 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
* 23:22 legoktm: live hacking zuul config on contint1001
* 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020
== 2018-05-14 ==
* 19:41 bearND: BC: Update mobileapps to {{Gerrit|39c16e4}}
* 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
* 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
* 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0
== 2018-05-11 ==
* 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
* 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default
== 2018-05-10 ==
* 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
* 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
* 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
* 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
* 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
* 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
* 19:36 legoktm: kicked postmerge queue in jenkins
* 19:10 twentyafterfour: reset jenkins slave on deployment-tin
* 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' {{!}} [[phab:T194376|T194376]]
* 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'
== 2018-05-09 ==
* 22:53 awight: ORES: wheels fixups
* 21:31 awight: Bump ORES wheels
* 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
* 21:04 awight: ORES: drafttopic in beta
* 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space
== 2018-05-08 ==
* 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
* 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 ([[phab:T190717|T190717]])
== 2018-05-07 ==
* 20:19 bearND: Update mobileapps to {{Gerrit|c1f4de6}} in BC
* 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
* 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
* 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
* 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495
== 2018-05-06 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
* 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290
== 2018-05-05 ==
* afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
* 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
* 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
* 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055
== 2018-05-04 ==
* 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
* 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock
== 2018-05-03 ==
* 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 {{!}} [[phab:T193767|T193767]]
* 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159
== 2018-05-02 ==
* 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
* 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
* 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for [[phab:T193611|T193611]]
* 10:43 hashar: Building quibble docker iamges 0.0.11-2  [[phab:T118371|T118371]] [[phab:T193222|T193222]]
* 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
* 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
* 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291
== 2018-05-01 ==
* 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'
== 2018-04-30 ==
* 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble/  (thanks zeljkof )
* 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740
== 2018-04-27 ==
* 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
* 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins.  Triggered with a custom gear_client.py script from contint1001 . PID 13134
== 2018-04-26 ==
* 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
* 19:00 hashar: Building releng/quibble 0.0.11 docker images
* 17:41 mobrovac: stopped cpjobqueue and purging logs
* 16:51 thcipriani: mark integration-slave-jessie-1001 online
* 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline
== 2018-04-25 ==
* 21:27 hasharAway: building releng/quibble docker images @0.0.10
* 21:13 hasharAway: tagged quibble 0.0.10 {{Gerrit|a4e88ae}}
* 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
* 18:55 awight: ORES: Revscoring 2.2.2
* 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769
== 2018-04-24 ==
* 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859
== 2018-04-23 ==
* 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
* 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
* 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to {{Gerrit|b011b2a}}
* 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
* 07:27 hashar: Upgrading Blue Ocean on CI Jenkins  1.4.2 -> 1.5.0
== 2018-04-20 ==
* 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
* 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
* 08:38 hashar: Cleaned integration-slave-docker-1005  disk (deleting workspace and all docker images)
* 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
* 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
* 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
* 00:46 awight: roll back ORES beta to master
* 00:08 awight: Push ORES git-lfs to look at stuff
== 2018-04-19 ==
* 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
* 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
* 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 {{!}} [[phab:T192576|T192576]]
* 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
* 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
* 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
* 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
* 09:41 hashar: building releng/quibble-stretch:0.0.8-2
* 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done [[phab:T192476|T192476]]
== 2018-04-18 ==
* 22:53 eddiegp: started nutcracker on deployment-jobrunner03
* 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref [[phab:T178457|T178457]]
* 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
* 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
* 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
* 20:29 Reedy: reloaded zuul for 427469
* 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to {{Gerrit|fb161d7}}
* 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - [[phab:T192471|T192471]]
* 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
* 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
* 13:18 addshore: added WMDE-Fisch to deployment-prep
* 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
* 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/  [[phab:T192432|T192432]]
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
* 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
* 04:26 legoktm: try once more to refresh nodepool snapshot
* 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
== 2018-04-17 ==
* 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
* 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
* 20:28 hasharAway: building releng/quibble.*:0.0.8 images
* 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
* 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009  (m4executor label)
* 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only  use of that role in WMCS
* 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only  use of that role in WMCS
* 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)
== 2018-04-16 ==
* 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
* 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
* 22:25 legoktm: kick stuch beta-update jenkins jobs
* 20:35 mdholloway: restarted restbase and mobileapps services for testing ([[phab:T192287|T192287]])
* 18:58 awight: Update ORES editquality; [[phab:T185903|T185903]]
* 13:13 _joe_: also shutting down deployment-jobrunner02 [[phab:T192071|T192071]]
* 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
* 12:48 _joe_: turning off deployment-mediawiki{04,05} [[phab:T192071|T192071]], will be deleted by EOW
* 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837
== 2018-04-13 ==
* 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
* 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
* 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
* 08:48 eddiegp: debugging [[phab:T189493|T189493]] on beta
* 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state [[phab:T192071|T192071]]
* 07:22 legoktm: restarting jenkins
* 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
* 07:01 hashar: rebuilding quibble containers to use 0.0.7
* 00:53 awight: ORES: Test large file in LFS
== 2018-04-12 ==
* 23:01 awight: Try gerrit-based submodules for ORES, [[phab:T180627|T180627]]
* 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
* 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
* 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep ([[phab:T192071|T192071]])
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
* 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
* 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver [[phab:T192071|T192071]]
* 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/
== 2018-04-11 ==
* 21:04 awight: ORES experiment with git-lfs, [[phab:T180627|T180627]]
* 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446
== 2018-04-10 ==
* 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module ([[phab:T184854|T184854]])
* 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package {{!}} [[phab:T191771|T191771]]
== 2018-04-09 ==
* 19:50 awight: Redundant virtualenv for ORES
* 18:06 awight: Restore to ORES master branch
* 17:17 awight: Test git-lfs in ORES
* 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix [[phab:T191786|T191786]] (success: Notice: Applied catalog in 27.11 seconds)
== 2018-04-06 ==
* 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for [[phab:T173887|T173887]]
* 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 ([[phab:T190717|T190717]])
* 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep ([[phab:T191578|T191578]])
== 2018-04-05 ==
* 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
* 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt {{!}} mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
* 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak [[phab:T173887|T173887]]
* 17:08 bearND: Update mobileapps to {{Gerrit|dbc0687}}
* 17:05 hashar: docker: building releng/quibble:0.0.5
* 17:00 bearND: Update mobileapps to {{Gerrit|2d5ab5b}} on beta
* 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
* 07:52 moritzm: removed  unused/defunct  deployment-videoscaler01 from deployment-prep ([[phab:T191293|T191293]])
* 07:52 moritzm: removed  unused/defunct  deployment-tmh01 from deployment-prep ([[phab:T191293|T191293]])
* 07:47 hashar: deployment-prep: made Muehlenhoff an admin
== 2018-04-04 ==
* 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
* 21:24 awight: Roll back beta ORES
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to {{Gerrit|2d5ab5b}}
* 20:12 awight: Try dsh scap config for ORES
* 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
* 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
* 15:30 addshore: reload zuul for  https://gerrit.wikimedia.org/r/423652
* 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
* 11:58 hashar: Building releng/quibble-stretch:0.0.4
* 11:58 hashar: Building releng/quibble:0.0.4
* 07:11 hashar: deployment-prep: adding EddieGP as a member
== 2018-04-03 ==
* 09:49 hashar: building releng/quibble:0.0.3
* 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
== 2018-04-02 ==
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to {{Gerrit|58a0a88}}
== 2018-03-31 ==
* 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
== 2018-03-30 ==
* 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis ([[phab:T189694|T189694]])
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} [[phab:T191077|T191077]]
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001).  Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException {{!}} TT191077
== 2018-03-29 ==
* 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript  updateCollation.php --wiki=fawiki ([[phab:T190965|T190965]])
* 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
* 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
* 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1  https://gerrit.wikimedia.org/r/#/c/422959/
* 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
* 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
* 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
* 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host ([[phab:T191029|T191029]])
* 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug [[phab:T184234|T184234]]" due to git sync failures
== 2018-03-28 ==
* 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
* 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
* 20:45 bearND: Update mobileapps to {{Gerrit|a5833a0}} on BC
* 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
* 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
* 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)
== 2018-03-27 ==
* 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 {{!}} https://gerrit.wikimedia.org/r/#/c/422168/
* 08:30 kart_: Update cxserver to {{Gerrit|9e8ebda}}
* 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs
== 2018-03-26 ==
* 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to {{Gerrit|534f95d}}
* 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 {{!}} [[phab:T190393|T190393]]
* 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)
== 2018-03-23 ==
* 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin
== 2018-03-22 ==
* 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)
== 2018-03-21 ==
* 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
* 23:22 bd808: Raised security group quota from 20 to 40
* 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
* 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to {{Gerrit|e6b50a0}}
* 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
* 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
* 17:52 hashar: deployment-prep:  scap sync-file  docroot/wwwportal/portal '[[phab:T173887|T173887]]'
* 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins
== 2018-03-20 ==
* 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
* 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to {{Gerrit|634a15f}}
* 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
* 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
* 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
* 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
== 2018-03-19 ==
* 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
* 16:57 awight: ORES beta service is restored.
* 16:45 awight: Put ORES-beta back onto master branch
* 16:43 awight: ORES-beta has been down since January.
* 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
* 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
* 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
* 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1  {{!}} https://gerrit.wikimedia.org/r/#/c/420311/ {{!}} [[phab:T190032|T190032]]
* 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
* 07:43 kart_: Add cx_version to cx_translations table ([[phab:T190009|T190009]])
== 2018-03-18 ==
* 22:22 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/420230
* 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344
== 2018-03-16 ==
* 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
* 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
* 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
* 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/*  # some pack of 6GB belonged to root
* 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
* 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock [[phab:T189859|T189859]]
* 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance
== 2018-03-15 ==
* 17:15 bearND: Update mobileapps to {{Gerrit|c5e1522}}
* 16:43 Amir1: created deployment-ores01 to use stretch
* 16:36 Amir1: deleting deployment-ores01 to reimage
* 15:47 awight: ORES with git-lfs, scap config
* 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
* 00:21 awight: ORES with git-lfs
== 2018-03-14 ==
* 23:30 awight: new ORES submodule, pre-git-lfs
* 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to {{Gerrit|9f4a80c}}
* 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
* 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
* 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
* 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
* 12:16 kart_: update cxserver to {{Gerrit|c355d0c}}
* 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
* 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
* 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch  to Nodepool (they use too much disk space) {{!}} [[phab:T189365|T189365]]
== 2018-03-13 ==
* 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:08 awight: Enable Extension:JADE, [[phab:T176333|T176333]]
* 11:26 kart_: Update cxserver to {{Gerrit|bd2ccfc}}
* 10:39 kart_: Update cxserver to {{Gerrit|fd2c4be}}
* 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 {{!}} [[phab:T188112|T188112]]
* 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ {{!}} [[phab:T188112|T188112]]
* 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3  (HACK) {{!}} [[phab:T188112|T188112]]
* 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 {{!}} [[phab:T188112|T188112]]
== 2018-03-12 ==
* 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to {{Gerrit|5c90db7}}
* 20:09 bd808: Forced puppet run on deployment-logstash2 for {{gerrit{{!}}418986}}
* 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
* 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
* 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
* 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ {{!}} [[phab:T155840|T155840]]
* 12:24 hashar: Jenkins: uninstalled the "cvs" plugin
== 2018-03-10 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - [[phab:T189365|T189365]]
== 2018-03-09 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0  https://gerrit.wikimedia.org/r/#/c/417960/
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container {{!}} [[phab:T188686|T188686]]
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343
== 2018-03-08 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 .  Upgrade packages and reboot.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 18:36 bearND: Update mobileapps to {{Gerrit|afb0167}}
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc  # 1.4GBytes
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 10:27 hashar: Deploy docker images for /deploy repositories {{!}} https://gerrit.wikimedia.org/r/#/c/417217/
== 2018-03-07 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" {{!}} [[phab:T188686|T188686]]
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 02:28 legoktm: manually triggering jenkins jobs
== 2018-03-06 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets  parent permission group to scoring/ores
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic  scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from  scoring/ores itself inheriting from scoring/
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to {{Gerrit|afbe9af}}
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h
== 2018-03-05 ==
* 16:38 Reedy: deleted the stack traces too
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-03-04 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 02:12 Krenair: Regenerated captcha images for [[phab:T164047|T164047]]
== 2018-03-02 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs
== 2018-03-01 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 02:11 legoktm: manually queued jenkins jobs
== 2018-02-28 ==
* 16:31 legoktm: manually queued jenkins jobs
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 07:29 legoktm: mass queuing jenkins jobs again
== 2018-02-27 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 -  https://gerrit.wikimedia.org/r/#/c/415102/
* 10:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d9ed9d4dded7d646fc9c4b54155613eef99752a9}}
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.
== 2018-02-26 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx ([[phab:T188288|T188288]])
== 2018-02-24 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
== 2018-02-23 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 11:50 hashar: deployment-mediawiki04 :  apt-get clean && apt-get autoremove --purge
* 11:50 hashar: deployment-mediawiki04 :  rm /var/cache/hhvm/*.sq3  and restarting hhvm
* 11:48 hashar: deployment-mediawiki04 is out of disk space on /  causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only {{!}} [[phab:T187995|T187995]]
== 2018-02-22 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/413374/
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to {{Gerrit|a1339a9}}
== 2018-02-21 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0  and reloading Zuul {{!}} https://gerrit.wikimedia.org/r/413227
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 16:15 zeljkof: Reloading Zuul to deploy {{Gerrit|d90d617e0794725280f9f2660effabea74533cbe}}
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis
== 2018-02-20 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
== 2018-02-19 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions
== 2018-02-18 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
== 2018-02-16 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely {{!}} https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-02-15 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/  (no jobs touched)
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only {{!}} [[phab:T187428|T187428]]
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255
== 2018-02-14 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to {{Gerrit|f23519f}}
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB
== 2018-02-13 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 10:51 moritzm: shutdown deployment-videoscaler01 ([[phab:T187063|T187063]])
* 10:49 moritzm: shutdown deployment-tmh01 ([[phab:T187063|T187063]])
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/  # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089
== 2018-02-12 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for [[phab:T186870|T186870]]
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. [[phab:T54919|T54919]]
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - [[phab:T186870|T186870]])
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - [[phab:T186870|T186870]])
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - [[phab:T186870|T186870]]
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/
== 2018-02-10 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - [[phab:T186870|T186870]])
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
== 2018-02-09 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 {{!}} https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 [[phab:T186856|T186856]]
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. {{!}} [[phab:T179378|T179378]] [[phab:T186856|T186856]]
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for {{Gerrit|I73a3142271d76f9640005752b1a336123b921b9f}}
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - [[phab:T186856|T186856]]
* 00:59 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
* 00:58 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])
== 2018-02-08 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test {{!}} [[phab:T179552|T179552]]
== 2018-02-07 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230
== 2018-02-06 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix  [[phab:T186381|T186381]]
* 21:14 legoktm: restarted zuul due to patch being stuck ([[phab:T186381|T186381]])
* 19:25 hashar: Restarted Zuul due to [[phab:T186381|T186381]]
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480
== 2018-02-05 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 16:09 mdholloway: mobileapps deployment to BC failed with error ([[phab:T186532|T186532]])
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 {{!}} [[phab:T186494|T186494]]
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 {{!}} [[phab:T186494|T186494]]
== 2018-02-04 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
== 2018-02-03 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins
== 2018-02-01 ==
* 16:14 Amir1: deleting deployment-sca03 ([[phab:T184501|T184501]])
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-31 ==
* 22:01 mdholloway: updated mobileapps to {{Gerrit|3d717fa}} on beta cluster
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-30 ==
* 20:09 Krinkle: Reloading Zuul to deploy  https://gerrit.wikimedia.org/r/404778
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
== 2018-01-29 ==
* 23:24 awight: Experiment with versioned ORES venv, [[phab:T181071|T181071]]
== 2018-01-24 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
== 2018-01-23 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from  integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase {{!}} https://wikitech.wikimedia.org/wiki/Keyholder
== 2018-01-22 ==
* afk: restarting jenkins
== 2018-01-20 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
== 2018-01-19 ==
* 17:23 zeljkof: Reloading Zuul to deploy {{Gerrit|25cab6fdb53f1e5d284192159c43f0ce28f6e604}}
* 17:10 zeljkof: Reloading Zuul to deploy {{Gerrit|7d6b4eecc6d7854de7db6714bcfd006d860f3e28}}
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
== 2018-01-18 ==
* 18:29 bearND: (beta): Update mobileapps to {{Gerrit|2690899}}
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state
== 2018-01-17 ==
* 16:24 zeljkof: Reloading Zuul to deploy {{Gerrit|5f757310f499a6a2cdf036dde3d258046377186f}}
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for [[phab:T185024|T185024]]  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
== 2018-01-16 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images{{!}}grep wmfreleng{{!}}awk '{print \$3}'{{!}}sort{{!}}uniq{{!}}xargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images {{!}} https://gerrit.wikimedia.org/r/#/c/404422/
== 2018-01-15 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy {{Gerrit|ff0a02decf1c75b2cef4f438c622d548c45fd270}}
* 17:14 zeljkof: Reloading Zuul to deploy {{Gerrit|fff6431628437cd9b3ccd5bafe5b58480b2a3714}}
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for [[phab:T180841|T180841]]
* 13:37 kart_: Ran update.php on metawiki betacluster ([[phab:T180841|T180841]])
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git {{!}} [[phab:T174338|T174338]]
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo
== 2018-01-12 ==
* 14:11 zeljkof: Reloading Zuul to deploy {{Gerrit|3816837e6d70ee7eb276c9b00034a5d2012a8afd}}
== 2018-01-11 ==
* 17:50 tgr: added Groovier1 to project members for [[phab:T158909|T158909]]
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403654
* 13:45 hashar: Migrate  composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image  https://gerrit.wikimedia.org/r/403647
== 2018-01-10 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers {{!}} https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826
== 2018-01-09 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' {{!}} [[phab:T158243|T158243]] [[phab:T162191|T162191]]
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git {{!}} [[phab:T183569|T183569]]
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki ([[phab:T184276|T184276]])
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores ([[phab:T184282|T184282]])
* 08:48 Amir1: stopping ores services in deployment-sca03 ([[phab:T184282|T184282]])
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 ([[phab:T184282|T184282]])
== 2018-01-08 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
== 2018-01-05 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
== 2018-01-03 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 {{!}} [[phab:T184114|T184114]]
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3  the version in apt.wm.o
* 19:09 hashar: apt-get upgrade on deployment-tin  . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)
== 2018-01-01 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for [[phab:T183862|T183862]]
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki ([[phab:T183862|T183862]])
* 19:56 Amir1: restarting ores services in deployment-sca03 ([[phab:T183862|T183862]])


{{SAL-archives/Release Engineering}}
{{SAL-archives/Release Engineering}}


<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Latest revision as of 03:35, 3 February 2023

2023-02-03

  • 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old

2023-02-02

  • 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for T321309
  • 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete "' research/navigation-vectors'"`

2023-01-31

  • 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
  • 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
  • 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
  • 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for T326372

2023-01-30

2023-01-27

2023-01-25

  • 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner. T327949
  • 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for T327949

2023-01-24

  • 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now T326372
  • 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for T326372
  • 17:19 thcipriani: restarting ci jenkins for updates
  • 17:15 thcipriani: restarted releases-jenkins for plugin updates

2023-01-23

  • 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
  • 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for T292818

2023-01-20

2023-01-19

  • 20:01 James_F: Zuul: Add Izno to the CI allowlist
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # T299954
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
  • 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
  • 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
  • 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907

2023-01-18

2023-01-17

2023-01-16

2023-01-13

2023-01-12

2023-01-10

2023-01-09

  • 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
  • 16:30 wm-bot2: Increased quotas by 6 cores (T326568) - cookbook ran by arturo@nostromo
  • 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests

2023-01-08

  • 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
  • 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix

2023-01-07

2023-01-06

2023-01-05

2023-01-04

2023-01-03


Archives