You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Release Engineering/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Labslogbot
(deployment-puppetmaster is in a weird rebase state (thcipriani))
imported>Stashbot
(hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/923647)
 
Line 1: Line 1:
== 2016-02-23 ==
== 2023-05-26 ==
* 22:45 thcipriani: deployment-puppetmaster is in a weird rebase state
* 15:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/923647
* 22:25 legoktm: running sync-common manually on deployment-mediawiki02
* 09:59 hashar: Deleted a bunch of mwext-.*-jslint jobs that are no more in used (migrated to either 'npm' or  'jshint' / 'jsonlint' )


== 2016-02-22 ==
== 2023-05-25 ==
* 22:06 bd808: Restarted puppetmaster service on deployment-puppetmaster to "fix" error "invalid byte sequence in US-ASCII"
* 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # [[phab:T334521|T334521]]
* 17:46 jzerebecki: ssh integration-slave-trusty-1017.eqiad.wmflabs 'sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/.git/config.lock
* 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` #  [[phab:T334521|T334521]]
* 16:47 gehel: deployment-prep upgrading deployment-logstash2 to elasticsearch 1.7.5
* 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait  # [[phab:T334521|T334521]]
* 10:26 gehel: deployment-prep upgrading elastic-search to 1.7.5 on deployment-elastic0[5-8]
* 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd


== 2016-02-20 ==
== 2023-05-24 ==
* 20:19 Krinkle: beta-code-update-eqiad job repeatedly stuck at "IRC notifier plugin"
* 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, [[phab:T319477|T319477]]
* 19:29 Krinkle: beta-code-update-eqiad broken because deployment-tin:/srv/mediawiki-staging/php-master/extensions/MobileFrontend/includes/MobileFrontend.hooks.php was modified on the server without commit
* 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # [[phab:T336362|T336362]]
* 19:22 Krinkle: Various beta-mediawiki-config-update-eqiad jobs have been stuck 'queued' for > 24 hours


== 2016-02-19 ==
== 2023-05-23 ==
* 12:09 hashar: killed https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/  been running for 13 hours. Blocked because slave went offline due to labs reboots yesterday
* 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
* 10:15 hashar: Creating a bunch of repository in GitHub to fix Gerrit replication errors
* 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` [[phab:T337172|T337172]]


== 2016-02-18 ==
== 2023-05-21 ==
* 19:20 legoktm: deploying https://gerrit.wikimedia.org/r/271583 and https://gerrit.wikimedia.org/r/271581, both no-ops
* 10:54 Lucas_WMDE: aborted stuck job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/37265/console which was blocking gate-and-submit
* 18:14 legoktm: deploying https://gerrit.wikimedia.org/r/271012
* 17:36 legoktm: deploying https://gerrit.wikimedia.org/r/271555
* 16:01 hashar: deleting instance  integration-slave-precise-1003  think we have enough precise slaves
* 10:44 hashar: Nodepool: JenkinsException: Could not parse JSON info for server[https://integration.wikimedia.org/ci/]


== 2016-02-17 ==
== 2023-05-17 ==
* 07:36 legoktm: deploying https://gerrit.wikimedia.org/r/271201
* 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # [[phab:T336840|T336840]]
* 01:01 yuvipanda: attempting to turn off NFS on 52 instances on deployment-prep project
* 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)


== 2016-02-16 ==
== 2023-05-16 ==
* 23:22 yuvipanda: new instances on deployment-prep no longer get NFS because of https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=311783&oldid=311781
* 15:11 hashar: Killing stall https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ job
* 23:18 hashar: jenkins@gallium find /var/lib/jenkins/config-history/nodes -maxdepth 1 -type d -name 'ci-jessie*' -exec rm -vfR {} \;
* 15:04 hashar: CI Jenkins: updating IRC plugin from 2.42 to 3.829.v12d4b_d1f7650  and instant-messaging plugin from 1.48 to 2.666.va_6c1e97cc252  # [[phab:T283009|T283009]]
* 23:17 hashar: Jenkins accepting slave creations again. Root cause is /var/lib/jenkins/config-history/nodes/ has reached the 32k inode limit.
* 11:30 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919873 {{!}} [[phab:T336218|T336218]]
* 23:14 hashar: Jenkins: Could not create rootDir /var/lib/jenkins/config-history/nodes/ci-jessie-wikimedia-34969/2016-02-16_22-40-23
* 10:55 taavi: update operations-puppet-tests-buster-docker jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/914785
* 23:02 hashar: Nodepool can not authenticate with Jenkins anymore. Thus it can not add slaves it spawned.
* 22:56 hashar: contint: Nodepool instances pool exhausted
* 21:14 andrewbogott: deployment-logstash2 migration finished
* 20:49 jzerebecki: reloading zuul for 3bf7584..67fec7b
* 19:58 andrewbogott: migrating deployment-logstash2 to labvirt1010
* 19:00 hashar: tin: checking out mw 1.27.0-wmf.14
* 15:23 hashar: integration-make-wmfbranch : /mnt/make-wmf-branch  mount now has gid=wikidev and group setuid (i.e. mode 2775)
* 15:20 hashar: integration-make-wmfbranch : change tmpfs to /mnt/make-wmf-branch  (from /var/make-wmf-branch )
* 11:30 jzerebecki: T117710 integration-saltmaster:~# salt -v '*slave-trusty*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm-composer/src/skins/BlueSky'
* 09:52 hashar: will cut the wmf branches this afternoon starting around 14:00 CET


== 2016-02-15 ==
== 2023-05-12 ==
* 16:28 jzerebecki: reloading zuul for 2d16ad3..3bb0afa
* 15:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919347/ {{!}} [[phab:T333861|T333861]]
* 16:10 hashar: Image ci-jessie-wikimedia-1455552377 in wmflabs-eqiad is ready
* 15:25 jzerebecki: reloading zuul for e174335..2d16ad3
* 15:23 hashar: Image ci-jessie-wikimedia-1455549539 in wmflabs-eqiad is ready
* 15:19 hashar: Regenerating Nodepool snapshot. Slave scripts have 0 bytes...
* 15:04 hashar: Slave scripts added to Nodepool instances! Image ci-jessie-wikimedia-1455548346 in wmflabs-eqiad is ready
* 11:05 hashar: Image ci-jessie-wikimedia-1455534001 in wmflabs-eqiad is ready
* 07:52 legoktm: deploying https://gerrit.wikimedia.org/r/270686
* 06:52 legoktm: legoktm@gallium:/srv/org/wikimedia/doc$ sudo -u jenkins-slave rm -rf EventLogging/ GuidedTour/ MultimediaViewer/ TemplateData/
* 06:22 legoktm: deploying https://gerrit.wikimedia.org/r/270677
* 06:12 legoktm: deploying https://gerrit.wikimedia.org/r/270675
* 06:02 legoktm: deploying https://gerrit.wikimedia.org/r/270674
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/270673
* 05:32 legoktm: deploying https://gerrit.wikimedia.org/r/270670
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/270667
* 03:26 legoktm: deploying https://gerrit.wikimedia.org/r/270665
* 02:56 legoktm: deploying https://gerrit.wikimedia.org/r/270657


== 2016-02-14 ==
== 2023-05-11 ==
* 23:54 legoktm: deploying https://gerrit.wikimedia.org/r/270656
* 21:20 RhinosF1: beta scap is paused while MatmaRex tests some changes
* 23:25 legoktm: deploying https://gerrit.wikimedia.org/r/270654
* 20:41 thcipriani: pausing beta-code-update-eqiad/beta-mediawiki-config-update-eqiad/beta-scap-sync-world for debugging train
* 23:13 legoktm: also deploying https://gerrit.wikimedia.org/r/#/c/265098/
* 16:25 hashar: Reloading Zuul
* 23:11 legoktm: deploying https://gerrit.wikimedia.org/r/270651
* 07:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918970 {{!}} [[phab:T333125|T333125]]
* 05:18 bd808: tools.stashbot Testing after restart (T126419)
* 07:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/918474 {{!}} [[phab:T333125|T333125]]


== 2016-02-13 ==
== 2023-05-10 ==
* 06:42 bd808: restarted nutcracker on deployment-mediawiki01
* 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot ([[phab:T327424|T327424]])
* 06:32 bd808: jobrunner on deployment-jobrunner01 enabled after reverting changes from T87928 that caused T126830
* 05:51 bd808: disabled jobrunner process on jobrunner01; queue full of jobs broken by T126830
* 05:31 bd808: trebuchet clone of /srv/jobrunner/jobrunner broken on jobrunner01; failing puppet runs
* 05:25 bd808: jobrunner process on deployment-jobrunner01 badly broken; investigating
* 05:20 bd808: Ran https://phabricator.wikimedia.org/P2273 on deployment-jobrunner01.deployment-prep.eqiad.wmflabs; freed ~500M; disk utilization still at 94%


== 2016-02-12 ==
== 2023-05-09 ==
* 23:54 hashar: beta cluster broken since 20:30 UTC  https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor  havent looked
* 16:36 hashar: zuul enqueue --trigger gerrit --pipeline postmerge --project analytics/datahub --change 916483,4
* 17:36 hashar: salt -v '*slave-trusty*' cmd.run 'apt-get -y install texlive-generic-extra'    # T126422
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/917868 {{!}} [[phab:T329514|T329514]]
* 17:32 hashar: adding texlive-generic-extra on CI slaves by cherry picking https://gerrit.wikimedia.org/r/#/c/270322/ - T126422
* 09:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/916513/ and https://gerrit.wikimedia.org/r/c/integration/config/+/916793/
* 17:19 hashar: get rid of integration-dev   it is broken somehow
* 08:04 hashar: Reloading Zuul to enable Sonar Codehhealth on TitleBlacklist, TextExtracts, TemplateData and SyntaxHighlight_GeSHI Mediawiki extensions # [[phab:T321837|T321837]]
* 17:10 hashar: Nodepool back at spawning instances. contintcloud has been migrated in wmflabs
* 16:51 thcipriani: running  sudo salt '*' -b '10%' deploy.fixurl to fix deployment-prep trebuchet urls
* 16:31 hashar: bd808 added support for saltbot to update tasks automagically!!!! T108720
* 03:10 yurik: attempted to sync graphoid from gerrit 270166 from deployment-tin, but it wouldn't sync.  Tried to git pull sca02, submodules wouldn't pull


== 2016-02-11 ==
== 2023-05-05 ==
* 22:53 thcipriani: shutting down deployment-bastion
* 23:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916634
* 21:28 hashar: pooling back slaves 1001 to 1006
* 17:45 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916527
* 21:18 hashar: re enabling hhvm service on slaves ( https://phabricator.wikimedia.org/T126594 ) Some symlink is missing and only provided by the upstart script grrrrrrr https://phabricator.wikimedia.org/T126658
* 20:52 legoktm: deploying https://gerrit.wikimedia.org/r/270098
* 20:35 hashar: depooling the six recent slaves: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so cannot open shared object file
* 20:29 hashar: pooling integration-slave-trusty-1004 integration-slave-trusty-1005 integration-slave-trusty-1006
* 20:14 hashar: pooling integration-slave-trusty-1001 integration-slave-trusty-1002 integration-slave-trusty-1003
* 19:35 marxarelli: modifying deployment server node in jenkins to point to deployment-tin
* 19:27 thcipriani: running sudo salt -b '10%' '*' cmd.run 'puppet agent -t' from deployment-salt
* 19:27 twentyafterfour: Keeping notes on the ticket: https://phabricator.wikimedia.org/T126537
* 19:24 thcipriani: moving deployment-bastion to deployment-tin
* 17:59 hashar: recreated instances with proper names:  integration-slave-trusty-{1001-1006}
* 17:52 hashar: Created integration-slave-trusty-{1019-1026} as m1.large (note 1023 is an exception it is for Android).  Applied role::ci::slave , lets wait for puppet to finish
* 17:42 Krinkle: Currently testing https://gerrit.wikimedia.org/r/#/c/268802/ in Beta Labs
* 17:27 hashar: Depooling all the ci.medium slaves and deleting them.
* 17:27 hashar: I tried.  The ci.medium instances are too small and MediaWiki tests really need 1.5GBytes of memory :-(
* 16:00 hashar: rebuilding integration-dev https://phabricator.wikimedia.org/T126613
* 15:27 Krinkle: Deploy Zuul config change https://gerrit.wikimedia.org/r/269976
* 11:46 hashar: salt -v '*' cmd.run '/etc/init.d/apache2 restart'  might help for Wikidata browser tests failling
* 11:32 hashar: disabling hhvm service on CI slaves ( https://phabricator.wikimedia.org/T126594 , cherry picked both patches )
* 10:50 hashar: reenabled puppet on CI. All transitioned to a 128MB tmpfs (was 512MB)
* 10:16 hashar: pooling back integration-slave-trusty-1009 and integration-slave-trusty-1010  (tmpfs shrunken)
* 10:06 hashar: disabling puppet on all CI slaves. Trying to lower tmpfs 512MB to 128MB  ( https://gerrit.wikimedia.org/r/#/c/269880/ )
* 02:45 legoktm: deploying https://gerrit.wikimedia.org/r/269853 https://gerrit.wikimedia.org/r/269893


== 2016-02-10 ==
== 2023-05-04 ==
* 23:54 hashar_: depooling Trusty slaves that only have 2GB of ram that is not enough.  https://phabricator.wikimedia.org/T126545
* 13:51 _joe_: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/915652
* 22:55 hashar_: gallium: find /var/lib/jenkins/config-history/config -type f -wholename '*/2015*' -delete  ( https://phabricator.wikimedia.org/T126552 )
* 12:00 _joe_: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/914782
* 22:34 Krinkle: Zuul is back up and procesing Gerrit events, but jobs are still queued indefinitely. Jenkins is not accepting new jobs
* 22:31 Krinkle: Full restart of Zuul. Seems Gearman/Zuul got stuck. All executors were idling. No new Gerrit events processed either.
* 21:22 legoktm: cherry-picking https://gerrit.wikimedia.org/r/#/c/269370/ on integration-puppetmaster again
* 21:17 hashar: CI dust have settled.  Krinkle and I have pooled a lot more Trusty slaves to accommodate for the overload caused by switching to php55 (jobs run on Trusty)
* 21:08 hashar: pooling trusty slaves 1009, 1010, 1021, 1022  with 2 executors  (they are ci.medium)
* 20:38 hashar: cancelling mediawiki-core-jsduck-publish  and mediawiki-core-doxygen-publish jobs manually.  They will catch up on next merge
* 20:34 Krinkle: Pooled integration-slave-trusty-1019 (new)
* 20:28 Krinkle: Pooled integration-slave-trusty-1020 (new)
* 20:24 Krinkle: created integration-slave-trusty-1019 and integration-slave-trusty-1020 (ci1.medium)
* 20:18 hashar: created integration-slave-trusty-1009 and 1010 (trusty ci.medium)
* 20:06 hashar: creating integration-slave-trusty-1021 and integration-slave-trusty-1022 (ci.medium)
* 19:48 greg-g: that cleanup was done by apergos
* 19:48 greg-g: did cleanup across all integration slaves, some were very close to out of room. results:  https://phabricator.wikimedia.org/P2587
* 19:43 hashar: Dropping slaves Precise m1.large  integration-slave-precise-1014 and  integration-slave-precise-1013 , most load shifted to Trusty (php53 -> php55 transition)
* 18:20 Krinkle: Creating a Trusty slave to support increased demand following MediaWIki php53(precise)>php55(trusty) bump
* 16:06 jzerebecki: reloading zuul for 41a92d5..5b971d1
* 15:42 jzerebecki: reloading zuul for 639dd40..41a92d5
* 14:12 jzerebecki: recover a bit of disk space: integration-saltmaster:~# salt --show-timeout '*slave*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/*WikibaseQuality*'
* 13:46 jzerebecki: reloading zuul for 639dd40
* 13:15 jzerebecki: reloading zuul for 3be81c1..e8e0615
* 08:07 legoktm: deploying https://gerrit.wikimedia.org/r/269619
* 08:03 legoktm: deploying https://gerrit.wikimedia.org/r/269613 and https://gerrit.wikimedia.org/r/269618
* 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/269607
* 06:34 legoktm: deploying https://gerrit.wikimedia.org/r/269605
* 02:59 legoktm: deleting 14GB broken workspace of  mediawiki-core-php53lint from integration-slave-precise-1004
* 02:37 legoktm: deleting /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm-composer on trusty-1017, it had a skin cloned into it
* 02:26 legoktm: queuing mwext jobs server-side to identify failing ones
* 02:21 legoktm: deploying https://gerrit.wikimedia.org/r/269582
* 01:03 legoktm: deploying https://gerrit.wikimedia.org/r/269576


== 2016-02-09 ==
== 2023-05-03 ==
* 23:17 legoktm: deploying https://gerrit.wikimedia.org/r/269551
* 09:25 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/914715
* 23:02 legoktm: gracefully restarting zuul
* 22:57 legoktm: deploying https://gerrit.wikimedia.org/r/269547
* 22:29 legoktm: deploying https://gerrit.wikimedia.org/r/269540
* 22:18 legoktm: re-enabling puppet on all CI slaves
* 22:02 legoktm: reloading zuul to see if it'll pickup the new composer-php53 job
* 21:53 legoktm: enabling puppet on just integration-slave-trusty-1012
* 21:52 legoktm: cherry-picked https://gerrit.wikimedia.org/r/#/c/269370/ onto integration-puppetmaster
* 21:50 legoktm: disabling puppet on all trusty/precise CI slaves
* 21:40 legoktm: deploying https://gerrit.wikimedia.org/r/269533
* 17:49 marxarelli: disabled/enabled gearman in jenkins, connection works this time
* 17:49 marxarelli: performed stop/start of zuul on gallium to restore zuul and gearman
* 17:45 marxarelli: "Failed: Unable to Connect" in jenkins when testing gearman connection
* 17:40 marxarelli: killed old zull process manually and restarted service
* 17:39 marxarelli: restart of zuul fails as well. old process cannot be killed
* 17:38 marxarelli: reloading zuul fails with "failed to kill 13660: Operation not permitted"
* 16:06 bd808: Deleted corrupt integration-slave-precise-1003:/mnt/jenkins-workspace/workspace/mediawiki-core-php53lint/.git
* 15:11 hashar: mira: /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.13 php-1.27.0-wmf.13
* 14:51 hashar: ./make-wmf-branch -n 1.27.0-wmf.13 -o master
* 14:50 hashar: pooling back integration-slave-precise1001 - 1004.  Manually fetched git repos in workspace for mediawiki core php53
* 14:49 hashar: make-wmf-branch instance: created a local ssh key pair and set the config to use User: hashar
* 14:13 hashar: pooling  https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/  Mysql is back .. Blame puppet
* 14:12 hashar: de pooling  https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/  Mysql is gone somehow
* 14:04 hashar: Manually git fetching  mediawiki-core in /mnt/jenkins-workspace/workspace/mediawiki-core-php53lint of slaves precise 1001 to 1004  (git on Precise is remarkably too slow)
* 13:28 hashar: salt '*trusty*' cmd.run 'update-alternatives --set php /usr/bin/hhvm'
* 13:28 hashar: salt '*precise*' cmd.run 'update-alternatives --set php /usr/bin/php5'
* 13:18 hashar: salt -v --batch=3 '*slave*' cmd.run 'puppet agent -tv'
* 13:15 hashar: removing https://gerrit.wikimedia.org/r/#/c/269370/ from CI puppet master
* 13:14 hashar: slave recurse infinitely doing /bin/bash -eu /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh  then loop over /bin/bash /usr/bin/php maintenance/install.php --confpath /mnt/jenkins-workspace/workspace/mediawiki-core-qunit/src --dbtype=mysql --dbserver=127.0.0.1:3306 --dbuser=jenkins_u2 --dbpass=pw_jenkins_u2 --dbname=jenkins_u2_mw --pass testpass TestWiki WikiAdmin  https://phabricator.wikimedia.org/T126327
* 12:46 hashar: Mass testing php loop of death:  salt -v '*slave*' cmd.run 'timeout 2s /srv/deployment/integration/slave-scripts/bin/php --version'
* 12:40 hashar: mass rebooting CI slaves from wikitech
* 12:39 hashar: salt -v '*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'"
* 12:33 hashar: all slaves dieing due to PHP looping
* 12:02 legoktm: re-enabling puppet on all trusty/precise slaves
* 11:20 legoktm: cherry-picked https://gerrit.wikimedia.org/r/#/c/269370/ on integration-puppetmaster
* 11:20 legoktm: enabling puppet just on integration-slave-trusty-1012
* 11:13 legoktm: disabling puppet on all *(trusty|precise)* slaves
* 10:26 hashar: pooling in  integration-slave-trusty-1018
* 03:19 legoktm: deploying https://gerrit.wikimedia.org/r/269359
* 02:53 legoktm: deploying https://gerrit.wikimedia.org/r/238988
* 00:39 hashar: gallium edited /usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py  and modified:  replication_timeout = 300 -> replication_timeout = 10
* 00:37 hashar: live hacking Zuul code to have it stop sleeping() on force merge
* 00:36 hashar: killing zuul


== 2016-02-08 ==
== 2023-05-02 ==
* 23:48 legoktm: finally deploying https://gerrit.wikimedia.org/r/269327
* 16:55 dancy: scap version "4.52.0" installed on 15 hosts
* 23:14 hashar: zuul promote --pipeline gate-and-submit --changes 269065,2 https://gerrit.wikimedia.org/r/#/c/269065/
* 08:15 hashar: Reloaded Zuul to enable Sonar Codehealth on CategoryTree https://gerrit.wikimedia.org/r/909331 and PageImages https://gerrit.wikimedia.org/r/909330 # [[phab:T321837|T321837]]
* 23:10 hashar: pooling integration-slave-precise-1001 1002 1004
* 08:14 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/913541 https://gerrit.wikimedia.org/r/913925 https://gerrit.wikimedia.org/r/913191
* 22:47 hashar: Err need to reboot newly provisioned instances before adding them to Jenkins (kernel upgrade,apache restart etc)
* 22:45 hashar: Pooled https://integration.wikimedia.org/ci/computer/integration-slave-precise-1003/
* 22:25 hashar: integration-slave-precise-{1001-1004} applied role::ci::slave::labs, running puppet in slaves.  I have added the instances as Jenkins slaves and put them offline.  Whenever puppet is done, we can mark them online in Jenkins then monitor the jobs running on them are working properly
* 22:15 hashar: Provisioning integration-slave-precise-{1001-1004} https://phabricator.wikimedia.org/T126274 (need more php53 slots)
* 22:13 hashar: Deleted cache-rsync instance superseded by castor instance
* 22:10 hashar: Deleting pmcache.integration.eqiad.wmflabs (was to investigate various kind of central caches).
* 20:14 marxarelli: aborting pending mediawiki-extensions-php53 job for CheckUser
* 20:08 bd808: toggled "Enable Gearman" off and on in Jenkins to wake up deployment-bastion workers
* 14:54 hashar: nodepool: refreshed snapshot image , Image ci-jessie-wikimedia-1454942958 in wmflabs-eqiad is ready
* 14:47 hashar: regenerated nodepool reference image (got rid of grunt-cli https://gerrit.wikimedia.org/r/269126 )
* 09:41 legoktm: deploying https://gerrit.wikimedia.org/r/269093 https://gerrit.wikimedia.org/r/269094
* 09:36 hashar: restarting integration puppetmaster (out of memory / cannot fork)
* 06:11 bd808: tgr set $wgAuthenticationTokenVersion on beta cluster (test run for T124440)
* 02:09 legoktm[NE]: deploying https://gerrit.wikimedia.org/r/268047
* 00:57 legoktm[NE]: deploying https://gerrit.wikimedia.org/r/268031


== 2016-02-06 ==
== 2023-05-01 ==
* 18:34 jzerebecki: reloading zuul for bdb2ed4..46ccca9
* 17:20 brennen: experimenting with ldap user sync on gitlab.devtools


== 2016-02-05 ==
== 2023-04-28 ==
* 13:30 hashar: beta cleaning out /data/project/logs/archive  was from pre logstash area.  We no more log this way since May 2015 apparently
* 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for  [[phab:T335602|T335602]]
* 13:29 hashar: beta deleting /data/project/swift-disk  created in august 2014 , unused since june 2015.  Was a fail attempt at bringing swift to beta
* 13:27 hashar: beta: reclaiming disk space from extensions.git. On bastion: find /srv/mediawiki-staging/php-master/extensions/.git/modules -maxdepth 1 -type d -print -execdir git gc \;
* 13:03 hashar: integration-slave-trusty-1011 went out of disk space. Did some brute clean up and git gc.
* 05:21 Tim: configured mediawiki-extensions-qunit to only run on integration-slave-trusty-1017, did a rebuild and then switched it back


== 2016-02-04 ==
== 2023-04-27 ==
* 22:08 jzerebecki: reloading zuul for bed7be1..f57b7e2
* 17:10 James_F: Zuul: [wvui] Mark repo as archived for [[phab:T310244|T310244]]
* 21:51 hashar: salt-key -d integration-slave-jessie-1001.eqiad.wmflabs
* 21:50 hashar: salt-key -d integration-slave-precise-1011.eqiad.wmflabs
* 00:57 bd808: Got deployment-bastion processing Jenkins jobs again via instructions left by my past self at https://phabricator.wikimedia.org/T72597#747925
* 00:43 bd808: Jenkins agent on deployment-bastion.eqiad doing the trick where it doesn't pick up jobs again


== 2016-02-03 ==
== 2023-04-26 ==
* 22:24 bd808: Manually ran sync-common on deployment-jobrunner01.eqiad.wmflabs to pickup wmf-config changes that were missing (InitializeSettings, Wikibase, mobile)
* 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list
* 17:43 marxarelli: Reloading Zuul to deploy previously undeployed Icd349069ec53980ece2ce2d8df5ee481ff44d5d0 and Ib18fe48fe771a3fe381ff4b8c7ee2afb9ebb59e4
* 15:12 hashar: apt-get upgrade deployment-sentry2
* 15:03 hashar: redeployed rcstream/rcstream on deployment-stream by using git-deploy on deployment-bastion
* 14:55 hashar: upgrading deployment-stream
* 14:42 hashar: pooled back integration-slave-trusty-1015  Seems ok
* 14:35 hashar: manually triggered a bunch of browser tests jobs
* 11:40 hashar: apt-get upgrade deployment-ms-be01 and deployment-ms-be02
* 11:32 hashar: fixing puppet.conf on deployment-memc04
* 11:09 hashar: restarting beta cluster puppetmaster just in case
* 11:07 hashar: beta: apt-get upgrade on delpoyment-cache* hosts  and checking puppet
* 10:59 hashar: integration/beta:  deleting /etc/apt/apt.conf.d/*proxy  files.  There is no need for them, in fact web proxy is not reachable from labs
* 10:53 hashar: integration: switched puppet repo back to 'production' branch, rebased.
* 10:49 hashar: various beta cluster have puppet errors ..
* 10:46 hashar: integration-slave-trusty-1013 heading to out of disk space on /mnt ...
* 10:42 hashar: integration-slave-trusty-1016 out of disk space on /mnt ...
* 03:45 bd808: Puppet failing on deployment-fluorine with "Error: Could not set uid on user[datasets]: Execution of '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists"
* 03:44 bd808: Freed 28G by deleting deployment-fluorine:/srv/mw-log/archive/*2015*
* 03:42 bd808: Ran deployment-bastion.deployment-prep:/home/bd808/cleanup-var-crap.sh and freed 565M


== 2016-02-02 ==
== 2023-04-25 ==
* 18:32 marxarelli: Reloading Zuul to deploy If1f3cb60f4ccb2c1bca112900dbada03a8588370
* 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False  (manually it works, but that's the same command that puppet is supposed to run !?) - [[phab:T257317|T257317]] [[phab:T326368|T326368]]
* 17:42 marxarelli: cleaning mwext-donationinterfacecore125-testextension-php53 workspace on integration-slave-precise-1013
* 17:06 ostriches: running sync-common on mw2051 and mw1119
* 09:38 hashar: Jenkins is fully up and operational
* 09:33 hashar: restarting Jenkins
* 08:47 hashar: pooling back integration-slave-precise1011 , puppet run got fixed ( https://phabricator.wikimedia.org/T125474 )
* 03:48 legoktm: deploying https://gerrit.wikimedia.org/r/267828
* 03:29 legoktm: deploying https://gerrit.wikimedia.org/r/266941
* 00:42 legoktm: due to T125474
* 00:42 legoktm: marked integration-slave-precise-1011 as offline
* 00:39 legoktm: precise-1011 slave hasn't had a puppet run in 6 days


== 2016-02-01 ==
== 2023-04-24 ==
* 23:53 bd808: Logstash working again; I applied a change to the default mapping template for Elasticsearch that ensures that fields named "timestamp" are indexed as plain strings
* 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - [[phab:T327853|T327853]]
* 23:46 bd808: Elasticsearch index template for beta logstash cluster making crappy guesses about syslog events; dropped 2016-02-01 index; trying to fix default mappings
* 23:09 bd808: HHVM logs causing rejections during document parse when inserting in Elasticsearch from logstash. They contain a "timestamp" field that looks like "Feb  1 22:56:39" which is making the mapper in Elasticsearch sad.
* 23:04 bd808: Elasticsearch on deployment-logstash2 rejecting all documents with 400 status. Investigating
* 22:50 bd808: Copying deployment-logstash2.deployment-prep:/var/log/logstash/logstash.log to /srv for debugging later
* 22:48 bd808: deployment-logstash2.deployment-prep:/var/log/logstash/logstash.log is 11G of fail!
* 22:46 bd808: root partition on deployment-logstash2 full
* 22:43 bd808: No data in logstash since 2016-01-30T06:55:37.838Z; investigating
* 15:33 hashar: Image ci-jessie-wikimedia-1454339883 in wmflabs-eqiad is ready
* 15:01 hashar: Refreshing Nodepool image. Might have npm/grunt properly set up
* 03:15 legoktm: deploying https://gerrit.wikimedia.org/r/267630


== 2016-01-31 ==
== 2023-04-23 ==
* 13:35 hashar: Jenkins IRC bot started falling at Jan 30 01:04:00 2016  for whatever reason....  Should be fine now
* 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%<nowiki>{</nowiki>facts.fqdn<nowiki>}</nowiki>' instead of performance.discovery.wmnet as the latter doesn't resolve / would be an invalid cert for https://deployment-webperf21, ref [[phab:T291015|T291015]]
* 13:33 hashar: cancelling/aborting jobs that are stuck while reporting to IRC (mostly browser tests and beta cluster jobs)
* 13:32 hashar: Jenkins jobs are being blocked because they can no more report back to IRC :-(((
* 13:28 hashar: Jenkins jobs are being blocked because they can no more report back to IRC :-(((


== 2016-01-30 ==
== 2023-04-22 ==
* 12:46 hashar: integration-slave-jessie-1001 : fixed puppet.con server name and ran puppet
* 21:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/908941
* 21:47 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/909624
* 19:35 Krinkle: Create excimer_ui_server db user in Beta Cluster on deployment-db13 based on prod grants. Password stored in deployment-puppetmaster04:/var/lib/git/labs/private/ (passwords::excimer_ui_server::$excimer_db_pass). Picked db13 because recommendationapi is here (also prod m2), and because mdb (created for this purpose originally) appears broken since several OS iterations (likely forgotten due to unusual name) [[phab:T301637|T301637]], [[phab:T331956|T331956]]
* 18:05 Krinkle: Fix database hostname dns error at https://performance.wikimedia.beta.wmflabs.org/xhgui/, switch from mdb01 to mdb02, ref [[phab:T301637|T301637]]
* 18:00 Krinkle: Move deployment-xhgui03 config from to deployment-xhgui prefix, https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/1d03d4cf84a148bff7e055d9939c44f30c618d85/deployment-prep/deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud.yaml https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ab1c9d2968ba3c292b81ba76f8c0775151630f94%5E%21/, ref [[phab:T301637|T301637]]


== 2016-01-29 ==
== 2023-04-21 ==
* 18:43 thcipriani: updated scap on beta
* 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # [[phab:T335181|T335181]]
* 16:44 thcipriani: deployed scap updates on beta
* 11:58 _joe_: upgraded hhvm to 3.6 wm8 in deployment-prep


== 2016-01-28 ==
== 2023-04-20 ==
* 23:22 MaxSem: Updated portals on betalabs to master
* 09:59 taavi: update helm-lint jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/910433/
* 22:23 hashar: salt '*slave-precise*' cmd.run 'apt-get install php5-ldap'  ( https://phabricator.wikimedia.org/T124613 )  will need to be puppetized
* 09:13 joe: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/909996
* 18:17 thcipriani: cleaning npm cache on slave machines: salt -v '*slave*' cmd.run 'sudo -i -u jenkins-deploy -- npm cache clean'
* 18:12 thcipriani: running npm cache clean on integration-slave-precise-1011 sudo -i -u jenkins-deploy -- npm cache clean
* 15:25 hashar: apt-get upgrade deployment-sca01 and deployment-sca02
* 15:09 hashar: fixing puppet.conf hostname on deployment-upload  deployment-conftool  deployment-tmh01 deployment-zookeeper01 and deployment-urldownloader
* 15:06 hashar: fixing puppet.con hostname on deployment-upload.deployment-prep.eqiad.wmflabs and running puppet
* 15:00 hashar: Running puppet on deployment-memc02 and deployment-elastic07 . It is catching up with lot of changes
* 14:59 hashar: fixing puppet hostnames on deployment-elastic07
* 14:59 hashar: fixing puppet hostnames on deployment-memc02
* 14:55 hashar: Deleted salt keys deployment-pdf01.eqiad.wmflabs and deployment-memc04.eqiad.wmflabs  (obsolete,  entries with '.deployment-prep.' are already there)
* 07:38 jzerebecki: reload zuul for 4951444..43a030b
* 05:55 jzerebecki: doing https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update
* 03:49 mobrovac: deployment-prep re-enabled puppet on deployment-restbase0x
* 02:49 mobrovac: deployment-prep deployment-restbase01 disabled puppet to set up cassandra for
* 02:27 mobrovac: deployment-prep recreating deployment-restbase01 for T125003
* 02:23 mobrovac: deployment-prep deployment-restbase02 disabled puppet to recreate deployment-restbase01 for T125003
* 01:42 mobrovac: deployment-prep recreating deployment-sca02 for T125003
* 01:28 mobrovac: deployment-prep recreating deployment-sca01 for T125003
* 00:36 mobrovac: deployment-prep re-imaging deployment-mathoid for T125003
* 00:02 jzerebecki: integration-slave-trusty-1016:~$ sudo -i rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/skins/Donate


== 2016-01-27 ==
== 2023-04-19 ==
* 23:49 jzerebecki: integration-slave-precise-1011:~$ sudo -i /etc/init.d/salt-minion restart
* 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for [[phab:T334998|T334998]]
* 23:46 jzerebecki: work around https://phabricator.wikimedia.org/T117710 : salt --show-timeout '*slave*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/skins/BlueSky'
* 10:56 joe: Updating docker-pkg files on contint primary for helm-linter
* 21:19 cscott: updated OCG to version 64050af0456a43344b32e3e93561a79207565eaf (should be no-op after yesterday's deploy)
* 10:29 hashar: triggered bunch of browser tests, deployment-redis01 was dead/faulty
* 10:08 hashar: mass restarting redis-server process on deployment-redis01 (for https://phabricator.wikimedia.org/T124677 )
* 10:07 hashar: mass restarting redis-server process on deployment-redis01
* 09:00 hashar: beta:  commenting out "latency-monitor-threshold 100" parameter from any /etc/redis/redis.conf we have ( https://phabricator.wikimedia.org/T124677 ). Puppet will not reapply it unless distribution is Jessie


== 2016-01-26 ==
== 2023-04-18 ==
* 16:51 cscott: updated OCG to version 64050af0456a43344b32e3e93561a79207565eaf
* 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
* 12:14 hashar: Added Jenkins IRC bot (wmf-insecte) to #wikimedia-perf for https://gerrit.wikimedia.org/r/#/c/265631/
* 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for [[phab:T334397|T334397]]
* 09:30 hashar: restarting Jenkins to upgrade the gearman plugin with https://review.openstack.org/#/c/271543/
* 04:18 bd808: integration-slave-jessie-1001:/mnt full; cleaned up 15G of files in /mnt/pbuilder/build (27 hours after the last time I did that)


== 2016-01-25 ==
== 2023-04-17 ==
* 18:59 twentyafterfour: started redis-server on deployment-redis01 by commenting out latency-monitor-threshold from the redis.conf
* 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for [[phab:T281500|T281500]]
* 15:22 hashar: CI: fixing kernels not upgrading via: rm /boot/grub/menu.lst ; update-grub -y  (i.e.: regenerate the Grub menu from scratch)
* 14:21 hashar: integration-slave-trusty-1015.integration.eqiad.wmflabs  is gone. I have failed the kernel upgrade / grub update
* 01:35 bd808: integration-slave-jessie-1001:/mnt full; cleaned up 15G of files in /mnt/pbuilder/build


== 2016-01-24 ==
== 2023-04-12 ==
* 06:45 legoktm: deploying https://gerrit.wikimedia.org/r/266039
* 15:39 jnuche: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/908248
* 06:13 legoktm: deploying https://gerrit.wikimedia.org/r/266041
* 09:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/899523


== 2016-01-22 ==
== 2023-04-10 ==
* 23:58 legoktm: removed skins from mwext-qunit workspace on trusty-1013 slave
* 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
* 23:34 legoktm: rm -rf /mnt/jenkins-workspace/workspace/mediawiki-phpunit-php53 on slave precise 1012
* 18:44 thcipriani: restart zuul-merger
* 22:45 legoktm: deploying https://gerrit.wikimedia.org/r/265864
* 22:27 hashar: rebooted all CI slaves using OpenStackManager
* 22:09 hashar: rebooting deployment-redis01 (kernel upgrade)
* 21:22 hashar: Image ci-jessie-wikimedia-1453497269 in wmflabs-eqiad is ready (with node 4.2 for https://phabricator.wikimedia.org/T119143 )
* 21:14 hashar: updating nodepool snapshot based on new image
* 21:12 hashar: rebuilding nodepool reference image
* 20:04 hashar: Image ci-jessie-wikimedia-1453492820 in wmflabs-eqiad is ready
* 20:00 hashar: Refreshing nodepool image to hopefully get Nodejs 4.2.4 https://phabricator.wikimedia.org/T124447  https://gerrit.wikimedia.org/r/#/c/265802/
* 16:32 hashar: Nuked corrupted git repo on integration-slave-precise-1012 /mnt/jenkins-workspace/workspace/mediawiki-extensions-php53
* 12:23 hashar: beta: reinitialized keyholder on deployment-bastion.  The proxy apparently  had no identity
* 09:32 hashar: beta cluster Jenkins job have been stalled for 9hours and 25 minutes. Disabling/reenabling the Gearman plugin to remove the deadlock


== 2016-01-21 ==
== 2023-04-09 ==
* 21:41 hashar: restored role::mail::mx on deployment-mx
* 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2
* 21:36 hashar: dropping role::mail::mx from deployment-mx  to let  puppet  run
* 21:33 hashar: rebooting deployment-jobrunner01  / kernel upgrade /  /tmp is only 1MBytes
* 21:19 hashar: fixing up deployment-jobrunner01  /tmp and / disks are full
* 19:57 thcipriani: ran REPAIR TABLE globalnames; on centralauth db
* 19:48 legoktm: deploying https://gerrit.wikimedia.org/r/265552
* 19:39 legoktm: deploying jjb changes for https://gerrit.wikimedia.org/r/264990
* 19:25 legoktm: deploying https://gerrit.wikimedia.org/r/265546
* 01:59 jzerebecki: jenkins-deploy@deployment-bastion:/srv/mediawiki-staging/php-master/extensions/SpellingDictionary$ rm -r modules/jquery.uls && git rm modules/jquery.uls
* 01:00 jzerebecki: jenkins-deploy@deployment-bastion:/srv/mediawiki-staging/php-master/extensions$ git pull && git submodule update --init --recursive
* 00:57 jzerebecki: jenkins-deploy@deployment-bastion:/srv/mediawiki-staging/php-master/extensions$ git reset HEAD SpellingDictionary


== 2016-01-20 ==
== 2023-04-06 ==
* 20:05 hashar: beta sudo find /data/project/upload7/math -type f -delete  (probably some old left over)
* 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list
* 19:50 hashar: beta: on commons ran deleteArchivedFile.php : Nuked 7130 files
* 19:49 hashar: beta : foreachwiki deleteArchivedRevisions.php -delete
* 19:26 hasharAway: Nuked all files from http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload
* 19:19 hasharAway: beta: sudo find /data/project/upload7/*/*/temp -type f -delete
* 19:14 hasharAway: beta: sudo rm /data/project/upload7/*/*/lockdir/*
* 18:57 hasharAway: beta cluster code has been stalled for roughly 2h30
* 18:55 hasharAway: disconnecting Gearman plugin to remove deadlock for beta cluster rjobs
* 17:06 hashar: clearing files from beta-cluster to prepare for Swift migration.  python pwb.py delete.py -family:betacommons -lang:en -cat:'GWToolset Batch Upload' -verbose -putthrottle:0 -summary:'Clearing out old batched upload to save up disk space for Swift migration'


== 2016-01-19 ==
== 2023-04-05 ==
* 22:25 legoktm: deleting *zend* workspaces on precise slaves
* 18:05 James_F: Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep, for [[phab:T333661|T333661]]
* 21:58 thcipriani: trying https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update again
* 18:05 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production, for [[phab:T331718|T331718]]
* 21:57 thcipriani: beta-scap-eqiad still can't find executor on deployment-bastion.eqiad
* 09:44 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905664
* 21:52 thcipriani: following steps at https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update for deployment-bastion
* 19:34 legoktm: deleting all *zend* jobs from jenkins
* 09:40 hashar: Created github repo https://github.com/wikimedia/operations-debs-varnish4
* 03:59 legoktm: deploying https://gerrit.wikimedia.org/r/264912 and https://gerrit.wikimedia.org/r/264922


== 2016-01-17 ==
== 2023-04-04 ==
* 18:02 legoktm: deploying https://gerrit.wikimedia.org/r/264605
* 05:04 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905262


== 2016-01-16 ==
== 2023-04-03 ==
* 21:47 legoktm: deploying https://gerrit.wikimedia.org/r/264489
* 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
* 21:36 legoktm: deploying https://gerrit.wikimedia.org/r/264488
* 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 {{!}} https://gerrit.wikimedia.org/r/905190
* 21:29 legoktm: deploying https://gerrit.wikimedia.org/r/264487
* 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
* 21:21 legoktm: deploying https://gerrit.wikimedia.org/r/264483 https://gerrit.wikimedia.org/r/264485
* 10:20 hashar: Tag Quibble 1.5.4 @ {{Gerrit|0a37ce9367672e973b49b8293f57ebffe40abdf6}} # [[phab:T332930|T332930]] [[phab:T331280|T331280]] [[phab:T333412|T333412]]
* 20:58 legoktm: deploying https://gerrit.wikimedia.org/r/264492
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T333586|T333586]]
* 18:55 jzerebecki: reloadin zuul for 996c558..5f8eb50
* 08:11 hashar: gtlab: runner-1030: running `docker volume prune`  to discard all volumes metadata after they have been manually deleted from `/var/lib/docker`  ( Reclaimed 13.92GB) {{!}} [[phab:T33586|T33586]]
* 09:12 legoktm: deploying https://gerrit.wikimedia.org/r/264448
* 09:01 legoktm: deploying https://gerrit.wikimedia.org/r/264446 and https://gerrit.wikimedia.org/r/264447
* 07:46 legoktm: sudo -u jenkins-deploy mv /mnt/jenkins-workspace/workspace/mediawiki-core-phplint /mnt/jenkins-workspace/workspace/mediawiki-core-php53lint on all precise slaves
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/264444
* 06:31 legoktm: deploying https://gerrit.wikimedia.org/r/264441
* 06:10 legoktm: added phpflavor-php53 label to all phpflavor-zend slaves


== 2016-01-15 ==
== 2023-03-31 ==
* 12:17 hashar: restarting Jenkins for plugins updates
* 16:24 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904818
* 02:49 bd808: Trying to fix submodules in deployment-bastion:/srv/mediawiki-staging/php-master/extensions for T123701
* 16:20 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904810
* 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/904805/
* 14:34 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 14:26 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]


== 2016-01-14 ==
== 2023-03-29 ==
* 20:06 legoktm: deploying https://gerrit.wikimedia.org/r/264122
* 06:09 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/903839
* 19:32 legoktm: deploying https://gerrit.wikimedia.org/r/264114
* 00:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/903539
* 19:18 legoktm: deploying https://gerrit.wikimedia.org/r/264108


== 2016-01-13 ==
== 2023-03-23 ==
* 21:06 hashar: beta cluster code is up to date again. Got delayed by roughly 4 hours.
* 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.
* 20:55 hashar: unlocked Jenkins jobs for beta cluster by disabling/reenabling  Jenkins Gearman client
* 10:15 hashar: beta: fixed puppet on deployment-elastic06 . Was still using cert/hostname without .deployment-prep. .... Mass update occurring.


== 2016-01-12 ==
== 2023-03-22 ==
* 23:30 legoktm: deploying https://gerrit.wikimedia.org/r/263757 https://gerrit.wikimedia.org/r/263756
* 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # [[phab:T242031|T242031]]
* 13:32 hashar: beta cluster: running /usr/local/sbin/cleanup-pam-config
* 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql {{!}} sql centralauth --write # [[phab:T242031|T242031]]
* 13:29 hashar: integration running /usr/local/sbin/cleanup-pam-config  on slaves


== 2016-01-11 ==
== 2023-03-20 ==
* 22:24 hashar: Deleting old references on Zuul-merger for mediawiki/core : <tt>/usr/share/python/zuul/bin/python /home/hashar/zuul-clear-refs.py --until 15 /srv/ssd/zuul/git/mediawiki/core </tt>
* 21:43 brennen: phabricator: setting sbassett to administrator per [[phab:T331928|T331928]]
* 22:21 hashar: gallium in /srv/ssd/zuul/git/mediawiki/core$  git gc --prune=all && git remote update --prune
* 22:21 hashar: scandium  in /srv/ssd/zuul/git/mediawiki/core$  git gc --prune=all && git remote update --prune
* 07:35 legoktm: deploying https://gerrit.wikimedia.org/r/263319


== 2016-01-07 ==
== 2023-03-19 ==
* 23:16 legoktm: deleted /mnt/jenkins-workspace/workspace/mediawiki-extensions-qunit/src/extensions/PdfHandler/.git/refs/heads/wmf/1.26wmf16.lock on slave 1013
* 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in [[phab:T332507|T332507]]
* 06:32 legoktm: deploying https://gerrit.wikimedia.org/r/262868
* 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` [[phab:T322469|T322469]]
* 02:24 legoktm: deploying https://gerrit.wikimedia.org/r/262855
* 14:43 hashar: deployment-prep: databases update hourly job times out, filed [[phab:T332507|T332507]]
* 01:25 jzerebecki: reloading zuul for b0a5335..c16368a


== 2016-01-06 ==
== 2023-03-17 ==
* 21:13 thcipriani: kicking integration puppetmaster, weird node unable to find definition.
* 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 ([[phab:T332455|T332455]])
* 21:11 jzerebecki: on scandium: sudo -u zuul rm -rf /srv/ssd/zuul/git/mediawiki/services/mathoid
* 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 ([[phab:T332455|T332455]])
* 21:04 legoktm: ^ on gallium
* 10:36 mutante: disabled phab user 'Cloudduck1233'
* 21:04 legoktm: manually deleted /srv/ssd/zuul/git/mediawiki/services/mathoid to force zuul to re-clone it
* 20:17 hashar: beta: dropped a few more /etc/apt/apt.conf.d/*-proxy files.  webproxy is no more reachable from labs
* 09:44 hashar: CI/beta: deleting all git tags from /var/lib/git/operations/puppet and doing git repack
* 09:39 hashar: restoring puppet hacks on beta cluster puppetmaster.
* 09:35 hashar: beta/CI:  salt -v '*' cmd.run 'rm -v /etc/apt/apt.conf.d/*-proxy' https://phabricator.wikimedia.org/T122953


== 2016-01-05 ==
== 2023-03-16 ==
* 16:54 hashar_: Removed elastic search from CI slaves https://phabricator.wikimedia.org/T89083 https://gerrit.wikimedia.org/r/#/c/259301/
* 10:48 hashar: Updating all jobs to switch Castor from Stretch to Bullseye based image {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/899508/ {{!}} [[phab:T278203|T278203]]
* 03:45 Krinkle: integration-slave-trusty-1015: rm -rf /mnt/home/jenkins-deploy/.npm per https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/56577/console
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based) [[phab:T278203|T278203]]
* 10:41 hashar: Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based)
* 08:20 hashar: github: created https://github.com/wikimedia/mediawiki-services-machinetranslation for replication from Gerrit # [[phab:T331256|T331256]]


== 2016-01-04 ==
== 2023-03-15 ==
* 21:06 hashar: gallium has puppet enabled again
* 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T327708|T327708]]
* 20:53 hashar: stopping puppet on gallium and live hacking Zuul configuration for https://phabricator.wikimedia.org/T122656
* 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
* 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/898807 {{!}} [[phab:T331061|T331061]], [[phab:T331621|T331621]]
* 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034


== 2016-01-02 ==
== 2023-03-14 ==
* 03:17 yurik: purged varnishs on deployment-cache-text04
* 16:54 hashar: Building Docker images for Quibble 1.5.3
* 16:41 hashar: Tag Quibble 1.5.3 @ {{Gerrit|df49e8a04f5}} # [[phab:T331621|T331621]]  [[phab:T331061|T331061]]
* 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
* 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 02:44 James_F: MediaWiki being branched to REL1_40 from {{Gerrit|fd3ce7633984d0120288d4a6fe48343cf9ba86a2}} for [[phab:T329079|T329079]]


== 2016-01-01 ==
== 2023-03-13 ==
* 22:17 bd808: No nodepool ci-jessie-* hosts seen in Jenkins interface and rake-jessie jobs backing up
* 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging [[phab:T326192|T326192]]
* 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging [[phab:T326192|T326192]]
* 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior


== Archive ==
== 2023-03-12 ==
* [[/Archive 1|Archive 1]] (September 2014 - December 2015)
* 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
* 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
* 22:57 James_F: Zuul: Add Want to the CI whitelist
* 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 [[phab:T328240|T328240]]
 
== 2023-03-11 ==
* 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
 
== 2023-03-10 ==
* 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
* 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
* 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]
* 14:37 TheresNoTime: (deployment-prep) clear lock
* 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, [[phab:T324233|T324233]]`
 
== 2023-03-09 ==
* 20:53 brennen: scheduled tentative phabricator update window for tuesday 2023-03-14 @ 21:00 UTC
* 11:37 hashar: Rolling back selenium-daily Jenkins jobs back to NodeJS 14 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/896026/ {{!}} [[phab:T330426|T330426]]
* 10:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895839
 
== 2023-03-08 ==
* 20:12 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/893074
* 10:36 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895698 {{!}} [[phab:T331505|T331505]]
* 06:26 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895142
* 06:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895329 and https://gerrit.wikimedia.org/r/c/integration/config/+/895333 {{!}} [[phab:T321837|T321837]]
 
== 2023-03-07 ==
* 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, [[phab:T326163|T326163]]
* 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for [[phab:T326163|T326163]]
* 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893765
 
== 2023-03-06 ==
* 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~`  [[phab:T327853|T327853]]
* 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small {{!}} [[phab:T327853|T327853]]
* 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
* 12:24 hashar: Building Docker images for Quibble 1.5.2 {{!}} https://gerrit.wikimedia.org/r/894617
* 12:15 hashar: Tag Quibble 1.5.2 @ {{Gerrit|932b48cab06c}} # [[phab:T331236|T331236]] [[phab:T331061|T331061]] [[phab:T323750|T323750]]
 
== 2023-03-05 ==
* 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons ([[phab:T331214|T331214]]#8666432)
 
== 2023-03-03 ==
* 19:52 zabe: deployment-db13: start replication from deployment-db11 # [[phab:T331019|T331019]]
* 19:52 zabe: deployment-db13: import dump into mariadb # [[phab:T331019|T331019]]
* 08:56 hashar: Updating Jenkins jobs with JJB 4.0.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/893667/
* 00:37 zabe: take deployment-prep out of read-only # [[phab:T331019|T331019]]
 
== 2023-03-02 ==
* 22:01 zabe: enable read-only mode and create dump of all databases # [[phab:T331019|T331019]]
* 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T331019|T331019]]
* 16:35 zabe: create volume db13 and attach to deployment-db13 # [[phab:T331019|T331019]]
* 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # [[phab:T331019|T331019]]
* 16:24 zabe: shutoff deployment-db09 # [[phab:T331019|T331019]]
* 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # [[phab:T331019|T331019]]
* 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
* 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was {{Gerrit|ed287d58a1e9a2ea39f2eb5bbba31d8fc54269c2}} ) # [[phab:T330752|T330752]]
* 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # [[phab:T330752|T330752]]
* 12:13 zabe: delete deploymentwiki # [[phab:T198673|T198673]]
 
== 2023-03-01 ==
* 19:30 hashar: Building Docker images for Quibble 1.5.1 ( https://gerrit.wikimedia.org/r/c/integration/config/+/893528/ )
* 19:17 hashar: Tagged Quibble 1.5.1 @ {{Gerrit|9a163094a855}} # [[phab:T330750|T330750]]
* 18:30 brennen: gitlab: enable custom issue-tracker instance-wide
* 17:37 hashar: Deploying `device-analytics-*` jobs for for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 16:32 taavi: deployment-prep: block automatic/update.php execution of UpdateForMultipleDevicesSupport maintenance script in OATHAuth by pre-inserting rows in updatelog tables, will be run manually instead. [[phab:T242031|T242031]]
* 15:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 {{!}} [[phab:T320967|T320967]]
* 14:07 hashar: integration: removed krenair from Puppet config `passwords::root::extra_keys`
 
== 2023-02-28 ==
* 15:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892548/
* 14:43 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892993 {{!}} [[phab:T271418|T271418]]
* 14:08 hashar: Updating Quibble jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/891742 (node 14 shipped from the node14 images instead of being manually installed at build time)
* 13:56 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892546 {{!}} [[phab:T311462|T311462]]
* 13:06 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892952 {{!}} [[phab:T296348|T296348]]
* 12:11 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892919 {{!}} [[phab:T299263|T299263]]
* 10:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892910/ {{!}} [[phab:T299262|T299262]]
* 09:16 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892530 {{!}} [[phab:T299158|T299158]] {{!}} [[phab:T299161|T299161]]
 
== 2023-02-27 ==
* 15:08 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892458 {{!}} [[phab:T299283|T299283]]
* 13:02 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/892435 {{!}} [[phab:T327583|T327583]]
* 12:50 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891968
 
== 2023-02-25 ==
* 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
* 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)
 
== 2023-02-24 ==
* 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- [[phab:T289883|T289883]]
* 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)
 
== 2023-02-23 ==
* 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
* 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
* 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
* 14:00 taavi: update quibble* jenkins jobs for 891303
* 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config
 
== 2023-02-22 ==
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/891281 [[phab:T328240|T328240]]
* 14:19 hashar: Updating all jobs to add `QUIBBLE_API_KEY` to the Quibble jobs {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/891278/ {{!}} [[phab:T323750|T323750]]
* 02:48 James_F: Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 for [[phab:T314470|T314470]]
* 02:45 James_F: Zuul: Migrate all MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]
* 02:43 James_F: Zuul: Migrate all non-MediaWiki-land node jobs to node16 for [[phab:T314470|T314470]]
 
== 2023-02-21 ==
* 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
* 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs [[phab:T329034|T329034]]
* 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
* 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
* 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation {{!}} [[phab:T321837|T321837]]
 
== 2023-02-20 ==
* 00:21 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/890197
 
== 2023-02-16 ==
* 10:31 jnuche: restarted Zuul to clear stale events
 
== 2023-02-15 ==
* 20:07 taavi: unbroke puppet on deployment-ms-*
* 18:12 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/889499
* 16:03 thcipriani: restart ci jenkins for updates
* 16:02 thcipriani: restart releases jenkins for updates
 
== 2023-02-14 ==
* 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er [[phab:T329000|T329000]]
* 15:02 zabe: deployment-db12: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 14:04 zabe: delete deployment-db10 and volume db10 # [[phab:T329577|T329577]]
* 11:38 zabe: deployment-db11: start replicating from deployment-db09 # [[phab:T329577|T329577]]
* 02:19 zabe: deployment-db12: import dump into mariadb # [[phab:T329577|T329577]]
* 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # [[phab:T329577|T329577]]
* 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # [[phab:T329577|T329577]]
* 01:50 zabe: create volume db12 and attach to deployment-db12 # [[phab:T329577|T329577]]
* 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 01:14 zabe: deployment-db11: import dump into mariadb # [[phab:T329577|T329577]]
* 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # [[phab:T329577|T329577]]
* 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # [[phab:T329577|T329577]]
 
== 2023-02-13 ==
* 23:48 zabe: create volume db11 and attach to deployment-db11 # [[phab:T329577|T329577]]
* 23:44 zabe: shutoff deployment-db10 # [[phab:T329577|T329577]]
* 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # [[phab:T329577|T329577]]
* 23:20 zabe: created backup of all databases on deployment-db09 # [[phab:T329577|T329577]]
* 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
* 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
 
== 2023-02-10 ==
* 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for [[phab:T277440|T277440]]
* 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for [[phab:T324732|T324732]]
 
== 2023-02-08 ==
* 23:17 James_F: Zuul: Add Nik.xyz.in e-mail to CI allow list
* 21:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/887829 # [[phab:T329231|T329231]]
* 18:11 Krinkle: Disable spam account https://phabricator.wikimedia.org/people/manage/35991/
* 12:37 James_F: Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job
* 09:16 hashar: deployment-prep:  deployment-deploy03 `systemctl stop ferm`  to allow jenkins to connect to the instance # [[phab:T329056|T329056]]
 
== 2023-02-07 ==
* 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for [[phab:T329077|T329077]]
 
== 2023-02-06 ==
* 09:30 hashar: Built tox Docker images to include `bats` {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/871178/
 
== 2023-02-03 ==
* 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old
 
== 2023-02-02 ==
* 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for [[phab:T321309|T321309]]
* 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete  "' research/navigation-vectors'"`
 
== 2023-01-31 ==
* 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
* 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
* 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
* 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for [[phab:T326372|T326372]]
 
== 2023-01-30 ==
* 23:23 James_F: Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' for [[phab:T327332|T327332]]
* 18:35 James_F: Zuul: [mediawiki/extensions/PushAll] Add basic tests
* 14:39 hashar: Building Docker image for https://gerrit.wikimedia.org/r/884904
* 10:59 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/884835/
 
== 2023-01-27 ==
* 20:42 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/884372
* 19:35 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/884365/
* 14:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/884143 {{!}} [[phab:T315621|T315621]]
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884309
* 14:13 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/884283
* 08:39 hashar: devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master {{!}} [[phab:T325128|T325128]]
 
== 2023-01-25 ==
* 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner.  [[phab:T327949|T327949]]
* 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for [[phab:T327949|T327949]]
 
== 2023-01-24 ==
* 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [[phab:T326372|T326372]]
* 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for [[phab:T326372|T326372]]
* 17:19 thcipriani: restarting ci jenkins for updates
* 17:15 thcipriani: restarted releases-jenkins for plugin updates
 
== 2023-01-23 ==
* 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
* 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for [[phab:T292818|T292818]]
 
== 2023-01-20 ==
* 06:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881665 {{!}} [[phab:T327301|T327301]]
 
== 2023-01-19 ==
* 20:01 James_F: Zuul: Add Izno to the CI allowlist
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # [[phab:T299954|T299954]]
* 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
* 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
* 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
* 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907
 
== 2023-01-18 ==
* 21:33 dancy: Rebooting deployment-jobrunner04 for [[phab:T327329|T327329]]
* 17:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/881457 {{!}} [[phab:T327301|T327301]]
* 15:42 thcipriani: deployment-jobrunner04: sudo rm -rf /srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~
* 15:31 andrewbogott: shutting down deployment-imagescaler03 as it is long-overdue for replacement.  See [[phab:T294148|T294148]] for details.
* 10:10 arturo: bump trove quotas ([[phab:T326674|T326674]])
 
== 2023-01-17 ==
* 12:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880951/
* 10:05 hashar: Updating sonar-scanner jobs https://gerrit.wikimedia.org/r/c/integration/config/+/880893/
* 09:52 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-4
* 08:32 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/880503 {{!}} [[phab:T321837|T321837]]
* 00:21 James_F: Zuul: [operations/debs/cadvisor] Add debian-glue-unstable CI for [[phab:T325557|T325557]]
* 00:21 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] +AbuseFilter phan dep
 
== 2023-01-16 ==
* 22:29 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/880541
* 00:19 Krinkle: Reloading Zuul
 
== 2023-01-13 ==
* 10:41 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/879754
* 01:35 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879687
 
== 2023-01-12 ==
* 21:49 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879649
* 10:42 hashar: Deployed docroot change https://gerrit.wikimedia.org/r/878860 "zuul: Link to report_url if available"
 
== 2023-01-10 ==
* 21:14 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/27
* 16:50 James_F: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf for [[phab:T326653|T326653]]
* 15:28 James_F: Zuul: Add experimental jobs for fundraising/REL1_39 for [[phab:T326372|T326372]]
 
== 2023-01-09 ==
* 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
* 16:30 wm-bot2: Increased quotas by 6 cores ([[phab:T326568|T326568]]) - cookbook ran by arturo@nostromo
* 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests
 
== 2023-01-08 ==
* 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
* 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix
 
== 2023-01-07 ==
* 22:53 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876327
 
== 2023-01-06 ==
* 16:59 James_F: Docker: Publishing new CI images with composer 2.5.1 for [[phab:T319257|T319257]]
* 11:43 hashar: Refreshing Jenkins jobs to update Quibble jobs with php-excimer {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/876190
 
== 2023-01-05 ==
* 09:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868193 {{!}} Make CentralAuth a clone dependency for MassMessage {{!}} [[phab:T145430|T145430]]
 
== 2023-01-04 ==
* 19:42 MatmaRex: Ran maintenansce script refreshGlobalimagelinks.php for [[phab:T322588|T322588]]
* 02:47 TimStarling: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/874959
 
== 2023-01-03 ==
* 19:14 dduvall: restarting zuul
* 19:12 Krinkle: Disable publishing of rPMSM, ref [[phab:T143162|T143162]], [[phab:T326112|T326112]]
* 17:38 hashar: gerrit: reactivated account for `jmando` using `gerrit set-account 9431 --active` {{!}} [[phab:T326159|T326159]]
* 14:20 hashar: integration: restarting all instances for Linux kernel upgrade
* 07:52 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # [[phab:T322368|T322368]]
 
 
{{SAL-archives/Release Engineering}}


__NOTOC__
<noinclude>[[Category:SAL]]</noinclude>
<noinclude>[[Category:SAL]]</noinclude>

Latest revision as of 15:12, 26 May 2023

2023-05-26

2023-05-25

  • 17:30 hashar: gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # T334521
  • 16:44 hashar: gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` # T334521
  • 16:37 hashar: ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait # T334521
  • 16:27 brennen: phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd

2023-05-24

  • 10:09 eoghan: Switching doc.wikimedia.org from doc1002 -> doc1003, T319477
  • 09:24 Daimona: Creating new DB table on beta wikishared for the CampaignEvents extension # T336362

2023-05-23

  • 21:48 brennen: https://phab.wmflabs.org/ now running phorge 2023.17
  • 13:54 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=metawiki Diskdance [snip]` T337172

2023-05-21

2023-05-17

  • 09:53 hashar: integration-agent-docker-1029: erased a few files under `/srv`. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # T336840
  • 03:12 brennen: gitlab-webhooks restart after failed webhook requests (presumably earlier outage)

2023-05-16

2023-05-12

2023-05-11

2023-05-10

  • 23:19 brennen: reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot (T327424)

2023-05-09

2023-05-05

2023-05-04

2023-05-03

2023-05-02

2023-05-01

  • 17:20 brennen: experimenting with ldap user sync on gitlab.devtools

2023-04-28

  • 19:56 dancy: Ran `git submodule sync` as jenkins-deploy in deployment-deploy03:/srv/mediawiki-staging/php-master/extensions for T335602

2023-04-27

  • 17:10 James_F: Zuul: [wvui] Mark repo as archived for T310244

2023-04-26

  • 16:51 James_F: Zuul: Add Kevin Wilke to CI allow list

2023-04-25

  • 21:23 mutante: gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (manually it works, but that's the same command that puppet is supposed to run !?) - T257317 T326368

2023-04-24

  • 21:37 TheresNoTime: (deployment-prep) Add volume to `deployment-mwmaint02` (`mwmaint`) and moved `/srv` - T327853

2023-04-23

  • 16:55 Krinkle: Fix profile::tlsproxy::envoy::global_cert_name in Horizon for webperf host to use '%{facts.fqdn}' instead of performance.discovery.wmnet as the latter doesn't resolve / would be an invalid cert for https://deployment-webperf21, ref T291015

2023-04-22

2023-04-21

  • 10:04 Lucas_WMDE: `sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # T335181

2023-04-20

2023-04-19

  • 13:59 James_F: Beta: Setting `namespace: repos/abstract-wiki/wikifunctions image_name: function-orchestrator` on deployment-docker-wikifunctions01 for T334998
  • 10:56 joe: Updating docker-pkg files on contint primary for helm-linter

2023-04-18

  • 17:35 mutante: doc2001 - systemctl reset-failed - cleared monitoring for systemd state - remnants from PHP7.3 after upgrade to 7.4
  • 17:08 James_F: Zuul, jjb: [mediawiki/services/function-orchestrator] Archive, moved to GitLab for T334397

2023-04-17

  • 13:37 James_F: Zuul: [mediawiki/services/function-schemata] Mark as archived for T281500

2023-04-12

2023-04-10

  • 20:00 thcipriani: restarting zuul due to due to stuck gerrit ssh-connections
  • 18:44 thcipriani: restart zuul-merger

2023-04-09

  • 13:22 lucaswerkmeister: lucaswerkmeister@deployment-mwmaint02:~$ mwscript createAndPromote metawiki --custom-groups=oauthadmin --force 'Lucas Werkmeister' # made myself OAuth admin so I can manage the clients used to test m3api-oauth2

2023-04-06

  • 19:49 James_F: Zuul: Add all @wikia/fandom domains to the CI allow list

2023-04-05

2023-04-04

2023-04-03

  • 14:14 James_F: Zuul: [mediawiki/extensions/ContentStabilization] Add BlueSpiceBookshelf dep
  • 11:10 hashar: Switching Jenkins jobs to Quibble 1.5.4 | https://gerrit.wikimedia.org/r/905190
  • 10:25 hashar: Building Docker images for Quibble 1.5.4 https://gerrit.wikimedia.org/r/905167
  • 10:20 hashar: Tag Quibble 1.5.4 @ 0a37ce9 # T332930 T331280 T333412
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T333586
  • 08:11 hashar: gtlab: runner-1030: running `docker volume prune` to discard all volumes metadata after they have been manually deleted from `/var/lib/docker` ( Reclaimed 13.92GB) | T33586

2023-03-31

2023-03-29

2023-03-23

  • 13:22 James_F: Gerrit: Deleting mediawiki/extensions/PageProperties duplicate 'main' branch, development work is in 'master'.

2023-03-22

  • 21:14 taavi: taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # T242031
  • 20:34 taavi: cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql | sql centralauth --write # T242031

2023-03-20

  • 21:43 brennen: phabricator: setting sbassett to administrator per T331928

2023-03-19

  • 22:34 TheresNoTime: (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in T332507
  • 22:10 TheresNoTime: (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` T322469
  • 14:43 hashar: deployment-prep: databases update hourly job times out, filed T332507

2023-03-17

  • 23:00 urbanecm: deployment-prep: Reboot deployment-elastic10, deployment-elastic11 (T332455)
  • 22:58 urbanecm: deployment-prep: Reboot deployment-elastic09 (T332455)
  • 10:36 mutante: disabled phab user 'Cloudduck1233'

2023-03-16

2023-03-15

  • 09:01 dwalden: (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T327708
  • 08:09 hashar: Refreshing all Jenkins jobs to catch up with various changes made in the last few days
  • 08:01 hashar: Switching Jenkins jobs to Quibble 1.5.3 | https://gerrit.wikimedia.org/r/c/integration/config/+/898807 | T331061, T331621
  • 07:50 hashar: Cleared Jenkins workspaces on integration-agent-docker-1034

2023-03-14

  • 16:54 hashar: Building Docker images for Quibble 1.5.3
  • 16:41 hashar: Tag Quibble 1.5.3 @ df49e8a04f5 # T331621 T331061
  • 16:00 hashar: updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787
  • 07:25 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for T326163
  • 07:24 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for T326163
  • 07:17 dwalden: `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 02:44 James_F: MediaWiki being branched to REL1_40 from fd3ce76 for T329079

2023-03-13

  • 17:57 James_F: Moved deployment-docker-cpjobqueue01 from v0.9.5 to v0.10.5 of change-prop whilst debugging T326192
  • 17:44 James_F: Manually changed cloudmetrics1002 to cloudmetrics1003 on deployment-docker-cpjobqueue01 whilst debugging T326192
  • 17:41 brennen: gitlab: removing dzahn (mutante) as member of /repos to test notification behavior

2023-03-12

  • 23:01 James_F: Zuul: [mediawiki/extensions/Tabber] Fix archived extension
  • 23:01 James_F: Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions
  • 22:57 James_F: Zuul: Add Want to the CI whitelist
  • 10:23 taavi: reload zuul for https://gerrit.wikimedia.org/r/897068 T328240

2023-03-11

  • 01:22 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-10

  • 21:08 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431
  • 16:37 TheresNoTime: `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`
  • 15:07 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233
  • 14:37 TheresNoTime: (deployment-prep) clear lock
  • 14:28 TheresNoTime: (deployment-prep) `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233`

2023-03-09

2023-03-08

2023-03-07

  • 17:52 TheresNoTime: (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, T326163
  • 17:48 TheresNoTime: (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163
  • 08:17 hashar: Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/893765

2023-03-06

  • 13:48 hashar: Freed 3G of disk on deployment-mwmaint02 by deleting `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~` T327853
  • 13:44 hashar: Beta cluster no more updates due to deployment-mwmaint02 having a disk too small | T327853
  • 12:42 hashar: Updating Jenkins jobs to Quibble 1.5.2
  • 12:24 hashar: Building Docker images for Quibble 1.5.2 | https://gerrit.wikimedia.org/r/894617
  • 12:15 hashar: Tag Quibble 1.5.2 @ 932b48cab06c # T331236 T331061 T323750

2023-03-05

  • 14:09 lucaswerkmeister: manually deleted mediainfo slots of two pages on beta commons (T331214#8666432)

2023-03-03

2023-03-02

  • 22:01 zabe: enable read-only mode and create dump of all databases # T331019
  • 21:54 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T331019
  • 16:35 zabe: create volume db13 and attach to deployment-db13 # T331019
  • 16:29 zabe: create deployment-db13 as g3.cores8.ram16.disk20 # T331019
  • 16:24 zabe: shutoff deployment-db09 # T331019
  • 16:13 zabe: failover deployment-prep master from deployment-db09 to deployment-db11 # T331019
  • 14:00 hashar: gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)
  • 12:41 Lucas_WMDE: wikidata/query-builder/deploy.git: deleted master branch (was ed287d5 ) # T330752
  • 12:39 Luca: wikidata/query-builder/deploy.git: changed HEAD from master to production # T330752
  • 12:13 zabe: delete deploymentwiki # T198673

2023-03-01

2023-02-28

2023-02-27

2023-02-25

  • 22:57 James_F: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage)
  • 22:56 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen)

2023-02-24

  • 15:26 andrewbogott: deleting long-shutoff stretch instance deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud -- T289883
  • 01:49 James_F: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen)

2023-02-23

  • 22:15 James_F: Docker: [quibble-buster] Re-use node from the node image rather than our own
  • 21:13 James_F: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1
  • 14:11 hashar: Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter
  • 14:00 taavi: update quibble* jenkins jobs for 891303
  • 07:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config

2023-02-22

2023-02-21

  • 18:55 James_F: Zuul: [mediawiki/extensions/Flow] Also run with BetaFeatures
  • 16:04 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs T329034
  • 15:07 James_F: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage)
  • 13:50 hashar: Updating docker-pkg files for https://gerrit.wikimedia.org/r/890817
  • 11:30 hashar: Reloaded Zuul to enable Sonar Codehealth Checks on CheckUser, DiscussionTools and ContentTranslation | T321837

2023-02-20

2023-02-16

  • 10:31 jnuche: restarted Zuul to clear stale events

2023-02-15

2023-02-14

  • 16:23 taavi: make Dreamy Jazz a mediawiki/* +2'er T329000
  • 15:02 zabe: deployment-db12: start replicating from deployment-db09 # T329577
  • 14:04 zabe: delete deployment-db10 and volume db10 # T329577
  • 11:38 zabe: deployment-db11: start replicating from deployment-db09 # T329577
  • 02:19 zabe: deployment-db12: import dump into mariadb # T329577
  • 02:16 zabe: rsync dump from deployment-db09 to deployment-db12 # T329577
  • 02:09 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T329577
  • 01:50 zabe: create volume db12 and attach to deployment-db12 # T329577
  • 01:38 zabe: create deployment-db12 as g3.cores8.ram16.disk20 # T329577
  • 01:14 zabe: deployment-db11: import dump into mariadb # T329577
  • 01:06 zabe: rsync dump from deployment-db09 to deployment-db11 # T329577
  • 00:22 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db11 # T329577

2023-02-13

  • 23:48 zabe: create volume db11 and attach to deployment-db11 # T329577
  • 23:44 zabe: shutoff deployment-db10 # T329577
  • 23:35 zabe: create deployment-db11 as g3.cores8.ram16.disk20 # T329577
  • 23:20 zabe: created backup of all databases on deployment-db09 # T329577
  • 22:29 dancy: Disabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
  • 19:57 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm

2023-02-10

  • 21:52 James_F: Zuul: [operations/software/homer/deploy] Add CI for T277440
  • 03:21 James_F: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for T324732

2023-02-08

2023-02-07

  • 19:13 James_F: Zuul: Configure support for forthcoming REL1_40 branches for T329077

2023-02-06

2023-02-03

  • 03:35 Krinkle: Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old

2023-02-02

  • 15:53 James_F: Zuul: [operations/debs/gdnsd] Add debian-glue CI for T321309
  • 12:19 hashar: gerrit: deleted erroneous project which had a leading space ' research/navigation-vectors'. Done over ssh with: `ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete "' research/navigation-vectors'"`

2023-01-31

  • 20:30 brennen: deploy1002:/srv/deployment/phabricator/deployment: updating previous phabricator remotes to gitlab
  • 18:21 James_F: Zuul: [wikimedia/fundraising/crm] Make composer php74 voting via teamplate
  • 18:11 James_F: Zuul: [wikimedia/fundraising/crm] Switch to generic composer job
  • 18:06 James_F: Zuul: Switch FR things on mwgate-composer-php73-docker to php74 for T326372

2023-01-30

2023-01-27

2023-01-25

  • 21:35 dancy: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner. T327949
  • 19:37 dancy: Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for T327949

2023-01-24

  • 22:08 James_F: Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now T326372
  • 22:04 James_F: Zuul: Make fundraising/REL1_39 jobs voting for T326372
  • 17:19 thcipriani: restarting ci jenkins for updates
  • 17:15 thcipriani: restarted releases-jenkins for plugin updates

2023-01-23

  • 20:04 hashar: gerrit: changed mediawiki/tools/code-utils merge strategy from `fast-forward` only to `Rebase if necessary` *
  • 15:29 James_F: Docker: Building new version of helm-linter image with sextant installed for T292818

2023-01-20

2023-01-19

  • 20:01 James_F: Zuul: Add Izno to the CI allowlist
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php # T299954
  • 19:10 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateRevisionCommentTemp.php
  • 19:10 taavi: deployment-prep: pruned bunch of stale puppet certificates and puppetdb entries
  • 19:02 andrewbogott: deleting long-shutdown stretch instances: deployment-echostore01, deployment-ms-fe03, deployment-prometheus02
  • 17:42 jeena: restarting zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/879907

2023-01-18

2023-01-17

2023-01-16

2023-01-13

2023-01-12

2023-01-10

2023-01-09

  • 19:56 James_F: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches
  • 16:30 wm-bot2: Increased quotas by 6 cores (T326568) - cookbook ran by arturo@nostromo
  • 10:34 hashar: Restarted zuul-merger on contint1002 it was listening but not processing any requests

2023-01-08

  • 19:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/876355
  • 00:21 Krinkle: Mark various Gerrit repos under mediawiki/tools/ as read-only "ARCHIVED" that are 6y+ inactive, in prep for enabling Codesearch for this prefix

2023-01-07

2023-01-06

2023-01-05

2023-01-04

2023-01-03


Archives

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=601311"