You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Joal
(→‎How to deploy with Jenkins: Add build as a step for updating release parameters.)
imported>Ottomata
Line 23: Line 23:
===== If the maven jar upgrade job failed (step 5) =====
===== If the maven jar upgrade job failed (step 5) =====


= How to deploy =
= How to deploy from the CLI =


{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}
{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}

Revision as of 14:10, 8 September 2016

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins

  1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
  2. Make sure you are logged in to Jenkins to see the pages linked in the steps below.
  3. Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/. Everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form. If pre-filled values are not correct, the simplest solution is to trigger a build (instead of a release) for the project: https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec - It should update jenkins workspace with up-to-date values.
  4. Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left Build History panel (In case the last job fails, it will have a red dot before its build number - see below).
  5. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
If the maven release job failed (step 4)

First find why looking at job logs, and then two solutions:

  • Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
    • Clean maven commits:
      • git reset --hard YOUR_LAST_COMMIT_HASH
      • git push --force origin master
    • Clean maven tag:
      • git tag -d YOUR_VERSION_TAG
      • git push origin :refs/tags/YOUR_VERSION_TAG
  • Or, skip the version number you were currently deploying
    • Update the changelog,md with the skipped version info
    • Deploy again !
If the maven jar upgrade job failed (step 5)

How to deploy from the CLI

  • Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
  • Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
  • Check everything looks Ok:
cat release.properties
  • Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

Please see the refinery page to deploy the jars and oozie code.