You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Analytics/Cluster/Deploy/Refinery-source

From Wikitech-static
< Analytics‎ | Cluster
Revision as of 13:43, 5 December 2016 by imported>Joal (→‎How to deploy with Jenkins (and related steps): Add release and development version precisions)
Jump to navigation Jump to search

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins (and related steps)

  1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
  2. Make sure you are logged in to Jenkins to see the pages linked in the steps below.
  3. Releasing a new version of refinery-source jars to Maven
  4. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
  5. There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to
If the maven release job failed (step 3)

First find why looking at job logs, and then two solutions:

  • Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
    • Clean maven commits:
      • git reset --hard YOUR_LAST_COMMIT_HASH
      • git push --force origin master
    • Clean maven tag:
      • git tag -d YOUR_VERSION_TAG
      • git push origin :refs/tags/YOUR_VERSION_TAG
  • Or, skip the version number you were currently deploying
    • Update the changelog,md with the skipped version info
    • Deploy again !
If the maven jar upgrade job failed (step 4)

How to deploy from the CLI

  • Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
  • Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
  • Check everything looks Ok:
cat release.properties
  • Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

Please see the refinery page to deploy the jars and oozie code.