Analytics/Cluster/Deploy/Refinery-source

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins (and related steps)

1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
2. Make sure you are logged in to Jenkins to see the pages linked in the steps below.
3. Releasing a new version of refinery-source jars to Maven
   • If you need to trigger a build (not a release) you can do that hereː https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec This is known to refresh the values that you'll see in the next step, but usually they should be pre-filled
   • Visit https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/
   • Update the version number in the Release Version and Development Version boxes if needed
     • Release Version should be the version number you wish to deploy
     • Deployment Version should be the next iteration of devlopement version, usually Release Version + 1 with -SNAPSHOT at the end
   • Check the "Specify custom SCM tag" checkbox
   • Change refinery-x.y.z to vx.y.z in the "SCM tag" input textbox and update the number
   • Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left Build History panel (In case the last job fails, it will have a red dot before its build number - see below)
4. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
5. There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to
   • bump up jar version in the related oozie jobs (see an example here)
   • deploy refinery with your changes
   • restart the modified oozie jobs.

If the maven release job failed (step 3)

First find why looking at job logs, and then two solutions:

• Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
  • Clean maven commits:
    • git reset --hard YOUR_LAST_COMMIT_HASH
    • git push --force origin master
  • Clean maven tag:
    • git tag -d YOUR_VERSION_TAG
    • git push origin :refs/tags/YOUR_VERSION_TAG
• Or, skip the version number you were currently deploying
  • Update the changelog,md with the skipped version info
  • Deploy again !

How to deploy from the CLI

• Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
• Prepare deployment (change pom.xml files, push to git):

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare

• Check everything looks Ok:

cat release.properties

• Actually deploy (jar generation and uploads to archiva)

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

• Download the new version of the jars from archiva () For convenience, the commonly updated jars are:
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
• NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
• Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
• Update the symlinks in refinery/artifacts to the jars you just copied
• Make sure git fat is installed and configured according to the instructions in the Refinery README
• git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
• push for review

Please see the refinery page to deploy the jars and oozie code.