Analytics/Systems/Cluster/Deploy/Refinery-source

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins (and related steps)

Before starting, please check:

• Check the latest Refinery source version. This can be done in various way, the easiest one is to check git tag --list
• The Refinery source commit list and make sure that the changelog.md has been updated with the latest version and possibly that [maven-release-plugin] has committed the related version bump changes (the last step is optional, it could be triggered manually following the instruction below).
• Make sure you are logged in to Jenkins to see the pages linked in the steps below.

Have you read the instructions reported above? If so, please keep going, otherwise please read them!

Deploy procedure:

1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
2. Release a new version of refinery-source jars to Maven:
   • Visit https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ and make sure that:
     • Release Version should be the version number you wish to deploy
     • Deployment Version should be the next iteration of development version, usually Release Version + 1 with -SNAPSHOT at the end
     • Example: Release 0.0.40 - Development 0.0.41-SNAPSHOT
     • If the versions are not the ones that you expect, you can refresh them with https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec This is known to refresh the values that you'll see in the next step, but usually they should be pre-filled.
   • Check the "Specify custom SCM tag" checkbox
   • Change refinery-x.y.z to vx.y.z in the "SCM tag" input text-box and update the number. Example: refinery-0.0.40 is bad, v0.0.40 is good
   • Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left Build History panel (In case the last job fails, it will have a red dot before its build number - see below)
3. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
4. There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to
   • bump up jar version in the related oozie jobs (see an example here)
   • deploy refinery with your changes
   • restart the modified oozie jobs.

If the maven release job failed (step 2)

First find why looking at job logs, and then two solutions:

• Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
  • Clean maven commits:
    • git reset --hard YOUR_LAST_COMMIT_HASH
    • git push --force origin master
  • Clean maven tag:
    • git tag -d YOUR_VERSION_TAG
    • git push origin :refs/tags/YOUR_VERSION_TAG
• Or, skip the version number you were currently deploying
  • Update the changelog,md with the skipped version info
  • Deploy again!

Administration

• Archiva-CI credentials are stored in Jenkins at https://integration.wikimedia.org/ci/credentials/.
• Jenkins Maven / Archiva settings are at https://integration.wikimedia.org/ci/configfiles/.
• You can edit the analytics-refinery-release Jenkins job at https://integration.wikimedia.org/ci/job/analytics-refinery-release/configure

How to deploy from the CLI

• Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
• Prepare deployment (change pom.xml files, push to git):

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare

• Check everything looks Ok:

cat release.properties

• Actually deploy (jar generation and uploads to archiva)

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

• Download the new version of the jars from archiva () For convenience, the commonly updated jars are:
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
• NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
• Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
• Update the symlinks in refinery/artifacts to the jars you just copied
• Make sure git fat is installed and configured according to the instructions in the Refinery README
• git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
• push for review

Please see the refinery page to deploy the jars and oozie code.