Analytics/Cluster/Refinery-source

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy

• Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
• Prepare deployment (change pom.xml files, push to git):

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare

• Check everything looks Ok:

cat release.properties

• Actually deploy (jar generation and uploads to archiva)

mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

• Download the new version of the jars from archiva () For convenience, the commonly updated jars are:
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
  • https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
• NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
• Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
• Update the symlinks in refinery/artifacts to the jars you just copied
• Make sure git fat is installed and configured according to the instructions in the Refinery README
• git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
• push for review

Please see the refinery page to deploy the jars and oozie code.