You are browsing a read-only backup copy of Wikitech. The live site can be found at

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
(→‎How to deploy with Jenkins: Add some links and recovery procedures.)
m (Bot: Fixing double redirect to Analytics/Systems/Cluster/Deploy/Refinery-source)
(5 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [ analytics/refinery/source].
#REDIRECT [[Analytics/Systems/Cluster/Deploy/Refinery-source]]
== How to deploy with Jenkins ==
# Update the file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Make sure you are logged in to [ Jenkins] to see the pages linked in the steps below.
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address:, in the bottom-left <code>Build History</code> panel (In case the last job fails, it will have a red dot before its build number - see below).
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
===== If the maven release job failed (step 4) =====
First find why looking at job logs, and then two solutions:
* Clean Maven commits for version upgrade in pom files and associated tag (<u>can break things, be careful !</u>)
** Clean maven commits:
*** <code>git reset --hard YOUR_LAST_COMMIT_HASH</code>
*** <code>git push --force origin master</code>
** Clean maven tag:
*** <code>git tag -d YOUR_VERSION_TAG</code>
*** <code>git push origin :refs/tags/YOUR_VERSION_TAG</code>
* Or, skip the version number you were currently deploying
** Update the changelog,md with the skipped version info
** Deploy again !
===== If the maven jar upgrade job failed (step 5) =====
= How to deploy =
{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}
* Update the file at the root of the repository whith changes that are going to be deployed.
*  Prepare deployment (change pom.xml files, push to git):
mvn release:prepare
* Check everything looks Ok:
* Actually deploy (jar generation and uploads to archiva)
mvn release:perform
* Download the new version of the jars from archiva ({{Warning | git fat uses IDs to manage files, so it is important to use the correct jars. }}) For convenience, the commonly updated jars are:
* NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update with and download that
* Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
* Update the symlinks in refinery/artifacts to the jars you just copied
* Make sure git fat is installed and configured according to the instructions in the Refinery README
* git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
* push for review
Please see the [[Analytics/Cluster/Refinery|refinery page]] to deploy the jars and oozie code.

Revision as of 18:58, 13 July 2017