You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Joal
(→‎How to deploy with Jenkins: Add some links and recovery procedures.)
imported>MarcoAurelio
m (Bot: Fixing double redirect to Analytics/Systems/Cluster/Deploy/Refinery-source)
(5 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source].
#REDIRECT [[Analytics/Systems/Cluster/Deploy/Refinery-source]]
 
== How to deploy with Jenkins ==
# Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Make sure you are logged in to [https://integration.wikimedia.org/ci/ Jenkins] to see the pages linked in the steps below.
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left <code>Build History</code> panel (In case the last job fails, it will have a red dot before its build number - see below).
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
 
===== If the maven release job failed (step 4) =====
First find why looking at job logs, and then two solutions:
* Clean Maven commits for version upgrade in pom files and associated tag (<u>can break things, be careful !</u>)
** Clean maven commits:
*** <code>git reset --hard YOUR_LAST_COMMIT_HASH</code>
*** <code>git push --force origin master</code>
** Clean maven tag:
*** <code>git tag -d YOUR_VERSION_TAG</code>
*** <code>git push origin :refs/tags/YOUR_VERSION_TAG</code>
* Or, skip the version number you were currently deploying
** Update the changelog,md with the skipped version info
** Deploy again !
 
===== If the maven jar upgrade job failed (step 5) =====
 
= How to deploy =
 
{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}
 
* Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
*  Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
* Check everything looks Ok:
cat release.properties
* Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
* Download the new version of the jars from archiva ({{Warning | git fat uses IDs to manage files, so it is important to use the correct jars. }}) For convenience, the commonly updated jars are:
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
* NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
* Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
* Update the symlinks in refinery/artifacts to the jars you just copied
* Make sure git fat is installed and configured according to the instructions in the Refinery README
* git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
* push for review
 
Please see the [[Analytics/Cluster/Refinery|refinery page]] to deploy the jars and oozie code.

Revision as of 18:58, 13 July 2017

Redirect to:

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Analytics/Cluster/Refinery-source&oldid=290168"