You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Paladox
No edit summary
imported>MarcoAurelio
m (Bot: Fixing double redirect to Analytics/Systems/Cluster/Deploy/Refinery-source)
 
(7 intermediate revisions by 5 users not shown)
Line 1: Line 1:
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source].
#REDIRECT [[Analytics/Systems/Cluster/Deploy/Refinery-source]]
 
= How to deploy =
 
{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}
 
* Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
*  Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
* Check everything looks Ok:
cat release.properties
* Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
* Download the new version of the jars from archiva ({{Warning | git fat uses IDs to manage files, so it is important to use the correct jars. }}) For convenience, the commonly updated jars are:
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
* NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
* Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
* Update the symlinks in refinery/artifacts to the jars you just copied
* Make sure git fat is installed and configured according to the instructions in the Refinery README
* git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
* push for review
 
Please see the [[Analytics/Cluster/Refinery|refinery page]] to deploy the jars and oozie code.

Latest revision as of 18:58, 13 July 2017