You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Joal
m (Add link to refinery-source deployement)
imported>Milimetric
 
(10 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Refinery is the software infrastructure that is used on the [[Analytics/Cluster|Analytics Cluster]]. The source code is in the [http://git.wikimedia.org/summary/analytics%2Frefinery.git analytics/refinery repository].
#REDIRECT [[Analytics/Systems/Cluster/Deploy/Refinery]]
 
This repository uses jars created from [http://git.wikimedia.org/summary/analytics%2Frefinery%2Fsource.git analytics/refinery/source], see [[Analytics/Cluster/Refinery-source|this page]] for deploying those.
 
== How to deploy ==
 
# Ssh into [[Tin]]
# Run:
#:
#:  <code>cd /srv/deployment/analytics/refinery</code>
#:  <code>git deploy start</code>
#:  <code>git checkout master</code>
#:  <code>git pull</code>
#:  <code>git deploy sync</code>
#:<br />
#:(<code>git deploy sync</code> will complain that only “2/3 minions completed fetch”. You can say “y”es to that)
#:<br/>
#:This part brings the refinery code from gerrit to <code>stat1002</code>.
# Ssh into [[stat1002]]
# Run <code>sudo -u hdfs /srv/deployment/analytics/refinery/bin/refinery-deploy-to-hdfs --verbose --no-dry-run</code>
#:<br>
#:This part brings the refinery code to the HDFS (but it does not resubmit Oozie jobs).
 
=== How to deploy Oozie jobs ===
 
Please see the [[Analytics/Cluster/Oozie#How_to_deploy_Oozie_production_jobs|Deployment section in the Oozie docs]].

Latest revision as of 13:15, 10 July 2017