You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>QChris
m (QChris moved page Analytics/Refinery to Analytics/Cluster/Refinery: Refinery is too tied to the cluster to live directly underneath Analytics)
 
imported>Joal
m (Add link to refinery-source deployement)
Line 1: Line 1:
Refinery is the software infrastructure that is used on the [[Analytics/Cluster|Analytics Cluster]]. The source code is in the [http://git.wikimedia.org/summary/analytics%2Frefinery.git analytics/source repository].
Refinery is the software infrastructure that is used on the [[Analytics/Cluster|Analytics Cluster]]. The source code is in the [http://git.wikimedia.org/summary/analytics%2Frefinery.git analytics/refinery repository].
 
This repository uses jars created from [http://git.wikimedia.org/summary/analytics%2Frefinery%2Fsource.git analytics/refinery/source], see [[Analytics/Cluster/Refinery-source|this page]] for deploying those.


== How to deploy ==
== How to deploy ==

Revision as of 13:57, 17 December 2015

Refinery is the software infrastructure that is used on the Analytics Cluster. The source code is in the analytics/refinery repository.

This repository uses jars created from analytics/refinery/source, see this page for deploying those.

How to deploy

  1. Ssh into Tin
  2. Run:
    cd /srv/deployment/analytics/refinery
    git deploy start
    git checkout master
    git pull
    git deploy sync

    (git deploy sync will complain that only “2/3 minions completed fetch”. You can say “y”es to that)

    This part brings the refinery code from gerrit to stat1002.
  3. Ssh into stat1002
  4. Run sudo -u hdfs /srv/deployment/analytics/refinery/bin/refinery-deploy-to-hdfs --verbose --no-dry-run

    This part brings the refinery code to the HDFS (but it does not resubmit Oozie jobs).

How to deploy Oozie jobs

Please see the Deployment section in the Oozie docs.