You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Flink: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Physikerwelt
(Created page with "This page document setup of Flink on the math labs cluster.")
 
imported>Mschwarzer
(Added master node info)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
This page document setup of [[w:Apache_Flink|Flink]] on the math labs cluster.
This page document setup of [[w:Apache_Flink|Flink]] on the math labs cluster.
==Install Apache Flink and run job via YARN==
# A working Hadoop cluster needs to be installed first ([[User:QChris/TestClusterSetup|follow these instructions]]).
# Sign in to master node (e.g. <code>hadoop000</code>)
# Set up Apache Flink
#* Download Flink files: <code>wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code>
#* Extract files <code>tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code>
#* Let's Flink discover the Hadoop configuration directory <code>export HADOOP_CONF_DIR=/etc/hadoop/conf</code>
# Run Flink job on YARN
#* WordCount.jar prints output to console <code>flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar</code>
#* See [https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/yarn_setup.html official Flink YARN documentation] for more details.

Latest revision as of 09:22, 19 December 2016

This page document setup of Flink on the math labs cluster.

Install Apache Flink and run job via YARN

  1. A working Hadoop cluster needs to be installed first (follow these instructions).
  2. Sign in to master node (e.g. hadoop000)
  3. Set up Apache Flink
  4. Run Flink job on YARN
    • WordCount.jar prints output to console flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar
    • See official Flink YARN documentation for more details.