You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Flink: Difference between revisions
Jump to navigation
Jump to search
imported>Physikerwelt (Created page with "This page document setup of Flink on the math labs cluster.") |
imported>Mschwarzer (Added guide how to run an Apache Flink job on YARN) |
||
Line 1: | Line 1: | ||
This page document setup of [[w:Apache_Flink|Flink]] on the math labs cluster. | This page document setup of [[w:Apache_Flink|Flink]] on the math labs cluster. | ||
==Install Apache Flink and run job via YARN== | |||
# A working Hadoop cluster needs to be installed first ([[User:QChris/TestClusterSetup|follow these instructions]]). | |||
# Set up Apache Flink | |||
#* Download Flink files: <code>wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code> | |||
#* Extract files <code>tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code> | |||
#* Let's Flink discover the Hadoop configuration directory <code>export HADOOP_CONF_DIR=/etc/hadoop/conf</code> | |||
# Run Flink job on YARN | |||
#* WordCount.jar prints output to console <code>flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar</code> | |||
#* See [https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/yarn_setup.html official Flink YARN documentation] for more details. |
Revision as of 12:36, 17 December 2016
This page document setup of Flink on the math labs cluster.
Install Apache Flink and run job via YARN
- A working Hadoop cluster needs to be installed first (follow these instructions).
- Set up Apache Flink
- Download Flink files:
wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz
- Extract files
tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz
- Let's Flink discover the Hadoop configuration directory
export HADOOP_CONF_DIR=/etc/hadoop/conf
- Download Flink files:
- Run Flink job on YARN
- WordCount.jar prints output to console
flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar
- See official Flink YARN documentation for more details.
- WordCount.jar prints output to console