You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Flink: Difference between revisions
Jump to navigation
Jump to search
imported>Mschwarzer (Added guide how to run an Apache Flink job on YARN) |
imported>Mschwarzer (Added master node info) |
||
| Line 3: | Line 3: | ||
==Install Apache Flink and run job via YARN== | ==Install Apache Flink and run job via YARN== | ||
# A working Hadoop cluster needs to be installed first ([[User:QChris/TestClusterSetup|follow these instructions]]). | # A working Hadoop cluster needs to be installed first ([[User:QChris/TestClusterSetup|follow these instructions]]). | ||
# Sign in to master node (e.g. <code>hadoop000</code>) | |||
# Set up Apache Flink | # Set up Apache Flink | ||
#* Download Flink files: <code>wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code> | #* Download Flink files: <code>wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz</code> | ||
Latest revision as of 09:22, 19 December 2016
This page document setup of Flink on the math labs cluster.
Install Apache Flink and run job via YARN
- A working Hadoop cluster needs to be installed first (follow these instructions).
- Sign in to master node (e.g.
hadoop000) - Set up Apache Flink
- Download Flink files:
wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz - Extract files
tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz - Let's Flink discover the Hadoop configuration directory
export HADOOP_CONF_DIR=/etc/hadoop/conf
- Download Flink files:
- Run Flink job on YARN
- WordCount.jar prints output to console
flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar - See official Flink YARN documentation for more details.
- WordCount.jar prints output to console