You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Flink
Revision as of 12:36, 17 December 2016 by imported>Mschwarzer (Added guide how to run an Apache Flink job on YARN)
This page document setup of Flink on the math labs cluster.
Install Apache Flink and run job via YARN
- A working Hadoop cluster needs to be installed first (follow these instructions).
- Set up Apache Flink
- Download Flink files:
wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz - Extract files
tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz - Let's Flink discover the Hadoop configuration directory
export HADOOP_CONF_DIR=/etc/hadoop/conf
- Download Flink files:
- Run Flink job on YARN
- WordCount.jar prints output to console
flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar - See official Flink YARN documentation for more details.
- WordCount.jar prints output to console