You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Ports

From Wikitech-static
< Analytics‎ | Cluster
Revision as of 22:07, 21 September 2014 by imported>Gage (→‎Hadoop)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Ports

Hadoop

App Port Hosts Notes
Hadoop HDFS NameNode 8020 analytics1010
Hadoop HDFS over HTTP (HTTPFS) 14000 analytics1010
Hadoop HDFS NameNode HTTP UI 50070 analytics1010
Hadoop HDFS NameNode HTTPS UI 50470 analytics1010
Hadoop HDFS HA JournalNode 8485
Hadoop HDFS DataNode 1004 analytics1011-analytics1020
Hadoop HDFS DataNode HTTP UI 50075 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop HDFS DataNode IPC 50020 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop YARN ResourceManager 8032 analytics1010
Hadoop YARN ResourceManager Scheduler 8030 analytics1010
Hadoop YARN ResourceManager Tracker 8031 analytics1010
Hadoop YARN ResourceManager Admin 8033 analytics1010
Hadoop YARN ResourceManager HTTP UI 8088 analytics1010
Hadoop YARN NodeManager 8041 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop YARN NodeManager localizer 8040 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop YARN NodeManager HTTP UI 8042 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop MapReduce JobHistory Server 10020 analytics1010
Hadoop MapReduce Shuffle Port 13562 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop MapReduce JobHistory HTTP UI 19888 analytics1010
Hive Server 10000 analytics1027
Hive MetaStore 9083 analytics1027
Zookeeper (client port) 2181 analytics1023,analytics1024,analytics1025
Hue Server 8888 analytics1027
Oozie Server HTTP interface 11000 analytics1027
Oozie Server Admin Port 11001 analytics1027

Kafka

App Port Hosts Notes
Kafka Broker 9092 analytics1012,analytics1018,analytics1021,analytics1022

JMX

For use with jconsole, visualvm, jmxtrans etc.

App Port Hosts Notes
Hadoop Namenode(s) 9980 analytics1010,analytics1004
Hadoop Datanode 9981 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop YARN ResourceManager 9983 analytics1010
Hadoop YARN NodeManager 9984 analytics1011,analytics1013-analytics1017,analytics1019-analytics1020,analytics1028-analytics1041
Hadoop YARN ProxyServer 9985
Kafka 9999 analytics1012,analytics1018,analytics1021,analytics1022
Zookeeper 9998 analytics1023,analytics1024,analytics1025

iptables rules

NOTE: These rules are not active.

Hadoop Reject on ports unless on analytics nodes.

/etc/hadoop/conf/iptables.hadoop

# Generated by iptables-save v1.4.12 on Mon Mar  4 17:50:31 2013
*filter
:INPUT ACCEPT [40:8424]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [184:68687]
:LOGGING - [0:0]
-A INPUT -s 127.0.0.0/8 -p tcp -j ACCEPT
-A INPUT -s 208.80.154.154/32 -p tcp -m multiport --dports 8020,50070,50470,8032,8031,8032,8033,8088,1004,50075,50020,8041,8042,10020,19888 -j ACCEPT
-A INPUT -p tcp -m iprange --src-range 10.64.21.100-10.64.21.110 -m multiport --dports 8020,50070,50470,8032,8031,8032,8033,8088,1004,50075,50020,8041,8042,10020,19888 -j ACCEPT
-A INPUT -p tcp -m iprange --src-range 10.64.36.111-10.64.36.127 -m multiport --dports 8020,50070,50470,8032,8031,8032,8033,8088,1004,50075,50020,8041,8042,10020,19888 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8020,50070,50470,8032,8031,8032,8033,8088,1004,50075,50020,8041,8042,10020,19888 -j LOGGING
-A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Rejected: "
-A LOGGING -j REJECT --reject-with icmp-port-unreachable
COMMIT

/etc/network/if-pre-up.d/iptables-restore-hadoop

#!/bin/sh
iptables-restore < /etc/hadoop/conf/iptables.hadoop
exit 0


Accept all traffic from whitelisted nodes, reject all else

/root/iptables.analytics

# Generated by iptables-save v1.4.12 on Mon Mar 11 12:48:31 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1498:232999]
:LOGGING - [0:0]
:REJECT_LOG - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -s 208.80.154.154/32 -j ACCEPT
-A INPUT -s 10.64.21.0/24 -j ACCEPT
-A INPUT -s 10.64.36.0/24 -j ACCEPT
-A INPUT -s 208.80.154.149/32 -j ACCEPT
-A INPUT -s 208.80.154.14/32 -j ACCEPT
-A INPUT -s 208.80.152.161/32 -j ACCEPT
-A INPUT -s 208.80.154.155/32 -j ACCEPT
-A INPUT -s 208.80.152.146/32 -j ACCEPT
-A INPUT -s 208.80.154.15/32 -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -m state --state NEW -j REJECT_LOG
-A REJECT_LOG -m limit --limit 2/min -j LOG --log-prefix "IPTables-Rejected: "
-A REJECT_LOG -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Mar 11 12:48:31 2013

/etc/network/if-pre-up.d/iptables-restore-analytics

#!/bin/sh
iptables-restore < /root/iptables.analytics
exit 0