You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Hardware: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
No edit summary
 
imported>Ottomata
(Updates for recent host additions and renames.)
Line 20: Line 20:
| Dell PowerEdge R720 || 12 || analytics1011 - analytics1022 || 12 core EW-2620 @ 2.00GHz || 48G RAM || 24T = 12 * 2T
| Dell PowerEdge R720 || 12 || analytics1011 - analytics1022 || 12 core EW-2620 @ 2.00GHz || 48G RAM || 24T = 12 * 2T
|-
|-
| Dell PowerEdge R720 || 14 || analytics1028 - analytics1041 || 12 core EW-2620 @ 2.00GHz || 64G RAM || 48T = 12 * 4T
| Dell PowerEdge R720 || 14 || analytics1028 - analytics1057 || 12 core EW-2620 @ 2.00GHz || 64G RAM || 48T = 12 * 4T, + 2 SSDs
|-
|-
| Dell PowerEdge R310 || 5 || analytics1026 - analytics1027 || 4 core X3430 @ 2.40GHz || 8G RAM || 002G = 2 * 1G
| Dell PowerEdge R310 || 5 || analytics1026 - analytics1027 || 4 core X3430 @ 2.40GHz || 8G RAM || 002G = 2 * 1G
Line 30: Line 30:
! Type !! !!  Cisco UCS C250 M1 !! Dell PowerEdge R720 !! Dell PowerEdge R720 !! Dell PowerEdge R310
! Type !! !!  Cisco UCS C250 M1 !! Dell PowerEdge R720 !! Dell PowerEdge R720 !! Dell PowerEdge R310
|-
|-
| '''Hosts''' || analytics1001, analytics1002 || analytics1003 - analytics1010 || analytics1011 - analytics1022 || analytics1028-analytics1041 || analytics1026 - analytics1027
| '''Hosts''' || analytics1001, analytics1002 || analytics1003 - analytics1010 || analytics1011 - analytics1022 || analytics1028-analytics1057 || analytics1026 - analytics1027
|-
|-
| '''Processors''' || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 24 core X5650 @ 2.67 GHz || 12 core EW-2620 @ 2.00GHz || 12 core EW-2620 @ 2.00GHz || 4 core X3430 @ 2.40GHz
| '''Processors''' || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 24 core X5650 @ 2.67 GHz || 12 core EW-2620 @ 2.00GHz || 12 core EW-2620 @ 2.00GHz || 4 core X3430 @ 2.40GHz
Line 36: Line 36:
| '''Memory''' || 64G RAM || 192G RAM || 48G RAM || 64G RAM || 8G RAM
| '''Memory''' || 64G RAM || 192G RAM || 48G RAM || 64G RAM || 8G RAM
|-
|-
| '''Disk''' || 4 * 2T || 2.4T = 8 x 300G || 24T = 12 * 2T || 48T = 12 * 4T || 2G = 2 * 1G
| '''Disk''' || 4 * 2T || 2.4T = 8 x 300G || 24T = 12 * 2T || 48T = 12 * 4T +  2 SSDs || 2G = 2 * 1G
|}
|}


Line 56: Line 56:
| analytics1011,analytics1019,analytics1028 || 3 || R720 || Hadoop Journal + Worker Nodes
| analytics1011,analytics1019,analytics1028 || 3 || R720 || Hadoop Journal + Worker Nodes
|-
|-
| analytics1013-analytics1017,analytics1020,analytics1029-analytics1041 || 19 || R720 || Hadoop Worker Nodes (DataNodes)
| analytics1011,analytics1015-analytics1017,analytics1019,analytics1029-analytics1057 || 33 || R720 || Hadoop Worker Nodes (DataNodes)
|-
|-
| analytics1012,analytics1018,analytics1021,analytics1022 || 4 || R720 || Kafka Brokers
| kafka1012,kafka1013,kafka1014,kafka1018,kafka1020,kafka1022 || 6 || R720 || Kafka Brokers
|-
|-
| analytics1026 || 1 || R310 || Impala master (impala-state-store, impala-catalog, llama)
| analytics1026 || 1 || R310 || Impala master (impala-state-store, impala-catalog, llama)
Line 65: Line 65:
|}
|}


''Note: As of June 2015, analytics102[345] which were Zookeeper servers in the Analytics Cluster have been moved out into the general purpose production network and renamed as conf100[123]. These are still Zookeeper servers, but will also host etcd.''
''Notes:''
 
''- As of June 2015, analytics102[345] which were Zookeeper servers in the Analytics Cluster have been moved out into the general purpose production network and renamed as conf100[123]. These are still Zookeeper servers, but will also host etcd.''
 
- ''In August 2015, analytics10xx kafka brokers have been renamed to kafka10xx with the numbers (e.g. analytics1012 -> kafka1012).''

Revision as of 16:56, 21 August 2015

Information about the Analytics Cluster hardware and infrastructure.


Docs

Hardware Specs

By Host

Type # Hosts Processors Memory Disk
Cisco UCS C250 M1 4 analytics1003,analytics1004,analytics1009,analytics1010 24 core X5650 @ 2.67 GHz 192G RAM 02.4T = 8 x 300G
Dell PowerEdge R420 2 analytics1001, analytics1002 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz 64G RAM 4 * 2T
Dell PowerEdge R720 12 analytics1011 - analytics1022 12 core EW-2620 @ 2.00GHz 48G RAM 24T = 12 * 2T
Dell PowerEdge R720 14 analytics1028 - analytics1057 12 core EW-2620 @ 2.00GHz 64G RAM 48T = 12 * 4T, + 2 SSDs
Dell PowerEdge R310 5 analytics1026 - analytics1027 4 core X3430 @ 2.40GHz 8G RAM 002G = 2 * 1G

By Machine Type

Type Cisco UCS C250 M1 Dell PowerEdge R720 Dell PowerEdge R720 Dell PowerEdge R310
Hosts analytics1001, analytics1002 analytics1003 - analytics1010 analytics1011 - analytics1022 analytics1028-analytics1057 analytics1026 - analytics1027
Processors 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz 24 core X5650 @ 2.67 GHz 12 core EW-2620 @ 2.00GHz 12 core EW-2620 @ 2.00GHz 4 core X3430 @ 2.40GHz
Memory 64G RAM 192G RAM 48G RAM 64G RAM 8G RAM
Disk 4 * 2T 2.4T = 8 x 300G 24T = 12 * 2T 48T = 12 * 4T + 2 SSDs 2G = 2 * 1G

Roles

Hosts # Type Role
analytics1001 2 Primary NameNode, Hadoop master node.
analytics1002 2 Standby NameNode
analytics1003 2 Cisco
analytics1004 1 Cisco
analytics1010 1 Cisco
analytics1011,analytics1019,analytics1028 3 R720 Hadoop Journal + Worker Nodes
analytics1011,analytics1015-analytics1017,analytics1019,analytics1029-analytics1057 33 R720 Hadoop Worker Nodes (DataNodes)
kafka1012,kafka1013,kafka1014,kafka1018,kafka1020,kafka1022 6 R720 Kafka Brokers
analytics1026 1 R310 Impala master (impala-state-store, impala-catalog, llama)
analytics1027 1 R310 Frontend Web Service Host (Hive, Hue, Oozie, etc.)

Notes:

- As of June 2015, analytics102[345] which were Zookeeper servers in the Analytics Cluster have been moved out into the general purpose production network and renamed as conf100[123]. These are still Zookeeper servers, but will also host etcd.

- In August 2015, analytics10xx kafka brokers have been renamed to kafka10xx with the numbers (e.g. analytics1012 -> kafka1012).