You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Analytics/Cluster/Hardware: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
imported>Ottomata
Line 14: Line 14:
! Type !! # !! Hosts !! Processors !! Memory !! Disk
! Type !! # !! Hosts !! Processors !! Memory !! Disk
|-
|-
|Dell PowerEdge R420 || 2 || analytics1001, analytics1002 || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 64G RAM || 4 * 2T
|Dell PowerEdge R420 ||3|| analytics1001, analytics1002, analytics1003 || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 64G RAM || 4 * 2T
|-
|-
| Dell PowerEdge R720 || 12 || analytics10{15,17,21},aqs100{1,2,3},kafka10{12,13,14,18,20,22}|| 12 core EW-2620 @ 2.00GHz || 48G RAM || 24T = 12 * 2T
| Dell PowerEdge R720 || 12 || aqs100{1,2,3},kafka10{12,13,14,18,20,22}|| 12 core EW-2620 @ 2.00GHz || 48G RAM || 24T = 12 * 2T
|-
|-
| Dell PowerEdge R720 || 29 || analytics1028 - analytics1057 || 12 core EW-2620 @ 2.00GHz || 64G RAM || 48T = 12 * 4T, + 2 SSDs
| Dell PowerEdge R720 || 29 || analytics1028 - analytics1057 || 12 core EW-2620 @ 2.00GHz || 64G RAM || 48T = 12 * 4T, + 2 SSDs
Line 28: Line 28:
! Type !! Dell PowerEdge R420 !!  Dell PowerEdge R720 !! Dell PowerEdge R720 !! Dell PowerEdge R310
! Type !! Dell PowerEdge R420 !!  Dell PowerEdge R720 !! Dell PowerEdge R720 !! Dell PowerEdge R310
|-
|-
| '''Hosts''' || analytics1001, analytics1002 || analytics10{15,17,21},aqs100{1,2,3},kafka10{12,13,14,18,20,22} || analytics1028-analytics1057 || analytics1026 - analytics1027
| '''Hosts''' || analytics1001, analytics1002, analytics1003 || aqs100{1,2,3},kafka10{12,13,14,18,20,22} || analytics1028-analytics1057 || analytics1026 - analytics1027
|-
|-
| '''Processors''' || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 12 core EW-2620 @ 2.00GHz || 12 core EW-2620 @ 2.00GHz || 4 core X3430 @ 2.40GHz
| '''Processors''' || 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz || 12 core EW-2620 @ 2.00GHz || 12 core EW-2620 @ 2.00GHz || 4 core X3430 @ 2.40GHz
Line 45: Line 45:
|-
|-
| analytics1002 || 2 || R420 || Standby NameNode
| analytics1002 || 2 || R420 || Standby NameNode
|-
|analytics1003
|1
|R420
|Hive, Oozie, MySQL
|-
|-
|analytics1028-analytics1057 || 30 || R720 || Hadoop Worker Nodes (DataNodes)
|analytics1028-analytics1057 || 30 || R720 || Hadoop Worker Nodes (DataNodes)
Line 55: Line 60:
|-
|-
| aqs1001-aqs1003 || 3 || R720 || Analytics Query Service (RESTBase + Cassandra)
| aqs1001-aqs1003 || 3 || R720 || Analytics Query Service (RESTBase + Cassandra)
|-
| analytics1015,analytics1017,analytics1021 || 3 || R720 || ''spare''
|-
|-
|}
|}

Revision as of 18:03, 20 April 2016

Information about the Analytics Cluster hardware and infrastructure.


Docs

Hardware Specs

By Host

Type # Hosts Processors Memory Disk
Dell PowerEdge R420 3 analytics1001, analytics1002, analytics1003 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz 64G RAM 4 * 2T
Dell PowerEdge R720 12 aqs100{1,2,3},kafka10{12,13,14,18,20,22} 12 core EW-2620 @ 2.00GHz 48G RAM 24T = 12 * 2T
Dell PowerEdge R720 29 analytics1028 - analytics1057 12 core EW-2620 @ 2.00GHz 64G RAM 48T = 12 * 4T, + 2 SSDs
Dell PowerEdge R310 5 analytics1026 - analytics1027 4 core X3430 @ 2.40GHz 8G RAM 2T = 2 * 1T

By Machine Type

Type Dell PowerEdge R420 Dell PowerEdge R720 Dell PowerEdge R720 Dell PowerEdge R310
Hosts analytics1001, analytics1002, analytics1003 aqs100{1,2,3},kafka10{12,13,14,18,20,22} analytics1028-analytics1057 analytics1026 - analytics1027
Processors 16 core Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz 12 core EW-2620 @ 2.00GHz 12 core EW-2620 @ 2.00GHz 4 core X3430 @ 2.40GHz
Memory 64G RAM 48G RAM 64G RAM 8G RAM
Disk 4 * 2T 24T = 12 * 2T 48T = 12 * 4T + 2 SSDs 2T = 2 * 1T

Roles

Hosts # Type Role
analytics1001 2 R420 Primary NameNode, Hadoop master node.
analytics1002 2 R420 Standby NameNode
analytics1003 1 R420 Hive, Oozie, MySQL
analytics1028-analytics1057 30 R720 Hadoop Worker Nodes (DataNodes)
kafka1012,kafka1013,kafka1014,kafka1018,kafka1020,kafka1022 6 R720 Kafka Brokers
analytics1026 1 R310 spare
analytics1027 1 R310 Frontend Web Service Host (Hive, Hue, Oozie, etc.)
aqs1001-aqs1003 3 R720 Analytics Query Service (RESTBase + Cassandra)

Notes:

- As of June 2015, analytics102[345] which were Zookeeper servers in the Analytics Cluster have been moved out into the general purpose production network and renamed as conf100[123]. These are still Zookeeper servers, but will also host etcd.

- In August 2015, analytics10xx kafka brokers have been renamed to kafka10xx with the numbers (e.g. analytics1012 -> kafka1012).

- In October 2015, analytics1011,1016,1019 were repurposed as aqs1001,1002,1003.