Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Spindle per Cores


Copy link to this message
-
RE: Spindle per Cores
What empirical evidence is there for this rule of thumb?
In other words, what tests or metrics would indicate an optimal spindle/core ratio and how dependent is this on the nature of the data and of the map/reduce computation?

My understanding is that there are lots of clusters with more spindles than cores.  Specifically, typical 2U servers can hold 12 3.5" disk drives.  So lots of Hadoop clusters have dual 4 core processors and 12 spindles.  Would it be better to have 6 core processors if you are loading up the boxes with 12 disks?  And most importantly, how would one know that the mix was optimal?

Hank Cohen
Altior Inc.

-----Original Message-----
From: Patai Sangbutsarakum [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 12, 2012 10:46 AM
To: [EMAIL PROTECTED]
Subject: Spindle per Cores

I have read around about the hardware recommendation for hadoop cluster.
One of them is recommend 1:1 ratio between spindle per core.

Intel CPU come with Hyperthread which will double the number cores on one physical CPU. eg. 8 cores with Hyperthread it because 16 which is where we start to calculate about number of task slots per node.

Once it come to spindle, i strongly believe I should pick 8 cores and picks 8 disks in order to get 1:1 ratio.

Please suggest
Patai
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB