Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> More cores Vs More Nodes ?


Copy link to this message
-
Re: More cores Vs More Nodes ?
You're using OS virtualization in your test.  Are you using it in production?

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

On Dec 13, 2011, at 5:16 PM, Brad Sarsfield <[EMAIL PROTECTED]> wrote:

> The experiment was done in a cloud hosted environment running set of VMs.
>
> ~Brad
>
> -----Original Message-----
> From: Prashant Kommireddi [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, December 13, 2011 9:46 AM
> To: [EMAIL PROTECTED]
> Subject: Re: More cores Vs More Nodes ?
>
> Hi Brad, how many taskstrackers did you have on each node in both cases?
>
> Thanks,
> Prashant
>
> Sent from my iPhone
>
> On Dec 13, 2011, at 9:42 AM, Brad Sarsfield <[EMAIL PROTECTED]> wrote:
>
>> Praveenesh,
>>
>> Your question is not naïve; in fact, optimal hardware design can ultimately be a very difficult question to answer on what would be "better". If you made me pick one without much information I'd go for more machines.  But...
>>
>> It all depends; and there is no right answer.... :)
>>
>> More machines
>>  +May run your workload faster
>>  +Will give you a higher degree of reliability protection from node / hardware / hard drive failure.
>>  +More aggregate IO capabilities
>>  - capex / opex may be higher than allocating more cores More cores
>>  +May run your workload faster
>>  +More cores may allow for more tasks to run on the same machine
>>  +More cores/tasks may reduce network contention and increase increasing task to task data flow performance.
>>
>> Notice "May run your workload faster" is in both; as it can be very workload dependant.
>>
>> My Experience:
>> I did a recent experiment and found that given the same number of cores (64) with the exact same network / machine configuration;
>>  A: I had 8 machines with 8 cores
>>  B: I had 28 machines with 2 cores (and 1x8 core head node)
>>
>> B was able to outperform A by 2x using teragen and terasort. These machines were running in a virtualized environment; where some of the IO capabilities behind the scenes were being regulated to 400Mbps per node when running in the 2 core configuration vs 1Gbps on the 8 core.  So I would expect the non-throttled scenario to work even better.
>>
>> ~Brad
>>
>>
>> -----Original Message-----
>> From: praveenesh kumar [mailto:[EMAIL PROTECTED]]
>> Sent: Monday, December 12, 2011 8:51 PM
>> To: [EMAIL PROTECTED]
>> Subject: More cores Vs More Nodes ?
>>
>> Hey Guys,
>>
>> So I have a very naive question in my mind regarding Hadoop cluster nodes ?
>>
>> more cores or more nodes - Shall I spend money on going from 2-4 core machines, or spend money on buying more nodes less core eg. say 2 machines of 2 cores for example?
>>
>> Thanks,
>> Praveenesh
>>
>