Jitendra Yadav 2013-05-23, 18:56
When you take a mapreduce tasks, you need CPU cycles to do the processing, not just memory.
So ideally based on the processor type(hyperthreaded or not) compute the available cores. Then may be compute as, one core for each task slot.
Sent from remote device, Please excuse typos
From: Jitendra Yadav <[EMAIL PROTECTED]>
Date: Fri, 24 May 2013 00:26:29
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Hadoop Installation Mappers setting
Thanks for your clarification.
I have one more question.
How does cores factor influence slots calculation?
On 5/23/13, Amal G Jose <[EMAIL PROTECTED]> wrote:
> I am explaining it more.
> If your machine have 8 GB of memory.
> After reserving to Operating system and all other processes except
> tasktracker, you have 4 GB remaining(assume).
> The remaining process running is tasktracker.
> If the child jvm size is 200 MB,
> Then you can define a maximum slots of 4*1024 MB/ 200 MB
> Which is approximately 20.
> You can divide the slots into mapper and reducer slots as per your
> This is just an example that I explained based on my knowledge.
> On Thu, May 23, 2013 at 7:48 PM, <[EMAIL PROTECTED]> wrote:
>> I assume the question is on how many slots.
>> It dependents on
>> - the child/task jvm size and the available memory.
>> - available number of cores
>> Your available memory for tasks is total memory - memory used for OS and
>> other services running on your box.
>> Other services include non hadoop services as well as hadoop daemons.
>> Divide the available memory with child jvm size and that would get the
>> num of slots.
>> Also check whether sufficient number of cores are available as well.
>> Bejoy KS
>> Sent from remote device, Please excuse typos
>> *From: * Jitendra Yadav <[EMAIL PROTECTED]>
>> *Date: *Thu, 23 May 2013 18:10:38 +0530
>> *To: *<[EMAIL PROTECTED]>
>> *ReplyTo: * [EMAIL PROTECTED]
>> *Subject: *Hadoop Installation Mappers setting
>> While installing hadoop cluster, how we can calculate the exact number of
>> mappers value.