Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: why FairScheduler prefer to schedule MR jobs into the same node?


Copy link to this message
-
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Karthik Kambatla 2013-08-09, 21:48
It is possible that you have assignMultiple set to true in your fair
scheduler configuration - that leads to assigning as many tasks on a single
node heartbeat as the node can accommodate. Setting it to false, would
assign a single task on each heartbeat and can help in spreading out the
tasks.
On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <[EMAIL PROTECTED]> wrote:

> The hadoop version is 1.0.3
>
>
> 2013/8/9 Sandy Ryza <[EMAIL PROTECTED]>
>
>> Hi devdoer,
>>
>> What version are you using?
>>
>> -Sandy
>>
>>
>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <[EMAIL PROTECTED]> wrote:
>>
>>> HI:
>>>
>>> I configure the FairScheduler with default settings and my job has 19
>>>  reduce tasks. I found that all the reduce tasks are schedule to run in one
>>> node.
>>>
>>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>>> diffrent nodes.
>>>
>>> How can I configure FairSchedule to load more balance?
>>>
>>>
>>> --
>>> devdoer
>>> [EMAIL PROTECTED]
>>>
>>
>>
>
>
> --
> devdoer
> [EMAIL PROTECTED]
>