Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How does hadoop decide how many reducers to run?


Copy link to this message
-
Re: How does hadoop decide how many reducers to run?
Michael Segel 2013-01-12, 14:05
Since you are using EMR,  AWS pre configures the number of slots per node.
So you are already getting the optimum number of slots that their 'machines' can handle.

So when you run your job, you said that you saw 16 reducers and then 12 reducers running.

This could imply that your job required 28 reducers  and it was using the full resources of the machines.

On Jan 11, 2013, at 5:53 PM, Roy Smith <[EMAIL PROTECTED]> wrote:

> On Jan 11, 2013, at 6:20 PM, Michael Segel wrote:
>
>> Hi,
>>
>> First, not enough information.
>>
>> 1) EC2 got it.
>> 2) Which flavor of Hadoop? Is this EMR as well?
>
> Yes, EMR.  We're running AMI version 2.3.1, which includes hadoop 1.0.3.
>
>
>> 3) How many slots did you configure in your mapred-site.xml?
>
> Hmmm, no clue.  I've never even heard of that file.  We're using mrjob.  It may be that mrjob is building a mapred-site.xml file for me and I never even see it?
>
> ---
> Roy Smith
> [EMAIL PROTECTED]
>
>
>
>