Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Why number of reducers should be less than number of reducer slots.


Copy link to this message
-
Re: Why number of reducers should be less than number of reducer slots.
Karthik Kambatla 2013-04-23, 06:33
I wonder how accurate that is.

However, by setting the number of reducers slightly lesser than the reduce
slots, the difference acts as headroom for speculative reduce tasks. And,
the goal of a single wave is also preserved.
On Mon, Apr 22, 2013 at 11:10 PM, Darpan R <[EMAIL PROTECTED]> wrote:

> Hi guys,
>  I read somewhere that for better performance
>
> For maximum performance, the number of reducers should be slightly less
> than
> the number of reduce slots in the cluster. This allows the reducers to
> finish in
> one wave and fully utilizes the cluster during the reduce phase.
>
> I don't quite understand this, Can you please help me understand?
>
> Thank you.
>