Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: A small portion of map tasks slows down the job


Copy link to this message
-
Re: A small portion of map tasks slows down the job
Hemanth Yamijala 2012-10-03, 13:37
Hi,

Would reducing the output from the map tasks solve the problem ? i.e. are
reducers slowing down because a lot of data is being shuffled ?

If that's the case, you could see if the map output size will reduce by
using the framework's combiner or an in-mapper combining technique.

Thanks
Hemanth

On Wed, Oct 3, 2012 at 6:34 AM, Huanchen Zhang <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I have a small portion of map tasks whose output is much larger than
> others (more spills). So the reducer is mainly waiting for these a few map
> tasks. Is there a good solution for this problem ?
>
> Thank you.
>
> Best,
> Huanchen