Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: A small portion of map tasks slows down the job


Copy link to this message
-
Re: A small portion of map tasks slows down the job
Hi,

Would reducing the output from the map tasks solve the problem ? i.e. are
reducers slowing down because a lot of data is being shuffled ?

If that's the case, you could see if the map output size will reduce by
using the framework's combiner or an in-mapper combining technique.

Thanks
Hemanth

On Wed, Oct 3, 2012 at 6:34 AM, Huanchen Zhang <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I have a small portion of map tasks whose output is much larger than
> others (more spills). So the reducer is mainly waiting for these a few map
> tasks. Is there a good solution for this problem ?
>
> Thank you.
>
> Best,
> Huanchen
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB