Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> question about combiner


Copy link to this message
-
Re: question about combiner
@Kishore, Agreed but but shouldn't 'Reduce shuffle bytes' count decrease
with the use of Combiners?

Regards,
Shahab
On Fri, May 10, 2013 at 2:00 PM, Kishore <[EMAIL PROTECTED]> wrote:

> Combiner will be used between mapper and reduce, so the mapper output for
> both with used combiner and without used combiner are same.
>
> Thanks,
> Kishore.
>
> Sent from my iPhone
>
> On 10-May-2013, at 8:49 PM, Han JU <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> For a MapReduce job with lots of intermediate results between mapper and
> reducer, I implement a combiner function with a more compact representation
> of the result data and I verified the final result is good when using
> combiner. But when I look at the job counter "FILE_BYTES_WRITTEN" or
> "Reduce shuffle bytes", the number with combiner is twice bigger than
> without combiner. In my comprehension, these two counters represent the
> output size of mapper. And with a combiner, the size of mapper output
> should decrease, but it's not the case here.
>
> So it means that my combiner doesn't work and it actually increase the
> size of mapper output?
>
> Thanks!
> --
> *JU Han*
>
> Software Engineer Intern @ KXEN Inc.
> UTC   -  Université de Technologie de Compiègne
> *     **GI06 - Fouille de Données et Décisionnel*
>
> +33 0619608888
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB