Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> question about combiner

Copy link to this message
Re: question about combiner
@Kishore, Agreed but but shouldn't 'Reduce shuffle bytes' count decrease
with the use of Combiners?

On Fri, May 10, 2013 at 2:00 PM, Kishore <[EMAIL PROTECTED]> wrote:

> Combiner will be used between mapper and reduce, so the mapper output for
> both with used combiner and without used combiner are same.
> Thanks,
> Kishore.
> Sent from my iPhone
> On 10-May-2013, at 8:49 PM, Han JU <[EMAIL PROTECTED]> wrote:
> Hi,
> For a MapReduce job with lots of intermediate results between mapper and
> reducer, I implement a combiner function with a more compact representation
> of the result data and I verified the final result is good when using
> combiner. But when I look at the job counter "FILE_BYTES_WRITTEN" or
> "Reduce shuffle bytes", the number with combiner is twice bigger than
> without combiner. In my comprehension, these two counters represent the
> output size of mapper. And with a combiner, the size of mapper output
> should decrease, but it's not the case here.
> So it means that my combiner doesn't work and it actually increase the
> size of mapper output?
> Thanks!
> --
> *JU Han*
> Software Engineer Intern @ KXEN Inc.
> UTC   -  Université de Technologie de Compiègne
> *     **GI06 - Fouille de Données et Décisionnel*
> +33 0619608888