Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> question about combiner


Copy link to this message
-
Re: question about combiner
@Kishore, Agreed but but shouldn't 'Reduce shuffle bytes' count decrease
with the use of Combiners?

Regards,
Shahab
On Fri, May 10, 2013 at 2:00 PM, Kishore <[EMAIL PROTECTED]> wrote:

> Combiner will be used between mapper and reduce, so the mapper output for
> both with used combiner and without used combiner are same.
>
> Thanks,
> Kishore.
>
> Sent from my iPhone
>
> On 10-May-2013, at 8:49 PM, Han JU <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> For a MapReduce job with lots of intermediate results between mapper and
> reducer, I implement a combiner function with a more compact representation
> of the result data and I verified the final result is good when using
> combiner. But when I look at the job counter "FILE_BYTES_WRITTEN" or
> "Reduce shuffle bytes", the number with combiner is twice bigger than
> without combiner. In my comprehension, these two counters represent the
> output size of mapper. And with a combiner, the size of mapper output
> should decrease, but it's not the case here.
>
> So it means that my combiner doesn't work and it actually increase the
> size of mapper output?
>
> Thanks!
> --
> *JU Han*
>
> Software Engineer Intern @ KXEN Inc.
> UTC   -  Université de Technologie de Compiègne
> *     **GI06 - Fouille de Données et Décisionnel*
>
> +33 0619608888
>
>