@Kishore, Agreed but but shouldn't 'Reduce shuffle bytes' count decrease
with the use of Combiners?
On Fri, May 10, 2013 at 2:00 PM, Kishore <[EMAIL PROTECTED]> wrote:
> Combiner will be used between mapper and reduce, so the mapper output for
> both with used combiner and without used combiner are same.
> Sent from my iPhone
> On 10-May-2013, at 8:49 PM, Han JU <[EMAIL PROTECTED]> wrote:
> For a MapReduce job with lots of intermediate results between mapper and
> reducer, I implement a combiner function with a more compact representation
> of the result data and I verified the final result is good when using
> combiner. But when I look at the job counter "FILE_BYTES_WRITTEN" or
> "Reduce shuffle bytes", the number with combiner is twice bigger than
> without combiner. In my comprehension, these two counters represent the
> output size of mapper. And with a combiner, the size of mapper output
> should decrease, but it's not the case here.
> So it means that my combiner doesn't work and it actually increase the
> size of mapper output?
> *JU Han*
> Software Engineer Intern @ KXEN Inc.
> UTC - Université de Technologie de Compiègne
> * **GI06 - Fouille de Données et Décisionnel*
> +33 0619608888