Hi all,

As I know, Combiner is used in the mapper task, and most of the time,
combiner is the same as reducer.

So if combiner is used, the output type of mapper task must the same as
reducer task, is it right ?
Thank you

Jeff Zhang

if your combiner is the same as reducer,the output type of mapper must the
same as
the input type of reducer.

map: (k1,v1)  -> list(k2,v2)
combine: (k2 ,list(v2)) -> list(k2,v2)
reduce: (k2 ,list(v2)) -> list(k3,v3)
－－－－－
天天开心
身体健康
Sent from Guangzhou, Guangdong, China
Stephen Leacock<http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html>
- "I detest life-insurance agents: they always argue that I shall some
day
die, which is not so."

2009/11/22 Jeff Zhang <[EMAIL PROTECTED]>

> Hi all,
>
> As I know, Combiner is used in the mapper task, and most of the time,
> combiner is the same as reducer.
>
> So if combiner is used, the output type of mapper task must the same as
> reducer task, is it right ?
>
>
> Thank you
>
> Jeff Zhang
>

I have never tried it, but the following must also be possible.

map: (k1,v1)  -> list(k2,v2)
combine: (k2 ,list(v2)) -> list(k3,v3)
reduce: (k3 ,list(v3)) -> list(k4,v4)

Karthik Kambatla
On Sun, Nov 22, 2009 at 4:16 AM, Y G <[EMAIL PROTECTED]> wrote:

> if your combiner is the same as reducer,the output type of mapper must the
> same as
> the input type of reducer.
>
> map: (k1,v1)  -> list(k2,v2)
> combine: (k2 ,list(v2)) -> list(k2,v2)
> reduce: (k2 ,list(v2)) -> list(k3,v3)
> －－－－－
> 天天开心
> 身体健康
> Sent from Guangzhou, Guangdong, China
> Stephen Leacock<
> http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html>
> - "I detest life-insurance agents: they always argue that I shall some
> day
> die, which is not so."
>
> 2009/11/22 Jeff Zhang <[EMAIL PROTECTED]>
>
> > Hi all,
> >
> > As I know, Combiner is used in the mapper task, and most of the time,
> > combiner is the same as reducer.
> >
> > So if combiner is used, the output type of mapper task must the same as
> > reducer task, is it right ?
> >
> >
> > Thank you
> >
> > Jeff Zhang
> >
>

karthik,

In your case, then the Combiner is not the same as Reducer.
So it is not necessary to make the combiner same as reducer, right?

Thank you

Jeff Zhang
2009/11/22 Karthik Kambatla <[EMAIL PROTECTED]>

> I have never tried it, but the following must also be possible.
>
> map: (k1,v1)  -> list(k2,v2)
> combine: (k2 ,list(v2)) -> list(k3,v3)
> reduce: (k3 ,list(v3)) -> list(k4,v4)
>
> Karthik Kambatla
>
>
> On Sun, Nov 22, 2009 at 4:16 AM, Y G <[EMAIL PROTECTED]> wrote:
>
> > if your combiner is the same as reducer,the output type of mapper must
> the
> > same as
> > the input type of reducer.
> >
> > map: (k1,v1)  -> list(k2,v2)
> > combine: (k2 ,list(v2)) -> list(k2,v2)
> > reduce: (k2 ,list(v2)) -> list(k3,v3)
> > －－－－－
> > 天天开心
> > 身体健康
> > Sent from Guangzhou, Guangdong, China
> > Stephen Leacock<
> > http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html>
> > - "I detest life-insurance agents: they always argue that I shall some
> > day
> > die, which is not so."
> >
> > 2009/11/22 Jeff Zhang <[EMAIL PROTECTED]>
> >
> > > Hi all,
> > >
> > > As I know, Combiner is used in the mapper task, and most of the time,
> > > combiner is the same as reducer.
> > >
> > > So if combiner is used, the output type of mapper task must the same as
> > > reducer task, is it right ?
> > >
> > >
> > > Thank you
> > >
> > > Jeff Zhang
> > >
> >
>

Your combiner has to have the same output as the mapper, as it is run within
the context of the mapper.
On Sun, Nov 22, 2009 at 7:41 AM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> karthik,
>
> In your case, then the Combiner is not the same as Reducer.
> So it is not necessary to make the combiner same as reducer, right?
>
> Thank you
>
> Jeff Zhang
>
>
> 2009/11/22 Karthik Kambatla <[EMAIL PROTECTED]>
>
> > I have never tried it, but the following must also be possible.
> >
> > map: (k1,v1)  -> list(k2,v2)
> > combine: (k2 ,list(v2)) -> list(k3,v3)
> > reduce: (k3 ,list(v3)) -> list(k4,v4)
> >
> > Karthik Kambatla
> >
> >
> > On Sun, Nov 22, 2009 at 4:16 AM, Y G <[EMAIL PROTECTED]> wrote:
> >
> > > if your combiner is the same as reducer,the output type of mapper must
> > the
> > > same as
> > > the input type of reducer.
> > >
> > > map: (k1,v1)  -> list(k2,v2)
> > > combine: (k2 ,list(v2)) -> list(k2,v2)
> > > reduce: (k2 ,list(v2)) -> list(k3,v3)
> > > －－－－－
> > > 天天开心
> > > 身体健康
> > > Sent from Guangzhou, Guangdong, China
> > > Stephen Leacock<
> > > http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html>
> > > - "I detest life-insurance agents: they always argue that I shall some
> > > day
> > > die, which is not so."
> > >
> > > 2009/11/22 Jeff Zhang <[EMAIL PROTECTED]>
> > >
> > > > Hi all,
> > > >
> > > > As I know, Combiner is used in the mapper task, and most of the time,
> > > > combiner is the same as reducer.
> > > >
> > > > So if combiner is used, the output type of mapper task must the same
> as
> > > > reducer task, is it right ?
> > > >
> > > >
> > > > Thank you
> > > >
> > > > Jeff Zhang
> > > >
> > >
> >
>

--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

On Nov 22, 2009, at 7:12 AM, Karthik Kambatla wrote:

> I have never tried it, but the following must also be possible.
>
> map: (k1,v1)  -> list(k2,v2)
> combine: (k2 ,list(v2)) -> list(k3,v3)
> reduce: (k3 ,list(v3)) -> list(k4,v4)

No, this is not possible. Combiners run zero, one or many times  
depending on the details of the data. Combiners must have exactly the  
same input and output types. It naturally follows that the combiner  
input type must be the map output type and that the combiner output  
type must be the reduce input type.

map: k1,v1 -> k2,v2
combiner: k2,v2 -> k2,v2
reducer: k2,v2 -> k3,v3

There are only 3 pairs of types.

-- Owen