Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Can I have multiple reducers?


Copy link to this message
-
Re: Can I have multiple reducers?
Aaron Kimball 2009-10-23, 02:54
If you need another shuffle after your first reduce pass, then you need a
second MapReduce job to run after the first one. Just use an IdentityMapper.

This is a reasonably common situation.
- Aaron

On Thu, Oct 22, 2009 at 4:17 PM, Forhadoop <[EMAIL PROTECTED]> wrote:

>
> Hello,
>
> In my application I need to reduce the original reducer output keys
> further.
>
> I was reading about Chainreducer and Chainmappers but looks like it is for
> :
> one or more mapper -> reducer -> 0 or more mappers
>
> I need something like:
> one or more mapper -> reducer -> reducer
>
> Please help me figure out the best way to achieve it. Currently, the only
> options seems like I write another map reduce application and run it
> separately after the first map-reduce application. In this second
> application, the mapper will be dummy and won't do anything. The reducer
> will further club the first run outputs.
>
> Any other comments such as this is not a good programming practice are
> welcome, so that I know I am in the wrong direction..
> --
> View this message in context:
> http://www.nabble.com/Can-I-have-multiple-reducers--tp26018722p26018722.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>