Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Can I have multiple reducers?


Copy link to this message
-
Re: Can I have multiple reducers?
If you need another shuffle after your first reduce pass, then you need a
second MapReduce job to run after the first one. Just use an IdentityMapper.

This is a reasonably common situation.
- Aaron

On Thu, Oct 22, 2009 at 4:17 PM, Forhadoop <[EMAIL PROTECTED]> wrote:

>
> Hello,
>
> In my application I need to reduce the original reducer output keys
> further.
>
> I was reading about Chainreducer and Chainmappers but looks like it is for
> :
> one or more mapper -> reducer -> 0 or more mappers
>
> I need something like:
> one or more mapper -> reducer -> reducer
>
> Please help me figure out the best way to achieve it. Currently, the only
> options seems like I write another map reduce application and run it
> separately after the first map-reduce application. In this second
> application, the mapper will be dummy and won't do anything. The reducer
> will further club the first run outputs.
>
> Any other comments such as this is not a good programming practice are
> welcome, so that I know I am in the wrong direction..
> --
> View this message in context:
> http://www.nabble.com/Can-I-have-multiple-reducers--tp26018722p26018722.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB