Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Pipelining data from map to reduce


Copy link to this message
-
Re: Pipelining data from map to reduce
Bharath,

This idea is  kicking around in academia.. not made into apache yet..
https://issues.apache.org/jira/browse/MAPREDUCE-1211

You can get a working prototype from:
http://code.google.com/p/hop/

Ashutosh

On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 3/4/10 12:00 PM, bharath v wrote:
>>
>> Hi ,
>>
>> Can we pipeline the map output directly into reduce phase without
>> storing it in the local filesystem (avoiding disk IOs).
>> If yes , how to do that ?
>
> Bharath:
>
> No, there's no way to avoid going to disk after the mappers.
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB