Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Pipelining data from map to reduce


+
bharath v 2010-03-04, 17:00
+
E. Sammer 2010-03-04, 17:06
Copy link to this message
-
Re: Pipelining data from map to reduce
Bharath,

This idea is  kicking around in academia.. not made into apache yet..
https://issues.apache.org/jira/browse/MAPREDUCE-1211

You can get a working prototype from:
http://code.google.com/p/hop/

Ashutosh

On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 3/4/10 12:00 PM, bharath v wrote:
>>
>> Hi ,
>>
>> Can we pipeline the map output directly into reduce phase without
>> storing it in the local filesystem (avoiding disk IOs).
>> If yes , how to do that ?
>
> Bharath:
>
> No, there's no way to avoid going to disk after the mappers.
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>
+
Jeff Hammerbacher 2010-03-04, 22:18
+
Edward Capriolo 2010-03-04, 22:32
+
Scott Carey 2010-03-04, 23:46
+
Edward Capriolo 2010-03-05, 01:26
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB