Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Pipelining data from map to reduce


Copy link to this message
-
Re: Pipelining data from map to reduce
Ashutosh Chauhan 2010-03-04, 19:41
Bharath,

This idea is  kicking around in academia.. not made into apache yet..
https://issues.apache.org/jira/browse/MAPREDUCE-1211

You can get a working prototype from:
http://code.google.com/p/hop/

Ashutosh

On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 3/4/10 12:00 PM, bharath v wrote:
>>
>> Hi ,
>>
>> Can we pipeline the map output directly into reduce phase without
>> storing it in the local filesystem (avoiding disk IOs).
>> If yes , how to do that ?
>
> Bharath:
>
> No, there's no way to avoid going to disk after the mappers.
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>