Hi ,

Can we pipeline the map output directly into reduce phase without
storing it in the local filesystem (avoiding disk IOs).
If yes , how to do that ?

Any help is highly appreciated.

Thanks

On 3/4/10 12:00 PM, bharath v wrote:
> Hi ,
>
> Can we pipeline the map output directly into reduce phase without
> storing it in the local filesystem (avoiding disk IOs).
> If yes , how to do that ?

Bharath:

No, there's no way to avoid going to disk after the mappers.

--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com

Bharath,

This idea is  kicking around in academia.. not made into apache yet..
https://issues.apache.org/jira/browse/MAPREDUCE-1211

You can get a working prototype from:
http://code.google.com/p/hop/

Ashutosh

On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 3/4/10 12:00 PM, bharath v wrote:
>>
>> Hi ,
>>
>> Can we pipeline the map output directly into reduce phase without
>> storing it in the local filesystem (avoiding disk IOs).
>> If yes , how to do that ?
>
> Bharath:
>
> No, there's no way to avoid going to disk after the mappers.
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>

Also see "Breaking the MapReduce Stage Barrier" from UIUC:
http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf

On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan <
[EMAIL PROTECTED]> wrote:

> Bharath,
>
> This idea is  kicking around in academia.. not made into apache yet..
> https://issues.apache.org/jira/browse/MAPREDUCE-1211
>
> You can get a working prototype from:
> http://code.google.com/p/hop/
>
> Ashutosh
>
> On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
> > On 3/4/10 12:00 PM, bharath v wrote:
> >>
> >> Hi ,
> >>
> >> Can we pipeline the map output directly into reduce phase without
> >> storing it in the local filesystem (avoiding disk IOs).
> >> If yes , how to do that ?
> >
> > Bharath:
> >
> > No, there's no way to avoid going to disk after the mappers.
> >
> > --
> > Eric Sammer
> > [EMAIL PROTECTED]
> > http://esammer.blogspot.com
> >
>

On Thu, Mar 4, 2010 at 5:18 PM, Jeff Hammerbacher <[EMAIL PROTECTED]> wrote:
> Also see "Breaking the MapReduce Stage Barrier" from UIUC:
> http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf
>
> On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan <
> [EMAIL PROTECTED]> wrote:
>
>> Bharath,
>>
>> This idea is  kicking around in academia.. not made into apache yet..
>> https://issues.apache.org/jira/browse/MAPREDUCE-1211
>>
>> You can get a working prototype from:
>> http://code.google.com/p/hop/
>>
>> Ashutosh
>>
>> On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
>> > On 3/4/10 12:00 PM, bharath v wrote:
>> >>
>> >> Hi ,
>> >>
>> >> Can we pipeline the map output directly into reduce phase without
>> >> storing it in the local filesystem (avoiding disk IOs).
>> >> If yes , how to do that ?
>> >
>> > Bharath:
>> >
>> > No, there's no way to avoid going to disk after the mappers.
>> >
>> > --
>> > Eric Sammer
>> > [EMAIL PROTECTED]
>> > http://esammer.blogspot.com
>> >
>>
>

Jeff  "I have every cool thing on the internet bookmarked"
Hammerbacher, strikes again.

Interesting concept.  In some cases the reducer can not run unless it
has ALL the values for that key. How can you be sure you have all the
values for that key until all the maps are done? Anything that works
with combiner should be able to support this however.

Interesting article.  It claims to have the same fault tolerance but I don't see any explanation of how that can be.  

If a single mapper fails part-way through a task when it has transmitted partial results to a reducer, the whole job is corrupted.  With the current barrier between map and reduce, a job can recover from partially completed tasks and speculatively execute.

I would imagine that small low latency tasks can benefit greatly from such an approach, but larger tasks need the barrier or will not be very fault tolerant.  However, there is still a lot of optimizations to dot in Hadoop for low latency tasks while maintaining the barrier.
On Mar 4, 2010, at 2:18 PM, Jeff Hammerbacher wrote:

> Also see "Breaking the MapReduce Stage Barrier" from UIUC:
> http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf
>
> On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan <
> [EMAIL PROTECTED]> wrote:
>
>> Bharath,
>>
>> This idea is  kicking around in academia.. not made into apache yet..
>> https://issues.apache.org/jira/browse/MAPREDUCE-1211
>>
>> You can get a working prototype from:
>> http://code.google.com/p/hop/
>>
>> Ashutosh
>>
>> On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
>>> On 3/4/10 12:00 PM, bharath v wrote:
>>>>
>>>> Hi ,
>>>>
>>>> Can we pipeline the map output directly into reduce phase without
>>>> storing it in the local filesystem (avoiding disk IOs).
>>>> If yes , how to do that ?
>>>
>>> Bharath:
>>>
>>> No, there's no way to avoid going to disk after the mappers.
>>>
>>> --
>>> Eric Sammer
>>> [EMAIL PROTECTED]
>>> http://esammer.blogspot.com
>>>
>>

I guess if you emmitted the key as task-id+ key you would have more
overhead but if the data "replayed" the reducer could detect dups.

Ed

On 3/4/10, Scott Carey <[EMAIL PROTECTED]> wrote:
> Interesting article.  It claims to have the same fault tolerance but I don't
> see any explanation of how that can be.
>
> If a single mapper fails part-way through a task when it has transmitted
> partial results to a reducer, the whole job is corrupted.  With the current
> barrier between map and reduce, a job can recover from partially completed
> tasks and speculatively execute.
>
> I would imagine that small low latency tasks can benefit greatly from such
> an approach, but larger tasks need the barrier or will not be very fault
> tolerant.  However, there is still a lot of optimizations to dot in Hadoop
> for low latency tasks while maintaining the barrier.
>
>
> On Mar 4, 2010, at 2:18 PM, Jeff Hammerbacher wrote:
>
>> Also see "Breaking the MapReduce Stage Barrier" from UIUC:
>> http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf
>>
>> On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Bharath,
>>>
>>> This idea is  kicking around in academia.. not made into apache yet..
>>> https://issues.apache.org/jira/browse/MAPREDUCE-1211
>>>
>>> You can get a working prototype from:
>>> http://code.google.com/p/hop/
>>>
>>> Ashutosh
>>>
>>> On Thu, Mar 4, 2010 at 09:06, E. Sammer <[EMAIL PROTECTED]> wrote:
>>>> On 3/4/10 12:00 PM, bharath v wrote:
>>>>>
>>>>> Hi ,
>>>>>
>>>>> Can we pipeline the map output directly into reduce phase without
>>>>> storing it in the local filesystem (avoiding disk IOs).
>>>>> If yes , how to do that ?
>>>>
>>>> Bharath:
>>>>
>>>> No, there's no way to avoid going to disk after the mappers.
>>>>
>>>> --
>>>> Eric Sammer
>>>> [EMAIL PROTECTED]
>>>> http://esammer.blogspot.com
>>>>
>>>
>
>