Hadoop, mail # user - 33%, 66%, and 100% *reducer* optimization

33%, 66%, and 100% *reducer* optimization

Jay Vyas 2012-10-15, 21:32

Hi guys !

We all know that there are major milestones in reducers (33%, 66%)

In particular:

Im wondering: What should we optimize for in case a job hangs at 33%, as
compared with 66% ?

--
Jay Vyas
http://jayunit100.blogspot.com

Re: 33%, 66%, and 100% *reducer* optimization

Vinod Kumar Vavilapalli 2012-10-15, 23:33

Reduce has three phases - shuffle, sort and reduce.

So, 33% would imply the shuffle phase end, and 66% would refer to the end of sort phase.

Thanks,
+Vinod

On Oct 15, 2012, at 2:32 PM, Jay Vyas wrote:

> Hi guys !
>
> We all know that there are major milestones in reducers (33%, 66%)
>
> In particular:
>
> Im wondering: What should we optimize for in case a job hangs at 33%, as
> compared with 66% ?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com