Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Scheduler functionality in MRv2


Copy link to this message
-
Re: Scheduler functionality in MRv2
Arun,

Thanks for the response.

As of now there as less articles/blogs on MRv2, so anxiously waiting for
the documents.

Praveen

On Sun, Nov 27, 2011 at 3:55 AM, Arun Murthy <[EMAIL PROTECTED]> wrote:

> Praveen,
>
> There are many ways to prevent what you described...
>
> I'm in the process of adding more docs, for now pls take a look at the
> following older blog post for more details:
>
>
> http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/
>
> Arun
>
> Sent from my iPhone
>
> On Nov 25, 2011, at 11:55 PM, Praveen Sripati <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > Let's consider the following scenario
> >
> > -> The MR Job has an InputSplit on host h1 and h2
> > -> AM makes a request to the Scheduler for a container on h1 and h2
> > -> The scheduler responds with containers c1 and c2 on h1 and h2
> > -> But the AM uses c1 and releases c2 after 15 minutes
> >
> > In this scenario c2 cannot be allocated to any of the AM by the scheduler
> > for that span of 15 minutes and the c2 remains idle. And the released
> > container is also not accounted into the Queues quota. How can such
> > scenario be avoided?
> >
> > Also, it would be nice to specify a more complex requests to the
> scheduler,
> > like a request for (h1 or h2), (r1 or r2). Add some dependencies between
> > the requests.
> >
> > Thanks,
> > Praveen
>