Hi,

Let's consider the following scenario

-> The MR Job has an InputSplit on host h1 and h2
-> AM makes a request to the Scheduler for a container on h1 and h2
-> The scheduler responds with containers c1 and c2 on h1 and h2
-> But the AM uses c1 and releases c2 after 15 minutes

In this scenario c2 cannot be allocated to any of the AM by the scheduler
for that span of 15 minutes and the c2 remains idle. And the released
container is also not accounted into the Queues quota. How can such
scenario be avoided?

Also, it would be nice to specify a more complex requests to the scheduler,
like a request for (h1 or h2), (r1 or r2). Add some dependencies between
the requests.

Thanks,
Praveen

Praveen,

There are many ways to prevent what you described...

I'm in the process of adding more docs, for now pls take a look at the
following older blog post for more details:

http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/

Arun

Sent from my iPhone

On Nov 25, 2011, at 11:55 PM, Praveen Sripati <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Let's consider the following scenario
>
> -> The MR Job has an InputSplit on host h1 and h2
> -> AM makes a request to the Scheduler for a container on h1 and h2
> -> The scheduler responds with containers c1 and c2 on h1 and h2
> -> But the AM uses c1 and releases c2 after 15 minutes
>
> In this scenario c2 cannot be allocated to any of the AM by the scheduler
> for that span of 15 minutes and the c2 remains idle. And the released
> container is also not accounted into the Queues quota. How can such
> scenario be avoided?
>
> Also, it would be nice to specify a more complex requests to the scheduler,
> like a request for (h1 or h2), (r1 or r2). Add some dependencies between
> the requests.
>
> Thanks,
> Praveen

Arun,

Thanks for the response.

As of now there as less articles/blogs on MRv2, so anxiously waiting for
the documents.

Praveen

On Sun, Nov 27, 2011 at 3:55 AM, Arun Murthy <[EMAIL PROTECTED]> wrote:

> Praveen,
>
> There are many ways to prevent what you described...
>
> I'm in the process of adding more docs, for now pls take a look at the
> following older blog post for more details:
>
>
> http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/
>
> Arun
>
> Sent from my iPhone
>
> On Nov 25, 2011, at 11:55 PM, Praveen Sripati <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > Let's consider the following scenario
> >
> > -> The MR Job has an InputSplit on host h1 and h2
> > -> AM makes a request to the Scheduler for a container on h1 and h2
> > -> The scheduler responds with containers c1 and c2 on h1 and h2
> > -> But the AM uses c1 and releases c2 after 15 minutes
> >
> > In this scenario c2 cannot be allocated to any of the AM by the scheduler
> > for that span of 15 minutes and the c2 remains idle. And the released
> > container is also not accounted into the Queues quota. How can such
> > scenario be avoided?
> >
> > Also, it would be nice to specify a more complex requests to the
> scheduler,
> > like a request for (h1 or h2), (r1 or r2). Add some dependencies between
> > the requests.
> >
> > Thanks,
> > Praveen
>