In addition to default FIFO scheduler, there are fair scheduler and capacity scheduler. In some sense, fair scheduler can be considered a user-based scheduling while capacity scheduler does a queue-based scheduling. Is there or will there be a hybrid scheduler that combines the good parts of the two (or a capacity scheduler that allows preemption, then different users are asked to submit jobs to different queues, in this way implicitly follow user-based scheduling as well, more or less)?

Thanks,

--Michael
      

You can write your own one based on them. They are open source.

On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <[EMAIL PROTECTED]> wrote:

> In addition to default FIFO scheduler, there are fair scheduler and
> capacity scheduler. In some sense, fair scheduler can be considered a
> user-based scheduling while capacity scheduler does a queue-based
> scheduling. Is there or will there be a hybrid scheduler that combines the
> good parts of the two (or a capacity scheduler that allows preemption, then
> different users are asked to submit jobs to different queues, in this way
> implicitly follow user-based scheduling as well, more or less)?
>
> Thanks,
>
> --Michael
>
>
>

Yeah, will give it a try :)

Thanks,
-Michael

--- On Mon, 6/28/10, He Chen <[EMAIL PROTECTED]> wrote:

From: He Chen <[EMAIL PROTECTED]>
Subject: Re: hybrid map/reducer scheduler?
To: [EMAIL PROTECTED]
Date: Monday, June 28, 2010, 7:09 PM

You can write your own one based on them. They are open source.

On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <[EMAIL PROTECTED]> wrote:

> In addition to default FIFO scheduler, there are fair scheduler and
> capacity scheduler. In some sense, fair scheduler can be considered a
> user-based scheduling while capacity scheduler does a queue-based
> scheduling. Is there or will there be a hybrid scheduler that combines the
> good parts of the two (or a capacity scheduler that allows preemption, then
> different users are asked to submit jobs to different queues, in this way
> implicitly follow user-based scheduling as well, more or less)?
>
> Thanks,
>
> --Michael
>
>
>

      

Michael,

> In addition to default FIFO scheduler, there are fair scheduler and capacity scheduler. In some sense, fair scheduler can be considered a user-based scheduling while capacity scheduler does a queue-based scheduling. Is there or will there be a hybrid scheduler that combines the good parts of the two (or a capacity scheduler that allows preemption, then different users are asked to submit jobs to different queues, in this way implicitly follow user-based scheduling as well, more or less)?

What user specific features are looking on top of capacity scheduler ?
Please note that the capacity scheduler already supports some user
specific concepts like limits per user per queue.

Thanks
hemanth

jiang licht wrote:
> Yeah, will give it a try :)
>
> Thanks,
> -Michael

Hadoop 0.21 has the buy-cpu-time scheduler too, which does have
preemption and is apparently simpler than the main schedulers, which
Yahoo! and facebook use. Given the performance of the main schedulers
matters a lot to those two companies, they will be nervous about any
patch which could impact their performance. Starting off with the
simpler code (and using the SVN_TRUNK scheduler plugin API) would be a
good idea.

Any documentation you can make on writing/testing schedulers would be
nice too, its a good plugin point, and datacentre scheduling is right up
there with VM and data placement as cutting edge problems

>
> --- On Mon, 6/28/10, He Chen <[EMAIL PROTECTED]> wrote:
>
> From: He Chen <[EMAIL PROTECTED]>
> Subject: Re: hybrid map/reducer scheduler?
> To: [EMAIL PROTECTED]
> Date: Monday, June 28, 2010, 7:09 PM
>
> You can write your own one based on them. They are open source.
>
> On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <[EMAIL PROTECTED]> wrote:
>
>> In addition to default FIFO scheduler, there are fair scheduler and
>> capacity scheduler. In some sense, fair scheduler can be considered a
>> user-based scheduling while capacity scheduler does a queue-based
>> scheduling. Is there or will there be a hybrid scheduler that combines the
>> good parts of the two (or a capacity scheduler that allows preemption, then
>> different users are asked to submit jobs to different queues, in this way
>> implicitly follow user-based scheduling as well, more or less)?
>>
>> Thanks,
>>
>> --Michael
>>
>>
>>
>
>
>
>