I'm launching long-running tasks on a cluster running the Fair Scheduler.
 As I understand it, the Fair Scheduler is preemptive. What I expect to see
is that my long-running jobs sometimes get killed to make room for other
people's jobs. This never happens instead my long-running jobs hog mapper
and reducer slots and starve other people out.

Am I misunderstanding how the Fair Scheduler works?

On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:

> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>  As I understand it, the Fair Scheduler is preemptive. What I expect to see
> is that my long-running jobs sometimes get killed to make room for other
> people's jobs. This never happens instead my long-running jobs hog mapper
> and reducer slots and starve other people out.
>
> Am I misunderstanding how the Fair Scheduler works?
>

Try adding

                <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
                <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>

To one of your pools and see if that pool pre-empts other pools

Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that have applied that patch, such as Cloudera's distribution. If you are running one of these, check out http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for information on how to enable preemption.

Matei

On May 31, 2011, at 12:20 PM, Edward Capriolo wrote:

> On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:
>
>> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>> As I understand it, the Fair Scheduler is preemptive. What I expect to see
>> is that my long-running jobs sometimes get killed to make room for other
>> people's jobs. This never happens instead my long-running jobs hog mapper
>> and reducer slots and starve other people out.
>>
>> Am I misunderstanding how the Fair Scheduler works?
>>
>
> Try adding
>
>                <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
>                <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>
>
> To one of your pools and see if that pool pre-empts other pools

Sorry, I meant 0.21+, not 0.20+ for the Apache releases.

Matei

On May 31, 2011, at 4:05 PM, Matei Zaharia wrote:

> Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that have applied that patch, such as Cloudera's distribution. If you are running one of these, check out http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for information on how to enable preemption.
>
> Matei
>
> On May 31, 2011, at 12:20 PM, Edward Capriolo wrote:
>
>> On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:
>>
>>> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>>> As I understand it, the Fair Scheduler is preemptive. What I expect to see
>>> is that my long-running jobs sometimes get killed to make room for other
>>> people's jobs. This never happens instead my long-running jobs hog mapper
>>> and reducer slots and starve other people out.
>>>
>>> Am I misunderstanding how the Fair Scheduler works?
>>>
>>
>> Try adding
>>
>>               <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
>>               <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>
>>
>> To one of your pools and see if that pool pre-empts other pools
>