Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Why don't my jobs get preempted?


Copy link to this message
-
Re: Why don't my jobs get preempted?
Edward Capriolo 2011-05-31, 19:20
On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:

> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>  As I understand it, the Fair Scheduler is preemptive. What I expect to see
> is that my long-running jobs sometimes get killed to make room for other
> people's jobs. This never happens instead my long-running jobs hog mapper
> and reducer slots and starve other people out.
>
> Am I misunderstanding how the Fair Scheduler works?
>

Try adding

                <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
                <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>

To one of your pools and see if that pool pre-empts other pools