Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Why don't my jobs get preempted?


Copy link to this message
-
Re: Why don't my jobs get preempted?
On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:

> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>  As I understand it, the Fair Scheduler is preemptive. What I expect to see
> is that my long-running jobs sometimes get killed to make room for other
> people's jobs. This never happens instead my long-running jobs hog mapper
> and reducer slots and starve other people out.
>
> Am I misunderstanding how the Fair Scheduler works?
>

Try adding

                <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
                <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>

To one of your pools and see if that pool pre-empts other pools
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB