Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Do we shoot ourselves by using all task slots?


Copy link to this message
-
Re: Do we shoot ourselves by using all task slots?

On May 28, 2010, at 11:43 AM, Todd Lipcon wrote:

> Hi Allen,
>
> Recent versions of the fair scheduler have configurations for "delay scheduling" - essentially, it will wait for a few seconds when a slot opens up to try to find a local task before assigning a non-local one. This is specifically to avoid the issue you're describing.
>
> Check out Matei's Eurosys 2010 paper here:
>
> http://www.cs.berkeley.edu/~matei/papers/2010/eurosys_delay_scheduling.pdf
>
> I believe this got lumped in with MAPREDUCE-706.
Thanks.  I'll take a look at this and see how applicable this is to Capacity.  
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB