Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Using FairScheduler to limit # of tasks


Copy link to this message
-
Using FairScheduler to limit # of tasks
Can I use the FairScheduler to limit the number of map/reduce tasks directly
from the job configuration? E.g. I have 1 job that I know should run a more
limited # of map/reduce tasks than is set as the default, I want to
configure a queue with a limited # of map/reduce tasks, but only apply it to
that job, I don't want to deploy this queue configuration to the cluster.

 

Assuming the above answer is 'yes', if I were to limit the # of map tasks to
10 in a cluster of 10 nodes, would the fair scheduler tend to distribute
those 10 map tasks evenly across the nodes (assuming a cluster that's
otherwise unused at the moment), or would it be prone to over-loading a
single node just because those are the first open slots it sees?

 

David

 

+
Michel Segel 2013-05-13, 11:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB