Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Fair Scheduler question: Fair share and its effect on max capacity


Copy link to this message
-
Re: Fair Scheduler question: Fair share and its effect on max capacity
You set maxMaps to 200,  

so the maximum running mappers should be no more than 200

Best,

--
Nan Zhu
School of Computer Science,
McGill University

On Thursday, 8 November, 2012 at 6:12 PM, Matt Goeke wrote:

> Pretty straight forward question but can the fair share factor actually impact the total number of jobs / slots a pool can take up even if it is the only pool with active jobs submitted?
>
> We currently have a pool that has this configuration:
> "minMaps": 2,
> "minReduces": 1,
> "maxMaps": 200,
> "maxReduces": 66,
> "maxRunningJobs": 200,
> "minSharePreemptionTimeout": 300,
> "weight": "4.0"
>
> The total cluster capacity is over above 250 mappers but we are finding that this pool is never able to hit that max capacity for maps OR jobs even during load tests. I was about to bump the minMaps property but I wanted to confirm that this could potentially help alleviate our issue.
>
> --
> Matt
>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB