Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Capacity Scheduler on YARN


Copy link to this message
-
Re: Capacity Scheduler on YARN
Hans-Peter,

 The CS, by default, will cap a single user at the queue's capacity (not max-capacity) - this is why you are seeing that any given user cannot go beyond the queue's capacity… in other words, you will need more than one user to utilize max-capacity of the queue.

 If you want to allow this behavior, you can set user-limit-factor (a multiple of queue-capacity), see yarn.scheduler.capacity.<queue-path>.user-limit-factor in:
 http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

hth,
Arun

On Jul 30, 2013, at 2:33 AM, Hans-Peter Zorn <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I would like to use the capacity scheduler to implement the following requirements.
>
> 3 groups of users:
> * 2 of them are entitled to 50% of cluster capacity each but may use excess capacity if available
> * the third group only may use capacity that is currently not used by any of the other two.
>
> I set up three queues:
> yarn.scheduler.capacity.root.a.capacity 50
> yarn.scheduler.capacity.root.a.maximum-capacity 100
>
> yarn.scheduler.capacity.root.b.capacity 49
> yarn.scheduler.capacity.root.b.maximum-capacity 100
>
> yarn.scheduler.capacity.root.c.capacity 1
> yarn.scheduler.capacity.root.c.maximum-capacity 100
>
> These values are active, as they show up in the scheduler web gui.
>
> However, users of each queue are only able to use the assigned capacity, the value
> of maximum-capacity has no effect.
>
> Did I misunderstand the concept behind the maximum-capacity setting?
>
> I am using hadoop as shipped with Cloudera CDH4.3  (2.0.0-cdh4.3.0). I asked this question already on cdh-users, but the capacity scheduler seems not to be used so much with cdh I guess.
>
> Or is it possible to implement this using a different scheduler?
>
> Best & thanks,
> Hans-Peter

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB