Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Capacity Scheduler on YARN


Copy link to this message
-
Re: Capacity Scheduler on YARN
Hans-Peter,

 The CS, by default, will cap a single user at the queue's capacity (not max-capacity) - this is why you are seeing that any given user cannot go beyond the queue's capacity‚Ķ in other words, you will need more than one user to utilize max-capacity of the queue.

 If you want to allow this behavior, you can set user-limit-factor (a multiple of queue-capacity), see yarn.scheduler.capacity.<queue-path>.user-limit-factor in:
 http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

hth,
Arun

On Jul 30, 2013, at 2:33 AM, Hans-Peter Zorn <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I would like to use the capacity scheduler to implement the following requirements.
>
> 3 groups of users:
> * 2 of them are entitled to 50% of cluster capacity each but may use excess capacity if available
> * the third group only may use capacity that is currently not used by any of the other two.
>
> I set up three queues:
> yarn.scheduler.capacity.root.a.capacity 50
> yarn.scheduler.capacity.root.a.maximum-capacity 100
>
> yarn.scheduler.capacity.root.b.capacity 49
> yarn.scheduler.capacity.root.b.maximum-capacity 100
>
> yarn.scheduler.capacity.root.c.capacity 1
> yarn.scheduler.capacity.root.c.maximum-capacity 100
>
> These values are active, as they show up in the scheduler web gui.
>
> However, users of each queue are only able to use the assigned capacity, the value
> of maximum-capacity has no effect.
>
> Did I misunderstand the concept behind the maximum-capacity setting?
>
> I am using hadoop as shipped with Cloudera CDH4.3  (2.0.0-cdh4.3.0). I asked this question already on cdh-users, but the capacity scheduler seems not to be used so much with cdh I guess.
>
> Or is it possible to implement this using a different scheduler?
>
> Best & thanks,
> Hans-Peter

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/