Hans-Peter Zorn 2013-07-30, 09:33
-Re: Capacity Scheduler on YARN
Arun C Murthy 2013-07-30, 11:52
The CS, by default, will cap a single user at the queue's capacity (not max-capacity) - this is why you are seeing that any given user cannot go beyond the queue's capacity… in other words, you will need more than one user to utilize max-capacity of the queue.
If you want to allow this behavior, you can set user-limit-factor (a multiple of queue-capacity), see yarn.scheduler.capacity.<queue-path>.user-limit-factor in:
On Jul 30, 2013, at 2:33 AM, Hans-Peter Zorn <[EMAIL PROTECTED]> wrote:
> I would like to use the capacity scheduler to implement the following requirements.
> 3 groups of users:
> * 2 of them are entitled to 50% of cluster capacity each but may use excess capacity if available
> * the third group only may use capacity that is currently not used by any of the other two.
> I set up three queues:
> yarn.scheduler.capacity.root.a.capacity 50
> yarn.scheduler.capacity.root.a.maximum-capacity 100
> yarn.scheduler.capacity.root.b.capacity 49
> yarn.scheduler.capacity.root.b.maximum-capacity 100
> yarn.scheduler.capacity.root.c.capacity 1
> yarn.scheduler.capacity.root.c.maximum-capacity 100
> These values are active, as they show up in the scheduler web gui.
> However, users of each queue are only able to use the assigned capacity, the value
> of maximum-capacity has no effect.
> Did I misunderstand the concept behind the maximum-capacity setting?
> I am using hadoop as shipped with Cloudera CDH4.3 (2.0.0-cdh4.3.0). I asked this question already on cdh-users, but the capacity scheduler seems not to be used so much with cdh I guess.
> Or is it possible to implement this using a different scheduler?
> Best & thanks,
Arun C. Murthy
Hans-Peter Zorn 2013-07-30, 12:31