Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Limit on total jobs running using fair scheduler


Copy link to this message
-
Re: Limit on total jobs running using fair scheduler
fot MRv1, Its impossible.
 On 2013-11-22 5:46 AM, "Ivan Tretyakov" <[EMAIL PROTECTED]>
wrote:

> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools.  In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <[EMAIL PROTECTED]>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> [EMAIL PROTECTED]
>