Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: Capacity Scheduler Issue


Copy link to this message
-
Re: Capacity Scheduler Issue
Munna 2013-11-28, 12:40
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.

Thanks to all...

Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <[EMAIL PROTECTED]> wrote:

> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[EMAIL PROTECTED]> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)

*Regards*

*Munna*