Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Capacity Scheduler Issue


+
Munna 2013-11-27, 17:42
+
Adam Kawa 2013-11-28, 20:23
Copy link to this message
-
Re: Capacity Scheduler Issue
Did the application get accepted ? If the application gets accepted but
just not able to run, as I said, try increasing
yarn.scheduler.capacity.maximum-am-resource-percent
to a much larger value, maybe 0.8,  just for troubleshoot. How large it
should be depends on your requirements.

yarn.scheduler.capacity.maximum-am-resource-percent  controls the max
number of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs.

I have seen both of your queues only allow 1 active application.
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]

Jian
On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <[EMAIL PROTECTED]> wrote:

> I see that you have different settings for ACL:
>
> <name>yarn.scheduler.capacity.root.*default*
> .acl_submit_applications</name><value>*yarn,mapred*
> </value></property><property>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE:  [> configuredAcls ]
>
> <name>yarn.scheduler.capacity.root.*dev*
> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE:  [= configuredAcls ]
>
> Do you submit your jobs to dev queue using good user accounts? Did you
> include a right config xml in a previous post? (xml says *smunnavar,test*,
> but RM loads user,test).
>
> As a first troubleshooting steps, could you disable ACL for dev queue to
> check if you can submit a job there?
>
>   <property>
>     <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
>     <value>*</value>
>   </property>
>
>
>
> 2013/11/28 Munna <[EMAIL PROTECTED]>
>
>> Hi,
>>
>>
>> I think there is no solution on above issue, so i'll move to fair
>> scheduler.
>>
>> Thanks to all...
>>
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <[EMAIL PROTECTED]> wrote:
>>
>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>
>>>
>>> what is the best value?
>>>
>>> Tx,
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[EMAIL PROTECTED]> wrote:
>>>
>>>> The log shows the both queues are properly picked up by the RM.
>>>> If the problem is that your submitted application is not able to run,
>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>
>>>> Jian
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi Flocks,
>>>>>
>>>>>
>>>>>
>>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>>> how I have struggling L….
>>>>>
>>>>>
>>>>>
>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>>> shown in RM UI.
>>>>>
>>>>>
>>>>>
>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>> which showing all configured Q’s.
>>>>>
>>>>>
>>>>>
>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>> some idea about that, based on his views I dig more.
>>>>>
>>>>>
>>>>>
>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>> the jobs and its unable to identifying users Queue.
>>>>>
>>>>>
>>>>>
>>>>> I have attached Cap Scheduler configuration file for your information.
>>>>> Some O/P for ur information.
>>>>>
>>>>>
>>>>>
>>>>> *[user@host ~]$ mapred queue -list*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
+
Munna 2013-11-30, 05:53