Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Capacity Scheduler Issue


Copy link to this message
-
Re: Capacity Scheduler Issue
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.

Thanks to all...

Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <[EMAIL PROTECTED]> wrote:

> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[EMAIL PROTECTED]> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)

*Regards*

*Munna*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB