Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Capacity Scheduler Issue


Copy link to this message
-
Re: Capacity Scheduler Issue
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.

Jian
On Wed, Nov 27, 2013 at 9:42 AM, Munna <[EMAIL PROTECTED]> wrote:

> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB