Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Question related to resource allocation in Yarn!


+
Rahul Bhattacharjee 2013-09-05, 15:03
+
Rahul Bhattacharjee 2013-09-06, 04:13
+
Devaraj k 2013-09-06, 05:01
Copy link to this message
-
Re: Question related to resource allocation in Yarn!
Hi Devaraj,

I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.

There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.

2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available

I am requesting containers with 100 mb mem and 1 core. If  I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.

Thanks for the help!
Rahul

On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <[EMAIL PROTECTED]> wrote:

>  Hi Rahul,****
>
> ** **
>
>     Could you tell me, what is the version you are using?****
>
>      ****
>
> **·         **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:[EMAIL PROTECTED]]
> *Sent:* 06 September 2013 09:43
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch  a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB