Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Question related to resource allocation in Yarn!


+
Rahul Bhattacharjee 2013-09-05, 15:03
+
Rahul Bhattacharjee 2013-09-06, 04:13
+
Devaraj k 2013-09-06, 05:01
Copy link to this message
-
Re: Question related to resource allocation in Yarn!
Hi Devaraj,

I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.

There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.

2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available

I am requesting containers with 100 mb mem and 1 core. If  I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.

Thanks for the help!
Rahul

On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <[EMAIL PROTECTED]> wrote:

>  Hi Rahul,****
>
> ** **
>
>     Could you tell me, what is the version you are using?****
>
>      ****
>
> **·         **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:[EMAIL PROTECTED]]
> *Sent:* 06 September 2013 09:43
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch  a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>