Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How mapred.map.child.java.opts influence the Container resource allocation


+
YouPeng Yang 2013-02-04, 11:38
+
Arun C Murthy 2013-02-04, 15:43
Copy link to this message
-
Re: How mapred.map.child.java.opts influence the Container resource allocation
Hi  Arun

    Thanks a lot.

    With a container size of 2G, the max heap size i can allocate is ~1.6G.
    is the heap exclusively possessed by the container ,or shared by all
containers
which belong to a AM.

     I also notice that there are several containers  allocated in a NM.
     however only one or a few of them would process the job,
     (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING -->
EXITED_WITH_SUCCESS) ,
     Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE).
     So i wonder  what is the container allocation policy,and how many
containers  would be allocated  when a job is submitted.

Regards


2013/2/4 Arun C Murthy <[EMAIL PROTECTED]>

> Youpeng,
>
>  The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>
>  mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>
>  So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>
>  Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>
>  So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
>
> hth,
> Arun
>
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
>
> Hi All
>
>     I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and  Container resource  allocation.
>     if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
>      what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
>
>
>  --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>