Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How mapred.map.child.java.opts influence the Container resource allocation

YouPeng Yang 2013-02-04, 11:38
Arun C Murthy 2013-02-04, 15:43
Copy link to this message
Re: How mapred.map.child.java.opts influence the Container resource allocation
Hi  Arun

    Thanks a lot.

    With a container size of 2G, the max heap size i can allocate is ~1.6G.
    is the heap exclusively possessed by the container ,or shared by all
which belong to a AM.

     I also notice that there are several containers  allocated in a NM.
     however only one or a few of them would process the job,
     Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
     So i wonder  what is the container allocation policy,and how many
containers  would be allocated  when a job is submitted.


2013/2/4 Arun C Murthy <[EMAIL PROTECTED]>

> Youpeng,
>  The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>  mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>  So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>  Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>  So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
> hth,
> Arun
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
> Hi All
>     I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and  Container resource  allocation.
>     if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
>      what about mapred.reduce.child.java.opts?
> regards
> Youpeng Yang
>  --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/