Hi,

What is the difference between setting the mapred.job.map.memory.mb and
mapred.child.java.opts using -Xmx to control the maximum memory used by a
Mapper and Reduce task? Which one takes precedence?

Thanks,
Praveen

Hi,

Can someone please clarify me on the below query?

Thanks,
Praveen

On Sun, Nov 6, 2011 at 8:47 PM, Praveen Sripati <[EMAIL PROTECTED]>wrote:

>
> Hi,
>
> What is the difference between setting the mapred.job.map.memory.mb and
> mapred.child.java.opts using -Xmx to control the maximum memory used by a
> Mapper and Reduce task? Which one takes precedence?
>
> Thanks,
> Praveen
>

Praveen,

The mapred.job.map.memory.mb is a config that Hadoop uses to decided if your task is using too much memory and needs to be killed.  The memory used by this includes not only the memory for java itself but also any other child processes that may be launched by java.  The -Xmx in mapred.child.java.opts tells the JVM being launched how big of a heap to use.  Typically java itself will use a bit more memory then just what is stored in the heap so -Xmx should be slightly smaller then mapred.job.map.memory.mb.  But, if you are going to be launching sub-processes, like streaming tends to do, then you will typically want to have -Xmx be much smaller then mapred.job.map.memory.mb so that it can allow for the RAM that the native process will be using.

--Bobby Evans

On 11/7/11 7:29 PM, "Praveen Sripati" <[EMAIL PROTECTED]> wrote:

Hi,

Can someone please clarify me on the below query?

Thanks,
Praveen

On Sun, Nov 6, 2011 at 8:47 PM, Praveen Sripati <[EMAIL PROTECTED]> wrote:

Hi,

What is the difference between setting the mapred.job.map.memory.mb and mapred.child.java.opts using -Xmx to control the maximum memory used by a Mapper and Reduce task? Which one takes precedence?

Thanks,
Praveen