Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Child JVM memory allocation / Usage


Copy link to this message
-
Re: Child JVM memory allocation / Usage
Harsh J 2013-03-25, 04:56
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I configured  my child jvm heap to 2 GB. So, I thought I could really read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println("FREE MEMORY -- "
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println("MAX MEMORY ---" + Runtime.getRuntime().maxMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864  = 320 MB
> MAX MEMORY ---1908932608  = 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of heap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>

--
Harsh J