Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Heap Size question.


Copy link to this message
-
Re: Heap Size question.
Joey Echeverria 2011-06-01, 12:01
The values show you the maximum heap size and currently used heap of the job tracker, not running jobs. Furthermore, the HADOOP_HEAPSIZE setting only sets the maximum heap for the daemons, not the tasks in your job.

If you're getting OOMEs, you should add a setting to your mapred-site.xml file that looks like this:

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx1g</value>
</property>

-Joey

On Jun 1, 2011, at 7:35, Ken Williams <[EMAIL PROTECTED]> wrote:

>
>
>  Hi All,
>
>  I'm a bit confused about the values displayed on the 'jobtracker.jsp' page.
>  In particular, there's a section called 'Cluster Summary'.
>
>  I'm running a small 4-machine Hadoop cluster, and when I point a web-browser
>  at my master machine (http://master:50030/jobtracker.jsp) it displays,
>
>               Cluster Summary (Heap Size is 15.5 MB / 1.89 GB)
>
>   What exactly do these figures mean ?
>
>   I know that the second figure (1.89 GB) is determined by the value of
>   the HADOOP_HEAPSIZE variable set in  'conf/hadoop-env.sh'. What I'm not
>   sure about is exactly what it means, or where the first value (15.5 MB) is determined
>   or what it means.
>
>   I'm guessing the 1.89 GB is the amount of heap-memory allocated to Hadoop
>   on each machine in the cluster. (Correct ?)
>
>   I have no idea what the 15.5 MB means or where it comes from. It never changes,
>   not even when a job is running, and I can't find any explanation in the documentation.
>
>   This page, https://issues.apache.org/jira/browse/HADOOP-4435, seems to suggest
>   that the 15.5 MB should be the amount of heap memory currently in use but since
>   this value never changes - not even when a job is running and I refresh the page
>   - I'm not convinced this is working.  
>
>   I'm asking this question because I have a Mahout job which slowly comes to halt with
>   a lot of 'OutOfMemoryError: Java heap space' errors, before it is 'Killed'.
>
>   I'm using Hadoop 0.20.2 and the latest Mahout snapshot version.
>
>   Thanks for any help.
>
>           Ken
>
>