Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Heap Size question.


Copy link to this message
-
Re: Heap Size question.
The values show you the maximum heap size and currently used heap of the job tracker, not running jobs. Furthermore, the HADOOP_HEAPSIZE setting only sets the maximum heap for the daemons, not the tasks in your job.

If you're getting OOMEs, you should add a setting to your mapred-site.xml file that looks like this:

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx1g</value>
</property>

-Joey

On Jun 1, 2011, at 7:35, Ken Williams <[EMAIL PROTECTED]> wrote:

>
>
>  Hi All,
>
>  I'm a bit confused about the values displayed on the 'jobtracker.jsp' page.
>  In particular, there's a section called 'Cluster Summary'.
>
>  I'm running a small 4-machine Hadoop cluster, and when I point a web-browser
>  at my master machine (http://master:50030/jobtracker.jsp) it displays,
>
>               Cluster Summary (Heap Size is 15.5 MB / 1.89 GB)
>
>   What exactly do these figures mean ?
>
>   I know that the second figure (1.89 GB) is determined by the value of
>   the HADOOP_HEAPSIZE variable set in  'conf/hadoop-env.sh'. What I'm not
>   sure about is exactly what it means, or where the first value (15.5 MB) is determined
>   or what it means.
>
>   I'm guessing the 1.89 GB is the amount of heap-memory allocated to Hadoop
>   on each machine in the cluster. (Correct ?)
>
>   I have no idea what the 15.5 MB means or where it comes from. It never changes,
>   not even when a job is running, and I can't find any explanation in the documentation.
>
>   This page, https://issues.apache.org/jira/browse/HADOOP-4435, seems to suggest
>   that the 15.5 MB should be the amount of heap memory currently in use but since
>   this value never changes - not even when a job is running and I refresh the page
>   - I'm not convinced this is working.  
>
>   I'm asking this question because I have a Mahout job which slowly comes to halt with
>   a lot of 'OutOfMemoryError: Java heap space' errors, before it is 'Killed'.
>
>   I'm using Hadoop 0.20.2 and the latest Mahout snapshot version.
>
>   Thanks for any help.
>
>           Ken
>
>                        
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB