Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> java.lang.OutOfMemoryError: Java heap space


Copy link to this message
-
Re: java.lang.OutOfMemoryError: Java heap space
Normally task tracker isn't run on Name node.
Did you configure otherwise ?

On Mon, Jul 12, 2010 at 3:06 PM, Shuja Rehman <[EMAIL PROTECTED]> wrote:

> *Master Node output:*
>
>   total       used       free     shared    buffers     cached
> Mem:       2097328     515576    1581752          0      56060     254760
> -/+ buffers/cache:     204756    1892572
> Swap:       522104          0     522104
>
> *Slave Node output:*
>  total       used       free     shared    buffers     cached
> Mem:       1048752     860684     188068          0     148388     570948
> -/+ buffers/cache:     141348     907404
> Swap:       522104         40     522064
>
> it seems that on server there is more memory free.
>
>
> On Tue, Jul 13, 2010 at 2:57 AM, Alex Kozlov <[EMAIL PROTECTED]> wrote:
>
> > Maybe you do not have enough available memory on master?  What is the
> > output
> > of "*free*" on both nodes?  -- Alex K
> >
> > On Mon, Jul 12, 2010 at 2:08 PM, Shuja Rehman <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi
> > > I have added following line to my master node mapred-site.xml file
> > >
> > > <property>
> > >    <name>mapred.child.ulimit</name>
> > >    <value>3145728</value>
> > >  </property>
> > >
> > > and run the job again, and wow..., the jobs get completed in 4th
> attempt.
> > I
> > > checked the at 50030. Hadoop runs job 3 times on master server and it
> > fails
> > > but when it run on 2nd node, it succeeded and produce the desired
> result.
> > > Why it failed on master?
> > > Thanks
> > > Shuja
> > >
> > >
> > > On Tue, Jul 13, 2010 at 1:34 AM, Alex Kozlov <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hmm.  It means your options are not propagated to the nodes.  Can you
> > put
> > > *
> > > > mapred.child.ulimit* in the mapred-siet.xml and restart the
> > tasktrackers?
> > > >  I
> > > > was under impression that the below should be enough though.  Glad
> you
> > > got
> > > > it working in local mode.  -- Alex K
> > > >
> > > > On Mon, Jul 12, 2010 at 1:24 PM, Shuja Rehman <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > Hi Alex, I am using putty to connect to servers. and this is almost
> > my
> > > > > maximum screen output which i sent. putty is not allowed me to
> > increase
> > > > the
> > > > > size of terminal. is there any other way that i get the complete
> > output
> > > > of
> > > > > ps-aef?
> > > > >
> > > > > Now i run the following command and thnx God, it did not fails and
> > > > produce
> > > > > the desired output.
> > > > >
> > > > > hadoop jar
> > > > >
> > /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2+320.jar
> > > \
> > > > > -D mapred.child.java.opts=-Xmx1024m \
> > > > > -D mapred.child.ulimit=3145728 \
> > > > > -jt local \
> > > > > -inputformat StreamInputFormat \
> > > > > -inputreader "StreamXmlRecordReader,begin=<mdc xmlns:HTML=\"
> > > > > http://www.w3.org/TR/REC-xml <http://www.w3.org/TR/REC-xml%5C> <
> http://www.w3.org/TR/REC-xml%5C> <
> > http://www.w3.org/TR/REC-xml%5C> <
> > > http://www.w3.org/TR/REC-xml%5C> <
> > > > http://www.w3.org/TR/REC-xml%5C>">,end=</mdc>"
> > > > > \
> > > > > -input
> > > > >
> > > > >
> > > >
> > >
> >
> /user/root/RNCDATA/MDFDORKUCRAR02/A20100531.0000-0700-0015-0700_RNCCN-MDFDORKUCRAR02
> > > > > \
> > > > > -jobconf mapred.map.tasks=1 \
> > > > > -jobconf mapred.reduce.tasks=0 \
> > > > > -output RNC32 \
> > > > > -mapper /home/ftpuser1/Nodemapper5.groovy \
> > > > > -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
> > > > > -file /home/ftpuser1/Nodemapper5.groovy
> > > > >
> > > > >
> > > > > but when i omit the -jt local, it produces the same error.
> > > > > Thanks Alex for helping
> > > > > Regards
> > > > > Shuja
> > > > >
> > > > > On Tue, Jul 13, 2010 at 1:01 AM, Alex Kozlov <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Hi Shuja,
> > > > > >
> > > > > > Java listens to the last xmx, so if you have multiple "-Xmx ..."
> on
> > > the
> > > > > > command line, the last is valid.  Unfortunately you have
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB