Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Memory exception in the mapper


Copy link to this message
-
Re: Memory exception in the mapper
Mark Kerzner 2012-05-24, 02:40
Arun,

I am running the latest CDH3, which I re-installed yesterday, so I believe
it is Hadoop 0.21.

I have about 6000 maps emitted, and 16 spills, and then I see Mapper
cleanup() being called, after which I get this error

2012-05-23 20:22:58,108 FATAL org.apache.hadoop.mapred.Child: Error running
child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:355)

Thank you,
Mark

On Wed, May 23, 2012 at 9:29 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> What version of hadoop are you running?
>
> On May 23, 2012, at 12:16 PM, Mark Kerzner wrote:
>
> > Hi, all,
> >
> > I got the exception below in the mapper. I already have my global Hadoop
> > heap at 5 GB, but is there a specific other setting? Or maybe I should
> > troubleshoot for memory?
> >
> > But the same application works in the IDE.
> >
> > Thank you!
> >
> > Mark
> >
> > *stderr logs*
> >
> > Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError:
> > Java heap space
> >       at
> java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
> >       at
> java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
> >       at
> org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:292)
> >       at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:365)
> >       at org.apache.hadoop.mapred.Child$3.run(Child.java:157)
> > Exception in thread "communication thread" java.lang.OutOfMemoryError:
> > Java heap space
> >
> > Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "communication thread"
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>