Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Memory exception in the mapper


Copy link to this message
-
Re: Memory exception in the mapper
Mark Kerzner 2012-05-24, 03:21
Joey,

that did the trick!

Actually, I am writing to the log with System.out.println() statements, and
I write about 12,000 lines, would that be a problem? I don't really need
this output, so if you think it's inadvisable, I will remove that.

Also, I hope that if I have not 6,000 maps but 12,000 or even 30,000, it
will still work.

Well, I will see pretty soon, I guess, with more data.

Again, thank you.

Sincerely,
Mark

On Wed, May 23, 2012 at 9:43 PM, Joey Krabacher <[EMAIL PROTECTED]>wrote:

> Mark,
>
> Have you tried tweaking the mapred.child.java.opts property in your
> mapred-site.xml?
>
> <property>
>    <name>mapred.child.java.opts</name>
>    <value>-Xmx2048m</value>
>  </property>
>
> This might help.
> It looks like the fatal error came right after the log truncater fired off.
> Are you outputting anything to the logs manually, or have you looked at the
> user logs to see if there is anything taking up lots of room?
>
> / * Joey */
>
>
> On Wed, May 23, 2012 at 9:35 PM, Mark Kerzner <[EMAIL PROTECTED]
> >wrote:
>
> > Joey,
> >
> > my errors closely resembles this
> > one<
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201006.mbox/%[EMAIL PROTECTED]%3E
> > >in
> > the archives. I can now be much more specific with the errors message,
> > and it is quoted below. I tried -Xmx3096. But I got the same error.
> >
> > Thank you,
> > Mark
> >
> >
> > syslog logs
> > 2012-05-23 20:04:52,349 WARN org.apache.hadoop.util.NativeCodeLoader:
> > Unable to load native-hadoop library for your platform... using
> > builtin-java classes where applicable
> > 2012-05-23 20:04:52,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId> > 2012-05-23 20:04:52,695 INFO org.apache.hadoop.util.ProcessTree: setsid
> > exited with exit code 0
> > 2012-05-23 20:04:52,699 INFO org.apache.hadoop.mapred.Task:  Using
> > ResourceCalculatorPlugin :
> > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@d56b37
> > 2012-05-23 20:04:52,813 INFO org.apache.hadoop.mapred.MapTask:
> io.sort.mb > > 100
> > 2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: data
> buffer
> > = 79691776/99614720
> > 2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: record
> > buffer = 262144/327680
> > 2012-05-23 20:04:53,010 WARN
> > org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library
> not
> > loaded
> > 2012-05-23 20:12:29,120 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 0; bufend = 79542629; bufvoid = 99614720
> > 2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: kvstart > 0;
> > kvend = 228; length = 327680
> > 2012-05-23 20:12:31,248 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 0
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 79542629; bufend = 53863940; bufvoid = 99614720
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: kvstart > > 228; kvend = 431; length = 327680
> > 2012-05-23 20:13:03,294 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 1
> > 2012-05-23 20:13:48,121 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 53863940; bufend = 31696780; bufvoid = 99614720
> > 2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: kvstart > > 431; kvend = 861; length = 327680
> > 2012-05-23 20:13:49,818 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 2
> > 2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 31696780; bufend = 10267329; bufvoid = 99614720