Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Memory exception in the mapper


Copy link to this message
-
Re: Memory exception in the mapper
Joey,

that did the trick!

Actually, I am writing to the log with System.out.println() statements, and
I write about 12,000 lines, would that be a problem? I don't really need
this output, so if you think it's inadvisable, I will remove that.

Also, I hope that if I have not 6,000 maps but 12,000 or even 30,000, it
will still work.

Well, I will see pretty soon, I guess, with more data.

Again, thank you.

Sincerely,
Mark

On Wed, May 23, 2012 at 9:43 PM, Joey Krabacher <[EMAIL PROTECTED]>wrote:

> Mark,
>
> Have you tried tweaking the mapred.child.java.opts property in your
> mapred-site.xml?
>
> <property>
>    <name>mapred.child.java.opts</name>
>    <value>-Xmx2048m</value>
>  </property>
>
> This might help.
> It looks like the fatal error came right after the log truncater fired off.
> Are you outputting anything to the logs manually, or have you looked at the
> user logs to see if there is anything taking up lots of room?
>
> / * Joey */
>
>
> On Wed, May 23, 2012 at 9:35 PM, Mark Kerzner <[EMAIL PROTECTED]
> >wrote:
>
> > Joey,
> >
> > my errors closely resembles this
> > one<
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201006.mbox/%[EMAIL PROTECTED]%3E
> > >in
> > the archives. I can now be much more specific with the errors message,
> > and it is quoted below. I tried -Xmx3096. But I got the same error.
> >
> > Thank you,
> > Mark
> >
> >
> > syslog logs
> > 2012-05-23 20:04:52,349 WARN org.apache.hadoop.util.NativeCodeLoader:
> > Unable to load native-hadoop library for your platform... using
> > builtin-java classes where applicable
> > 2012-05-23 20:04:52,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId> > 2012-05-23 20:04:52,695 INFO org.apache.hadoop.util.ProcessTree: setsid
> > exited with exit code 0
> > 2012-05-23 20:04:52,699 INFO org.apache.hadoop.mapred.Task:  Using
> > ResourceCalculatorPlugin :
> > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@d56b37
> > 2012-05-23 20:04:52,813 INFO org.apache.hadoop.mapred.MapTask:
> io.sort.mb > > 100
> > 2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: data
> buffer
> > = 79691776/99614720
> > 2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: record
> > buffer = 262144/327680
> > 2012-05-23 20:04:53,010 WARN
> > org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library
> not
> > loaded
> > 2012-05-23 20:12:29,120 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 0; bufend = 79542629; bufvoid = 99614720
> > 2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: kvstart > 0;
> > kvend = 228; length = 327680
> > 2012-05-23 20:12:31,248 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 0
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 79542629; bufend = 53863940; bufvoid = 99614720
> > 2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: kvstart > > 228; kvend = 431; length = 327680
> > 2012-05-23 20:13:03,294 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 1
> > 2012-05-23 20:13:48,121 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 53863940; bufend = 31696780; bufvoid = 99614720
> > 2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: kvstart > > 431; kvend = 861; length = 327680
> > 2012-05-23 20:13:49,818 INFO org.apache.hadoop.mapred.MapTask: Finished
> > spill 2
> > 2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: Spilling
> map
> > output: buffer full= true
> > 2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: bufstart > > 31696780; bufend = 10267329; bufvoid = 99614720
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB