Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Memory exception in the mapper


Copy link to this message
-
Re: Memory exception in the mapper
Joey,

my errors closely resembles this
one<http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201006.mbox/%[EMAIL PROTECTED]%3E>in
the archives. I can now be much more specific with the errors message,
and it is quoted below. I tried -Xmx3096. But I got the same error.

Thank you,
Mark
syslog logs
2012-05-23 20:04:52,349 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-05-23 20:04:52,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId2012-05-23 20:04:52,695 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2012-05-23 20:04:52,699 INFO org.apache.hadoop.mapred.Task:  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@d56b37
2012-05-23 20:04:52,813 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb 100
2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: data buffer
= 79691776/99614720
2012-05-23 20:04:52,998 INFO org.apache.hadoop.mapred.MapTask: record
buffer = 262144/327680
2012-05-23 20:04:53,010 WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not
loaded
2012-05-23 20:12:29,120 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: bufstart 0; bufend = 79542629; bufvoid = 99614720
2012-05-23 20:12:29,134 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0;
kvend = 228; length = 327680
2012-05-23 20:12:31,248 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 0
2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: bufstart 79542629; bufend = 53863940; bufvoid = 99614720
2012-05-23 20:13:01,862 INFO org.apache.hadoop.mapred.MapTask: kvstart 228; kvend = 431; length = 327680
2012-05-23 20:13:03,294 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 1
2012-05-23 20:13:48,121 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: bufstart 53863940; bufend = 31696780; bufvoid = 99614720
2012-05-23 20:13:48,122 INFO org.apache.hadoop.mapred.MapTask: kvstart 431; kvend = 861; length = 327680
2012-05-23 20:13:49,818 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 2
2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: bufstart 31696780; bufend = 10267329; bufvoid = 99614720
2012-05-23 20:15:25,618 INFO org.apache.hadoop.mapred.MapTask: kvstart 861; kvend = 1462; length = 327680
2012-05-23 20:15:27,068 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 3
2012-05-23 20:15:53,519 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:15:53,519 INFO org.apache.hadoop.mapred.MapTask: bufstart 10267329; bufend = 85241086; bufvoid = 99614720
2012-05-23 20:15:53,519 INFO org.apache.hadoop.mapred.MapTask: kvstart 1462; kvend = 1642; length = 327680
2012-05-23 20:15:54,760 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 4
2012-05-23 20:16:26,284 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:16:26,284 INFO org.apache.hadoop.mapred.MapTask: bufstart 85241086; bufend = 51305930; bufvoid = 99614720
2012-05-23 20:16:26,284 INFO org.apache.hadoop.mapred.MapTask: kvstart 1642; kvend = 1946; length = 327680
2012-05-23 20:16:27,566 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 5
2012-05-23 20:16:57,046 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:16:57,046 INFO org.apache.hadoop.mapred.MapTask: bufstart 51305930; bufend = 31353466; bufvoid = 99614720
2012-05-23 20:16:57,046 INFO org.apache.hadoop.mapred.MapTask: kvstart 1946; kvend = 2263; length = 327680
2012-05-23 20:16:58,076 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 6
2012-05-23 20:17:52,820 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:17:52,820 INFO org.apache.hadoop.mapred.MapTask: bufstart 31353466; bufend = 10945750; bufvoid = 99614720
2012-05-23 20:17:52,820 INFO org.apache.hadoop.mapred.MapTask: kvstart 2263; kvend = 2755; length = 327680
2012-05-23 20:17:53,939 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 7
2012-05-23 20:18:19,528 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:18:19,528 INFO org.apache.hadoop.mapred.MapTask: bufstart 10945750; bufend = 81838103; bufvoid = 99614720
2012-05-23 20:18:19,528 INFO org.apache.hadoop.mapred.MapTask: kvstart 2755; kvend = 2967; length = 327680
2012-05-23 20:18:21,145 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 8
2012-05-23 20:18:52,353 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:18:52,353 INFO org.apache.hadoop.mapred.MapTask: bufstart 81838103; bufend = 61751422; bufvoid = 99614720
2012-05-23 20:18:52,353 INFO org.apache.hadoop.mapred.MapTask: kvstart 2967; kvend = 3202; length = 327680
2012-05-23 20:18:53,485 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 9
2012-05-23 20:19:42,993 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:19:42,993 INFO org.apache.hadoop.mapred.MapTask: bufstart 61751422; bufend = 38440188; bufvoid = 99614720
2012-05-23 20:19:42,993 INFO org.apache.hadoop.mapred.MapTask: kvstart 3202; kvend = 3628; length = 327680
2012-05-23 20:19:44,346 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 10
2012-05-23 20:20:47,957 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: buffer full= true
2012-05-23 20:20:47,957 INFO org.apache.hadoop.mapred.MapTask: bufstart 38440188; bufend = 18376423; bufvoid = 99614720
2012-05