Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> EOF when Combiner works


Copy link to this message
-
EOF when Combiner works
Hi!

When I run job with this options:
  -Dmapred.map.child.java.opts=-Xmx2048M
  -Dio.sort.mb=1424
  -Dio.sort.record.percent=0.08
all tasks fails on combiner step with:

...
2013-01-23 12:20:28,143 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 1424
2013-01-23 12:23:03,772 INFO org.apache.hadoop.mapred.MapTask: data buffer = 1098974720/1373718448
2013-01-23 12:23:03,772 INFO org.apache.hadoop.mapred.MapTask: record buffer = 5972689/7465861
2013-01-23 12:23:03,790 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2013-01-23 12:23:03,792 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev Unknown build revision]
2013-01-23 12:51:47,211 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true
2013-01-23 12:51:47,211 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1098974551; bufvoid = 1373718448
2013-01-23 12:51:47,211 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 5661226; length = 7465861
2013-01-23 12:52:01,389 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
2013-01-23 12:52:02,526 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-01-23 12:52:02,528 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user (auth:SIMPLE) cause:java.io.IOException: Spill failed
2013-01-23 12:52:02,529 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:886)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at my.mapreduce.StatisticsHostMapper.calculateCounters(StatisticsHostMapper.java:104)
at my.mapreduce.StatisticsHostMapper.map(StatisticsHostMapper.java:116)
at my.mapreduce.StatisticsHostMapper.map(StatisticsHostMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.lang.RuntimeException: next value iterator failed
at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:166)
at my.mapreduce.HostCounters.calculate(HostCounters.java:147)
at my.mapreduce.StatisticsHostCombiner.reduce(StatisticsHostCombiner.java:17)
at my.mapreduce.StatisticsHostCombiner.reduce(StatisticsHostCombiner.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1445)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1291)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:712)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1199)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at my.mapreduce.HostCounters.readFields(HostCounters.java:54)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:163)
... 8 more
2013-01-23 12:52:02,578 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
When I run the same task with this options:
  -Dmapred.map.child.java.opts=-Xmx2048M
  -Dio.sort.mb=1024
  -Dio.sort.record.percent=0.08
everything is OK:

...
2013-01-23 13:01:46,168 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 1024
2013-01-23 13:01:47,593 INFO org.apache.hadoop.mapred.MapTask: data buffer = 790273984/987842480
2013-01-23 13:01:47,593 INFO org.apache.hadoop.mapred.MapTask: record buffer = 4294967/5368709
2013-01-23 13:01:47,602 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2013-01-23 13:01:47,621 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev Unknown build revision]
2013-01-23 13:28:42,168 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true
2013-01-23 13:28:42,169 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 790273835; bufvoid = 987842480
2013-01-23 13:28:42,169 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 4072970; length = 5368709
2013-01-23 13:28:56,417 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
2013-01-23 13:29:18,998 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
...
Please help me to understand the reason of task fails.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB