Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Unexpected end of input stream (GZ)


Copy link to this message
-
Unexpected end of input stream (GZ)
Oleg Ruchovets 2012-07-24, 08:19
Hi , I got such exception running hadoop job:

java.io.EOFException: Unexpected end of input stream at
org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:99)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:87)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at java.io.InputStream.read(InputStream.java:85) at
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:114)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at
org.apache.hadoop.mapred.Child$4.run(Child.

As I understood some of my files are corrupted ( I am working with GZ
format).

I resolve the issue using conf.set("mapred.max.map.failures.percent" , "1"),

But I don't know what file cause the problem.

Question:
 How can I get a filename which is corrupted.

Thanks in advance
Oleg.