Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Unexpected end of input stream


Copy link to this message
-
Re: Unexpected end of input stream
Jagat Singh 2012-08-29, 10:32
Hi,

I had same error few days back.

Now difficulty we have is to find which gz file is corrupt. Its not corrupt
as such but some how hadoop says it is. If you made the file in Windows and
then transfer to hadoop of can give. This error. If you want to see which
file is corrupt do select count query and watch job tracker for error , it
would give name of gz file currently processed and then of it fails you can
find and remove that file. You can then again gzip that in some Linux
machine and upload it would work.

Thanks,

-----------
Sent from Mobile , short and crisp.
On 29-Aug-2012 12:23 AM, "Kiwon Lee" <[EMAIL PROTECTED]> wrote:

> Hi
>
> I have a lot of compressed gzip files on hdfs.
> An exception has occurred at TaskTracker, during processing of MR.
> If any file is invalid, may I know that?
>
>
> 2012-08-28 09:17:56,320 INFO ExecMapper: ExecMapper: processed 0 rows:
> used memory = 125190136
> 2012-08-28 09:17:56,324 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>  2012-08-28 09:17:56,326 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:ubuntu (auth:SIMPLE) cause:java.io.IOException: java.io.EOFException:
> Unexpected end of input stream
> 2012-08-28 09:17:56,326 WARN org.apache.hadoop.mapred.Child: Error running
> child
> java.io.IOException: java.io.EOFException: Unexpected end of input stream
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:275)
>         at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
>         at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.io.EOFException: Unexpected end of input stream
>         at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
>         at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:83)
>         at java.io.InputStream.read(InputStream.java:82)
>         at
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:209)
>         at org.apache.hadoop.util.LineReader.readLine(LineReader.java:173)
>         at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:160)
>         at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:38)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:273)
>         ... 13 more
>
>
> --
>
> *Best Regards.** Ethan (Kiwon Lee)*
>    [EMAIL PROTECTED]
>
>
>