Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Errors reading lzo-compressed files from Hadoop


+
Dmitriy Ryaboy 2010-04-01, 07:16
+
Todd Lipcon 2010-04-01, 18:36
+
Alex Roetter 2010-04-06, 16:50
+
Todd Lipcon 2010-04-06, 17:35
+
Todd Lipcon 2010-04-08, 00:24
Copy link to this message
-
Re: Errors reading lzo-compressed files from Hadoop
Todd, you rock. We'll integrate this into Kevin's branch right away.

-Dmitriy

On Wed, Apr 7, 2010 at 5:24 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> For Dmitriy and anyone else who has seen this error, I just committed a fix
> to my github repository:
>
> http://github.com/toddlipcon/hadoop-lzo/commit/f3bc3f8d003bb8e24f254b25bca2053f731cdd58
>
> The problem turned out to be an assumption that InputStream.read() would
> return all the bytes that were asked for. This turns out to almost always be
> true on local filesystems, but on HDFS it's not true if the read crosses a
> block boundary. So, every couple of TB of lzo compressed data one might see
> this error.
>
> Big thanks to Alex Roetter who was able to provide a file that exhibited the
> bug!
>
> Thanks
> -Todd
>
>
> On Tue, Apr 6, 2010 at 10:35 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi Alex,
>> Unfortunately I wasn't able to reproduce, and the data Dmitriy is
>> working with is sensitive.
>> Do you have some data you could upload (or send me off list) that
>> exhibits the issue?
>> -Todd
>>
>> On Tue, Apr 6, 2010 at 9:50 AM, Alex Roetter <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > Todd Lipcon <todd@...> writes:
>> >
>> > >
>> > > Hey Dmitriy,
>> > >
>> > > This is very interesting (and worrisome in a way!) I'll try to take a
>> look
>> > > this afternoon.
>> > >
>> > > -Todd
>> > >
>> >
>> > Hi Todd,
>> >
>> > I wanted to see if you made any progress on this front. I'm seeing a very
>> > similar error, trying to run a MR (Hadoop 0.20.1) over a bunch of
>> > LZOP compressed / indexed files (using Kevin Weil's package), and I have
>> one
>> > map task that always fails in what looks like the same place as described
>> in
>> > the previous post. I haven't yet done the experimentation mentioned above
>> > (isolating the input file corresponding to the failed map task,
>> decompressing
>> > it / recompressing it, testing it out operating directly on local disk
>> > instead of HDFS, etc).
>> >
>> > However, since I am crashing in exactly the same place it seems likely
>> this
>> > is related, and thought I'd check on your work in the meantime.
>> >
>> > FYI, my stack track is below:
>> >
>> > 2010-04-05 18:15:16,895 FATAL org.apache.hadoop.mapred.TaskTracker: Error
>> > running child : java.lang.InternalError: lzo1x_decompress_safe returned:
>> >        at
>> com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect
>> > (Native Method)
>> >        at com.hadoop.compression.lzo.LzoDecompressor.decompress
>> > (LzoDecompressor.java:303)
>> >        at
>> > com.hadoop.compression.lzo.LzopDecompressor.decompress
>> > (LzopDecompressor.java:104)
>> >        at com.hadoop.compression.lzo.LzopInputStream.decompress
>> > (LzopInputStream.java:223)
>> >        at
>> > org.apache.hadoop.io.compress.DecompressorStream.read
>> > (DecompressorStream.java:74)
>> >        at java.io.InputStream.read(InputStream.java:85)
>> >        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>> >        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:187)
>> >        at
>> > com.hadoop.mapreduce.LzoLineRecordReader.nextKeyValue
>> > (LzoLineRecordReader.java:126)
>> >        at
>> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue
>> > (MapTask.java:423)
>> >        at
>> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >
>> >
>> > Any update much appreciated,
>> > Alex
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
+
Todd Lipcon 2010-04-08, 17:39
+
Todd Lipcon 2010-04-08, 17:46
+
Dmitriy Ryaboy 2010-04-08, 18:20