I have a bit of an edge case on using lzo which I think might be related
to HIVE-524.
When running a query like this:
select distinct login_cldr_id as cldr_id from chatsessions_load;
I get a java.io.IOException:java.io.EOFException without much of a
description.
I know the output should be a single value and noticed it decided to use
2 reducers.
One of the reducers produced a 0 byte file which I imagine will be the
cause of the IOException. It I do "set mapred.reduce.tasks=1" it works
correctly since there is no 0 byte file anymore.

I also noticed when using gzip I don't see this problem at all.

Since I use

Looks like a lzo codec problem. Can you try a simple mapreduce program
outputs to lzo compression and the same output file format as you hive
table?

On 2/9/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
> I have a bit of an edge case on using lzo which I think might be related
> to HIVE-524.
> When running a query like this:
> select distinct login_cldr_id as cldr_id from chatsessions_load;
> I get a java.io.IOException:java.io.EOFException without much of a
> description.
> I know the output should be a single value and noticed it decided to use
> 2 reducers.
> One of the reducers produced a 0 byte file which I imagine will be the
> cause of the IOException. It I do "set mapred.reduce.tasks=1" it works
> correctly since there is no 0 byte file anymore.
>
> I also noticed when using gzip I don't see this problem at all.
>
> Since I use
>

--
Sent from my mobile device

Yours,
Zheng

You are probably right.
Hopefully I'll get a chance to look at this today.

Thanks..

Zheng Shao wrote:
> Looks like a lzo codec problem. Can you try a simple mapreduce program
> outputs to lzo compression and the same output file format as you hive
> table?
>
> On 2/9/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>  
>> I have a bit of an edge case on using lzo which I think might be related
>> to HIVE-524.
>> When running a query like this:
>> select distinct login_cldr_id as cldr_id from chatsessions_load;
>> I get a java.io.IOException:java.io.EOFException without much of a
>> description.
>> I know the output should be a single value and noticed it decided to use
>> 2 reducers.
>> One of the reducers produced a 0 byte file which I imagine will be the
>> cause of the IOException. It I do "set mapred.reduce.tasks=1" it works
>> correctly since there is no 0 byte file anymore.
>>
>> I also noticed when using gzip I don't see this problem at all.
>>
>> Since I use
>>
>>    
>
>