Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Lzo problem throwing java.io.IOException:java.io.EOFException


Copy link to this message
-
Re: Lzo problem throwing java.io.IOException:java.io.EOFException
You are probably right.
Hopefully I'll get a chance to look at this today.

Thanks..

Zheng Shao wrote:
> Looks like a lzo codec problem. Can you try a simple mapreduce program
> outputs to lzo compression and the same output file format as you hive
> table?
>
> On 2/9/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>  
>> I have a bit of an edge case on using lzo which I think might be related
>> to HIVE-524.
>> When running a query like this:
>> select distinct login_cldr_id as cldr_id from chatsessions_load;
>> I get a java.io.IOException:java.io.EOFException without much of a
>> description.
>> I know the output should be a single value and noticed it decided to use
>> 2 reducers.
>> One of the reducers produced a 0 byte file which I imagine will be the
>> cause of the IOException. It I do "set mapred.reduce.tasks=1" it works
>> correctly since there is no 0 byte file anymore.
>>
>> I also noticed when using gzip I don't see this problem at all.
>>
>> Since I use
>>
>>    
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB