Hi,

I need to load data directly from a ctl A delimiter zipped file from the Linux box directly.
Do I need to 1) un-zip the files and then load them to Hive tables, or 2) is there a direct command that can load zipped data to Hive table directly?

Thanks,
Wei

If the file ends in .bz2 .gz or .deflate there is nothing special you
need to. TextInputFormat (the default) will automatically unzip and
read these. However these types are not split-table so if the file is
large it can not be processed in parallel.
On Sun, Mar 4, 2012 at 9:26 PM, Lu, Wei <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I need to load data directly from a ctl A delimiter zipped file from the
> Linux box directly.
>
> Do I need to 1) un-zip the files and then load them to Hive tables, or 2) is
> there a direct command that can load zipped data to Hive table directly?
>
>
>
> Thanks,
>
> Wei

So I cannot directly load files end in .zip to hive, right?
I tried to put a .zip file to HDFS and load it to a Hive table, but the data in the table turn out to be messy....

Wei

-----Original Message-----
From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
Sent: Monday, March 05, 2012 10:54 AM
To: [EMAIL PROTECTED]
Subject: Re: load zip file to hive table

If the file ends in .bz2 .gz or .deflate there is nothing special you
need to. TextInputFormat (the default) will automatically unzip and
read these. However these types are not split-table so if the file is
large it can not be processed in parallel.
On Sun, Mar 4, 2012 at 9:26 PM, Lu, Wei <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I need to load data directly from a ctl A delimiter zipped file from the
> Linux box directly.
>
> Do I need to 1) un-zip the files and then load them to Hive tables, or 2) is
> there a direct command that can load zipped data to Hive table directly?
>
>
>
> Thanks,
>
> Wei

Hadoop does not support opening zip files out of the box see.
https://issues.apache.org/jira/browse/MAPREDUCE-210

Maybe someone out there has written a custom input format.

On Sun, Mar 4, 2012 at 11:03 PM, Lu, Wei <[EMAIL PROTECTED]> wrote:
> So I cannot directly load files end in .zip to hive, right?
> I tried to put a .zip file to HDFS and load it to a Hive table, but the data in the table turn out to be messy....
>
> Wei
>
> -----Original Message-----
> From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
> Sent: Monday, March 05, 2012 10:54 AM
> To: [EMAIL PROTECTED]
> Subject: Re: load zip file to hive table
>
> If the file ends in .bz2 .gz or .deflate there is nothing special you
> need to. TextInputFormat (the default) will automatically unzip and
> read these. However these types are not split-table so if the file is
> large it can not be processed in parallel.
>
>
> On Sun, Mar 4, 2012 at 9:26 PM, Lu, Wei <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>>
>>
>> I need to load data directly from a ctl A delimiter zipped file from the
>> Linux box directly.
>>
>> Do I need to 1) un-zip the files and then load them to Hive tables, or 2) is
>> there a direct command that can load zipped data to Hive table directly?
>>
>>
>>
>> Thanks,
>>
>> Wei