Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive and Lzo Compression


Copy link to this message
-
Re: Hive and Lzo Compression
Nitin Pawar 2013-08-17, 14:40
As per my understanding,

Underlying hadoop framework identifies if the files are compressed or not
in a transparent manner. If they are compressed then the framework does
take care of the decompression part when the compression codecs are
available .
On Thu, Aug 15, 2013 at 4:20 AM, Sanjay Subramanian <
[EMAIL PROTECTED]> wrote:

>  I am not sure if in this cade data is loaded
> OR partition  added with location specified (to some location in HDFS)
>
>  Yes u r stating the question correctly
>
>  sanjay
>
>   From: Nitin Pawar <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wednesday, August 14, 2013 10:54 AM
>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Hive and Lzo Compression
>
>   Please correct me if I understood the question correctly
>
>  You created a table def without mentioning a stored as clause
> then you load data into table from a compressed a file
> then do a select query and it still works
> but how did it figured out which compression codec to use?
>
>  Am I stating it correctly ?
>
>
>
> On Wed, Aug 14, 2013 at 11:11 PM, Sanjay Subramanian <
> [EMAIL PROTECTED]> wrote:
>
>>  That is really interesting…let me try and think of a reason…meanwhile
>> any other LZO Hive Samurais out there ? Please help with some guidance
>>
>>  sanjay
>>
>>   From: w00t w00t <[EMAIL PROTECTED]>
>> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
>> [EMAIL PROTECTED]>
>>  Date: Wednesday, August 14, 2013 1:15 AM
>>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Hive and Lzo Compression
>>
>>
>>  Thanks for your reply.
>>
>>  The interesting thing I experience is that the SELECT query still works
>> - even when I do not specify the STORED AS clause... that puzzles me a bit.
>>
>>   ------------------------------
>> *Von:* Sanjay Subramanian <[EMAIL PROTECTED]>
>> *An:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; w00t w00t <
>> [EMAIL PROTECTED]>
>> *Gesendet:* 3:44 Mittwoch, 14.August 2013
>> *Betreff:* Re: Hive and Lzo Compression
>>
>>  Hi
>>
>>  I think the CREATE TABLE without the STORED AS clause will not give any
>> errors while creating the table.
>> However when you query that table and since that table contains .lzo
>> files , you would  get errors.
>> With external tables , u r separating the table creation(definition) from
>> the data. So only at the time of query of that table, hive might report
>> errors.
>>
>>  LZO compression rocks ! I am so glad I used it in our projects here.
>>
>>  Regards
>>
>>  sanjay
>>
>>   From: w00t w00t <[EMAIL PROTECTED]>
>> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
>> [EMAIL PROTECTED]>
>> Date: Tuesday, August 13, 2013 12:13 AM
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Hive and Lzo Compression
>>
>>   Thanks for your replies and the link.
>>
>>  I could get it working, but wondered why the CREATE TABLE statement
>> worked without the STORED AS Clause as well...that's what puzzles me a
>> bit...
>>
>>  But I will use the STORED AS Clause to be on the safe side.
>>
>>
>>   ------------------------------
>> *Von:* Lefty Leverenz <[EMAIL PROTECTED]>
>> *An:* [EMAIL PROTECTED]
>> *CC:* w00t w00t <[EMAIL PROTECTED]>
>> *Gesendet:* 19:06 Samstag, 10.August 2013
>> *Betreff:* Re: Hive and Lzo Compression
>>
>>  I'm not seeing any documentation link in Sanjay's message, so here it
>> is again (in the Hive wiki's language manual):
>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO.
>>
>>
>> On Thu, Aug 8, 2013 at 3:30 PM, Sanjay Subramanian <
>> [EMAIL PROTECTED]> wrote:
>>
>>  Please refer this documentation here
>> Let me know if u need more clarifications so that we can make this
>> document better and complete
>>
>>  Thanks
>>
>>  sanjay
>>
>>   From: w00t w00t <[EMAIL PROTECTED]>
>> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <

Nitin Pawar