Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive and Lzo Compression


Copy link to this message
-
Re: Hive and Lzo Compression
Nitin Pawar 2013-08-14, 17:54
Please correct me if I understood the question correctly

You created a table def without mentioning a stored as clause
then you load data into table from a compressed a file
then do a select query and it still works
but how did it figured out which compression codec to use?

Am I stating it correctly ?

On Wed, Aug 14, 2013 at 11:11 PM, Sanjay Subramanian <
[EMAIL PROTECTED]> wrote:

>  That is really interesting…let me try and think of a reason…meanwhile
> any other LZO Hive Samurais out there ? Please help with some guidance
>
>  sanjay
>
>   From: w00t w00t <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
> [EMAIL PROTECTED]>
> Date: Wednesday, August 14, 2013 1:15 AM
>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Hive and Lzo Compression
>
>
>  Thanks for your reply.
>
>  The interesting thing I experience is that the SELECT query still works
> - even when I do not specify the STORED AS clause... that puzzles me a bit.
>
>   ------------------------------
> *Von:* Sanjay Subramanian <[EMAIL PROTECTED]>
> *An:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; w00t w00t <
> [EMAIL PROTECTED]>
> *Gesendet:* 3:44 Mittwoch, 14.August 2013
> *Betreff:* Re: Hive and Lzo Compression
>
>  Hi
>
>  I think the CREATE TABLE without the STORED AS clause will not give any
> errors while creating the table.
> However when you query that table and since that table contains .lzo files
> , you would  get errors.
> With external tables , u r separating the table creation(definition) from
> the data. So only at the time of query of that table, hive might report
> errors.
>
>  LZO compression rocks ! I am so glad I used it in our projects here.
>
>  Regards
>
>  sanjay
>
>   From: w00t w00t <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
> [EMAIL PROTECTED]>
> Date: Tuesday, August 13, 2013 12:13 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Hive and Lzo Compression
>
>   Thanks for your replies and the link.
>
>  I could get it working, but wondered why the CREATE TABLE statement
> worked without the STORED AS Clause as well...that's what puzzles me a
> bit...
>
>  But I will use the STORED AS Clause to be on the safe side.
>
>
>   ------------------------------
> *Von:* Lefty Leverenz <[EMAIL PROTECTED]>
> *An:* [EMAIL PROTECTED]
> *CC:* w00t w00t <[EMAIL PROTECTED]>
> *Gesendet:* 19:06 Samstag, 10.August 2013
> *Betreff:* Re: Hive and Lzo Compression
>
>  I'm not seeing any documentation link in Sanjay's message, so here it is
> again (in the Hive wiki's language manual):
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO.
>
>
> On Thu, Aug 8, 2013 at 3:30 PM, Sanjay Subramanian <
> [EMAIL PROTECTED]> wrote:
>
>  Please refer this documentation here
> Let me know if u need more clarifications so that we can make this
> document better and complete
>
>  Thanks
>
>  sanjay
>
>   From: w00t w00t <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
> [EMAIL PROTECTED]>
> Date: Thursday, August 8, 2013 2:02 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Hive and Lzo Compression
>
>
>    Hello,
>
> I am started to run Hive with Lzo compression on Hortonworks 1.2
>
> I have managed to install/configure Lzo and  hive -e "set
> io.compression.codecs" shows me the Lzo Codecs:
> io.compression.codecs> org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> com.hadoop.compression.lzo.LzoCodec,
> com.hadoop.compression.lzo.LzopCodec,
> org.apache.hadoop.io.compress.BZip2Codec
>
> However, I have some questions where I would be happy if you could help me.
>
> (1) CREATE TABLE statement
>
>  I read in different postings, that in the CREATE TABLE statement, I have
> to use the following STORAGE clause:
>
>  CREATE EXTERNAL TABLE txt_table_lzo (
>     txt_line STRING
>  )
>  ROW FORMAT DELIMITED FIELDS TERMINATED BY '||||'

Nitin Pawar