Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive and Lzo Compression


Copy link to this message
-
Re: Hive and Lzo Compression
As per my understanding, its not file extensions as compressed files can be
renamed to anything without extensions.

First check, is file compressed if no then directly proceed to read else if
yes then find out the compression codec and use it. you can see by running
a file command on any compressed file on linux and it does tell all the
detail.
I am really not sure what happens when the compression codecs are not
available.
May be someone from mapred or hdfs dev forum can tell in detail how this is
handled.
On Mon, Aug 19, 2013 at 1:36 PM, w00t w00t <[EMAIL PROTECTED]> wrote:

> My scenario is a bit different - I am using external tables.
>
> So I uploaded some lzo compressed files into HDFS, generated the lzo-index
> files and finally I created the external table without the specific storage
> as clause .
> A SELECT statement on the table still works.
>
> Does it work transparently? So, Hadoop sees the lzo extension of my files
> and knows how to decompress it?
>
>
>
>   ------------------------------
>  *Von:* Nitin Pawar <[EMAIL PROTECTED]>
> *An:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> *Gesendet:* 19:54 Mittwoch, 14.August 2013
>
> *Betreff:* Re: Hive and Lzo Compression
>
> Please correct me if I understood the question correctly
>
> You created a table def without mentioning a stored as clause
> then you load data into table from a compressed a file
> then do a select query and it still works
> but how did it figured out which compression codec to use?
>
> Am I stating it correctly ?
>
>
>
> On Wed, Aug 14, 2013 at 11:11 PM, Sanjay Subramanian <
> [EMAIL PROTECTED]> wrote:
>
>  That is really interesting…let me try and think of a reason…meanwhile
> any other LZO Hive Samurais out there ? Please help with some guidance
>
>  sanjay
>
>   From: w00t w00t <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
> [EMAIL PROTECTED]>
> Date: Wednesday, August 14, 2013 1:15 AM
>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Hive and Lzo Compression
>
>
>  Thanks for your reply.
>
>  The interesting thing I experience is that the SELECT query still works
> - even when I do not specify the STORED AS clause... that puzzles me a bit.
>
>   ------------------------------
> *Von:* Sanjay Subramanian <[EMAIL PROTECTED]>
> *An:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; w00t w00t <
> [EMAIL PROTECTED]>
> *Gesendet:* 3:44 Mittwoch, 14.August 2013
> *Betreff:* Re: Hive and Lzo Compression
>
>  Hi
>
>  I think the CREATE TABLE without the STORED AS clause will not give any
> errors while creating the table.
> However when you query that table and since that table contains .lzo files
> , you would  get errors.
> With external tables , u r separating the table creation(definition) from
> the data. So only at the time of query of that table, hive might report
> errors.
>
>  LZO compression rocks ! I am so glad I used it in our projects here.
>
>  Regards
>
>  sanjay
>
>   From: w00t w00t <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, w00t w00t <
> [EMAIL PROTECTED]>
> Date: Tuesday, August 13, 2013 12:13 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Hive and Lzo Compression
>
>   Thanks for your replies and the link.
>
>  I could get it working, but wondered why the CREATE TABLE statement
> worked without the STORED AS Clause as well...that's what puzzles me a
> bit...
>
>  But I will use the STORED AS Clause to be on the safe side.
>
>
>   ------------------------------
> *Von:* Lefty Leverenz <[EMAIL PROTECTED]>
> *An:* [EMAIL PROTECTED]
> *CC:* w00t w00t <[EMAIL PROTECTED]>
> *Gesendet:* 19:06 Samstag, 10.August 2013
> *Betreff:* Re: Hive and Lzo Compression
>
>  I'm not seeing any documentation link in Sanjay's message, so here it is
> again (in the Hive wiki's language manual):
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO.
>
>
> On Thu, Aug 8, 2013 at 3:30 PM, Sanjay Subramanian <
Nitin Pawar
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB