Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Table Wrapper


Copy link to this message
-
Re: Table Wrapper
Good luck, bro. :)   May i ask why are you doing this to yourself? I think
your instincts are correct going down the path you describe sounds a tad
more painful than just hitting yourself in the head with a hammer.
Different strokes for different folks though.

so can we back up? what - briefly if possible - do you want to achieve with
a "wrapper"? (i'm going to regret asking that i know.)
On Tue, Jun 25, 2013 at 7:29 AM, Peter Marron <
[EMAIL PROTECTED]> wrote:

>  Hi,****
>
> ** **
>
> Running Hive 0.11.0 over Hadoop 1.0.4.****
>
> ** **
>
> I would like to be able to “wrap” a Hive table.****
>
> ** **
>
> So, if I have table “X” which uses SerDe “s” and InputFormat “i”****
>
> then I would like to be able to create a table “Y” which has a ****
>
> SerDe “ws” which is a wrapper of “s” (and so can encapsulate an instance
> of “s”)****
>
> and an InputFormat “wi” which is a wrapper of “I” (and similarly
> encapsulates an****
>
> instance of “i”).  So far I have done this by creating a table like this**
> **
>
> ** **
>
> CREATE TABLE Y (… copy of underlying table’s columns...)****
>
> ROW FORMAT SERDE 'ws'****
>
> WITH SERDEPROPERTIES (…****
>
> 'wrapped.serde.name'='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',
> ****
>
> ‘wrapped.inputformat.name’=’TextInputFormat’,****
>
> 'serialization.format'='|',              'field.delim'='|'****
>
> )****
>
> STORED AS****
>
>   INPUTFORMAT 'wi'****
>
>   OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> ****
>
> TBLPROPERTIES (…);****
>
> ** **
>
> I have to add the names of the underlying classes “s” and “I”****
>
> into the table properties so that I know what to instantiate.****
>
> I also have to replicate all the column details of the wrapped table****
>
> to ensure the correct information is passed down to the underlying SerDe**
> **
>
> when I instantiate it. I also have to know the output format. I have to***
> *
>
> explicitly add the default SerDe properties to get it to work.****
>
> I have to explicitly provide the default output format too.****
>
> If any changes are made to the underlying table then I need to reflect****
>
> those changes in my “wrapper” table.****
>
> It’s a mess.****
>
> ** **
>
> What I’d like to be able to do is to just parameterise my wrapper****
>
> table with the name of the underlying table and using that name****
>
> be able to instantiate the correct SerDe and InputFormat.****
>
> ** **
>
> Is there an easier way to do this?****
>
> ** **
>
> Any pointers appreciated.****
>
> ** **
>
> Z****
>
> ** **
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB