Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Any advice about complex Hive tables?


Copy link to this message
-
Re: Any advice about complex Hive tables?
Hi Chuck,

I have a similar complex hive tables with many fields and some are nested
like array of structs (but only upto 3 levels). How did you define you ROW
FORMAT as to separate the items? The COLLECTION ITEMS TERMINATED BY clause
works only for the first level.How did you handle level 2 , 3, etc? Is it
through SERDE FORMATs?  Could you
 share your CREATE TABLE statement?  I am having problem correctly defining
my DDL to load the data file correctly.

Much appreciated.

Thanks,
Sadu
On Mon, Oct 8, 2012 at 3:37 PM, Connell, Chuck <[EMAIL PROTECTED]>wrote:

> (Follow up to the thread below...)
>
> I have a complex Hive table -- many fields, many nested structs. Hive
> fails to create the table at all. I can't even start to load data or query
> the data. Anyone else run into this? It seems to be a showstopper to using
> Hive for us, which is a shame since everything else about Hive is just what
> we want.
>
> Any suggestions? Workarounds?
>
> Thanks,
> Chuck
>
>
> ________________________________________
> From: Connell, Chuck
> Sent: Thursday, October 04, 2012 4:31 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Limit to columns or nesting of Hive table?
>
> The issue apparently is not just the number of levels of nesting. I just
> created a Hive table with 20 levels of structs within each other. It
> created fine. This is more levels than the table that was failing for me.
> The failing table had many more fields throughout the levels.
>
> Chuck
>
>
>
> -----Original Message-----
> From: Connell, Chuck [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 04, 2012 12:09 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Limit to columns or nesting of Hive table?
>
> Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that
> this limit cannot easily be raised?
>
> Chuck
>
> -----Original Message-----
> From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 04, 2012 11:57 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Limit to columns or nesting of Hive table?
>
> There is an open jira ticket on this. There is a hard coded limit but it
> could be raised with some mostly minor code changes.
>
> One of the bigger problems is that hive stores the definition of a column
> in JDBC "column" and for some databases larger nested structs can case
> issues.
>
> Edward
>
> On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck <[EMAIL PROTECTED]>
> wrote:
> > I am trying to create a large Hive table, with many columns and deeply
> > nested structs. It is failing with
> java.lang.ArrayIndexOutOfBoundsException:
> > 10.
> >
> >
> >
> > Before I spend a lot of time debugging my table declaration, is there
> > some limit here I should know about? Max number of columns? Max depth
> > of struct nesting?
> >
> >
> >
> > Thanks,
> >
> > Chuck
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB