Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Any advice about complex Hive tables?


+
Connell, Chuck 2012-10-08, 20:37
Copy link to this message
-
Re: Any advice about complex Hive tables?
Sadananda Hegde 2012-10-12, 03:46
Hi Chuck,

I have a similar complex hive tables with many fields and some are nested
like array of structs (but only upto 3 levels). How did you define you ROW
FORMAT as to separate the items? The COLLECTION ITEMS TERMINATED BY clause
works only for the first level.How did you handle level 2 , 3, etc? Is it
through SERDE FORMATs?  Could you
 share your CREATE TABLE statement?  I am having problem correctly defining
my DDL to load the data file correctly.

Much appreciated.

Thanks,
Sadu
On Mon, Oct 8, 2012 at 3:37 PM, Connell, Chuck <[EMAIL PROTECTED]>wrote:

> (Follow up to the thread below...)
>
> I have a complex Hive table -- many fields, many nested structs. Hive
> fails to create the table at all. I can't even start to load data or query
> the data. Anyone else run into this? It seems to be a showstopper to using
> Hive for us, which is a shame since everything else about Hive is just what
> we want.
>
> Any suggestions? Workarounds?
>
> Thanks,
> Chuck
>
>
> ________________________________________
> From: Connell, Chuck
> Sent: Thursday, October 04, 2012 4:31 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Limit to columns or nesting of Hive table?
>
> The issue apparently is not just the number of levels of nesting. I just
> created a Hive table with 20 levels of structs within each other. It
> created fine. This is more levels than the table that was failing for me.
> The failing table had many more fields throughout the levels.
>
> Chuck
>
>
>
> -----Original Message-----
> From: Connell, Chuck [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 04, 2012 12:09 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Limit to columns or nesting of Hive table?
>
> Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that
> this limit cannot easily be raised?
>
> Chuck
>
> -----Original Message-----
> From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 04, 2012 11:57 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Limit to columns or nesting of Hive table?
>
> There is an open jira ticket on this. There is a hard coded limit but it
> could be raised with some mostly minor code changes.
>
> One of the bigger problems is that hive stores the definition of a column
> in JDBC "column" and for some databases larger nested structs can case
> issues.
>
> Edward
>
> On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck <[EMAIL PROTECTED]>
> wrote:
> > I am trying to create a large Hive table, with many columns and deeply
> > nested structs. It is failing with
> java.lang.ArrayIndexOutOfBoundsException:
> > 10.
> >
> >
> >
> > Before I spend a lot of time debugging my table declaration, is there
> > some limit here I should know about? Max number of columns? Max depth
> > of struct nesting?
> >
> >
> >
> > Thanks,
> >
> > Chuck
> >
> >
>
+
Connell, Chuck 2012-10-12, 14:17
+
Sadananda Hegde 2012-10-14, 00:31
+
Connell, Chuck 2012-10-14, 00:50
+
Sadananda Hegde 2012-10-15, 16:46
+
Connell, Chuck 2012-10-14, 00:48