Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Separators in struct


Copy link to this message
-
Re: Separators in struct
Issue logged. https://issues.apache.org/jira/browse/HIVE-3253

On Wed, Jul 11, 2012 at 4:13 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> We surely can make it bigger. However there is a sublte problem. I
> have ran into maximum column length limitations in the metastore with
> heavily nested columns. MySQL varchar maxes etc. You should open a
> jirra issues on issues.apache.org/jira/hive
>
> Edward
>
> On Wed, Jul 11, 2012 at 5:10 PM, [EMAIL PROTECTED]
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > I am not sure I understand the significance of separators very well in
> case
> > of structs. For instance, for deeply nested structs I usually hit this
> > exception:
> >
> > java.lang.ArrayIndexOutOfBoundsException: 9
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> >
> > Digging deeper into the code I found that the size of separators have
> been
> > hard-coded to be an array of size "8" with a comment on it.
> >
> > // Read the separators: We use 8 levels of separators by default, but we
> > should change this when we allow users to specify more than 10 levels of
> > separators through DDL.
> >
> > serdeParams.separators = new byte[8];
> >
> >
> > If someone can explain this to me, I would really appreciate that. Also
> is
> > there a way to change the number of separators so that this exception is
> not
> > thrown?
> >
> > Thanks,
> >
> >
> > --
> > Swarnim
>

--
Swarnim
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB