Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Separators in struct


Copy link to this message
-
Re: Separators in struct
We surely can make it bigger. However there is a sublte problem. I
have ran into maximum column length limitations in the metastore with
heavily nested columns. MySQL varchar maxes etc. You should open a
jirra issues on issues.apache.org/jira/hive

Edward

On Wed, Jul 11, 2012 at 5:10 PM, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> I am not sure I understand the significance of separators very well in case
> of structs. For instance, for deeply nested structs I usually hit this
> exception:
>
> java.lang.ArrayIndexOutOfBoundsException: 9
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
>
> Digging deeper into the code I found that the size of separators have been
> hard-coded to be an array of size "8" with a comment on it.
>
> // Read the separators: We use 8 levels of separators by default, but we
> should change this when we allow users to specify more than 10 levels of
> separators through DDL.
>
> serdeParams.separators = new byte[8];
>
>
> If someone can explain this to me, I would really appreciate that. Also is
> there a way to change the number of separators so that this exception is not
> thrown?
>
> Thanks,
>
>
> --
> Swarnim
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB