Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Separators in struct


Copy link to this message
-
Separators in struct
kulkarni.swarnim@...) 2012-07-11, 21:10
Hello,

I am not sure I understand the significance of separators very well in case
of structs. For instance, for deeply nested structs I usually hit this
exception:

java.lang.ArrayIndexOutOfBoundsException: 9
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)

Digging deeper into the code I found that the size of separators have been
hard-coded to be an array of size "8" with a comment on it.

// Read the separators: We use 8 levels of separators by default, but we should
change this when we allow users to specify more than 10 levels of
separators through DDL.

serdeParams.separators = new byte[8];
If someone can explain this to me, I would really appreciate that. Also is
there a way to change the number of separators so that this exception is
not thrown?

Thanks,

--
Swarnim