Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> alter table set serde changes table schema?


Copy link to this message
-
alter table set serde changes table schema?
Hey hive gurus -

I'm creating some tables to represent thrift structs, and noticed when
setting the serde the table schema changes. For example:

hive> describe foo_test;
OK
start_time_ms bigint
bar struct<transaction_id:string>

Set the serde:

hive> alter table foo_test
    set serde "com.twitter.elephantbird.hive.serde.ThriftSerDe"
    with serdeproperties ("serialization.class"="com.foo.Foo");

Now the schema is different. Notice how the field type is a class name - it
matches the thrift struct type stored in the field.

hive> describe foo_test;

                   OK
start_time_ms bigint from deserializer
bar com.foo.Bar from deserializer

I've spent some time watching how the object inspectors work in a debugger,
and its correctly identifying the subfields inside field "bar". However,
the table column type gets set to the class name, instead of expanding to
the schema of the thrift struct.

Can anyone clue me into how this is intended to work? I'm using the
standard thrift object inspector, and the serde I'm writing is:

https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/hive/serde/ThriftSerDe.java

Any help in understanding how to correctly set the table schema would be
greatly appreciated.

Thanks!
Travis
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB