Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Avro Backed Hive tables


Copy link to this message
-
Avro Backed Hive tables
Folks,

Wanted to get some help or feedback from the community on this one:

We are trying to create a table backed by avro schema:
Hive Version: 0.10

This is the syntax we are using to create the table. As you can notice the avro schema is defined inline.
Question:-  If you look at the avro schema we have a union of two records and when we issue the command it successfully creates the table with only the first record PageView and the columns searchTerms & filterCategories. It completely ignored the second union record.
It is probably where hive is not supporting avro union records as part of the CREATE table statement. Any clues on this one would help.

CREATE TABLE foo
ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES ('avro.schema.literal'='{
      "type" : "record",
      "name" : "PageView",
      "fields" : [ {
        "name" : "searchTerms",
        "type" : {
          "type" : "array",
          "items" : "string"
        }
      }, {
        "name" : "filterCategories",
        "type" : {
          "type" : "array",
          "items" : "string"
        }
      } ]
    }, {
      "type" : "record",
      "name" : "CategoryView",
      "fields" : [ {
        "name" : "categoryId",
        "type" : "string"
      }, {
        "name" : "refinements",
        "type" : {
          "type" : "array",
          "items" : {
            "type" : "record",
            "name" : "Refinement",
            "fields" : [ {
              "name" : "attribute",
              "type" : "string"
            }, {
              "name" : "value",
              "type" : "string"
            } ]
          }
        },
        "default" : [ ]
      } ]
}');

Thanks,
murtaza

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB