Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Serializing nested records


Copy link to this message
-
Serializing nested records
How do I serialize a record containing a nested record?  There doesn't seem
to be any way to create a DataFileWriter without tieing it to a single
record type.

Here's the scenario: I've defined an Avro schema called TransactionStatic
with nested internal record called TransactionStaticComponent:

{
  "namespace": "com.wily.apm.blackjack",
  "type": "record",
  "name": "TransactionStatic",
  "fields":
   [
       {"name": "id", "type": "int"},
       {"name": "isIdLocal", "type": "boolean", "default": "true"},
       {"name": "contextPath",  "type": [{ "type": "array", "items":
"string" },"null"] },
       {"name": "components", "type":
           [{ "type": "array", "items" :
               [{ "type": "record",
                   "name": "TransactionStaticComponent",
                   "fields":
                   [                      
                       {"name": "id", "type": "int"},
                       {"name": "isIdLocal", "type": "boolean", "default":
"true"},
                       {"name": "contextPath", "type": [{ "type": "array",
"items": "string" },"null"] },
                       {"name": "application", "type" : ["string","null"]},
                       {"name": "class", "type" :  ["string","null"]},
                       {"name": "method", "type" :  ["string","null"]},
                       {"name": "lineNumber", "type" :  ["int","null"]},
                       {"name": "payload", "type": [{"type": "map",
"values": "string"},"null"] },
                       {"name": "components", "type": [{ "type": "array",
"items": "TransactionStaticComponent" }], "default": "null" }
                   ]
               }]
           }]
       }
   ]
}

This compiles clean and I'm able to create data in the schema.  However, if
I try to serialize a record:

DataFileWriter<TransactionStatic> staticWriter
               = new DataFileWriter<TransactionStatic>(new
SpecificDatumWriter<TransactionStatic>(schema));
ByteArrayOutputStream staticOutputStream = new ByteArrayOutputStream(1024);
staticWriter.create(TransactionStatic.SCHEMA$, staticOutputStream);
staticWriter.append(servletA);
staticWriter.close();

I get an Avro exception stating that TransactionStaticInstance is not
defined:

Exception in thread "main"
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.AvroRuntimeException: Unknown datum type:
[Lcom.wily.apm.blackjack.TransactionStaticComponent;@3ffa1b16
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263

How can I serialize a TransactionStatic?  Also, where did the "L" come from
in "Lcom.wily..."?

--
View this message in context: http://apache-avro.679487.n3.nabble.com/Serializing-nested-records-tp4025992.html
Sent from the Avro - Users mailing list archive at Nabble.com.
+
robturner 2013-01-21, 22:29
+
yimca 2013-01-22, 18:34
+
Doug Cutting 2013-01-21, 22:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB