Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Serializing nested records


Copy link to this message
-
Serializing nested records
yimca 2013-01-21, 14:14
How do I serialize a record containing a nested record?  There doesn't seem
to be any way to create a DataFileWriter without tieing it to a single
record type.

Here's the scenario: I've defined an Avro schema called TransactionStatic
with nested internal record called TransactionStaticComponent:

{
  "namespace": "com.wily.apm.blackjack",
  "type": "record",
  "name": "TransactionStatic",
  "fields":
   [
       {"name": "id", "type": "int"},
       {"name": "isIdLocal", "type": "boolean", "default": "true"},
       {"name": "contextPath",  "type": [{ "type": "array", "items":
"string" },"null"] },
       {"name": "components", "type":
           [{ "type": "array", "items" :
               [{ "type": "record",
                   "name": "TransactionStaticComponent",
                   "fields":
                   [                      
                       {"name": "id", "type": "int"},
                       {"name": "isIdLocal", "type": "boolean", "default":
"true"},
                       {"name": "contextPath", "type": [{ "type": "array",
"items": "string" },"null"] },
                       {"name": "application", "type" : ["string","null"]},
                       {"name": "class", "type" :  ["string","null"]},
                       {"name": "method", "type" :  ["string","null"]},
                       {"name": "lineNumber", "type" :  ["int","null"]},
                       {"name": "payload", "type": [{"type": "map",
"values": "string"},"null"] },
                       {"name": "components", "type": [{ "type": "array",
"items": "TransactionStaticComponent" }], "default": "null" }
                   ]
               }]
           }]
       }
   ]
}

This compiles clean and I'm able to create data in the schema.  However, if
I try to serialize a record:

DataFileWriter<TransactionStatic> staticWriter
               = new DataFileWriter<TransactionStatic>(new
SpecificDatumWriter<TransactionStatic>(schema));
ByteArrayOutputStream staticOutputStream = new ByteArrayOutputStream(1024);
staticWriter.create(TransactionStatic.SCHEMA$, staticOutputStream);
staticWriter.append(servletA);
staticWriter.close();

I get an Avro exception stating that TransactionStaticInstance is not
defined:

Exception in thread "main"
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.AvroRuntimeException: Unknown datum type:
[Lcom.wily.apm.blackjack.TransactionStaticComponent;@3ffa1b16
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263

How can I serialize a TransactionStatic?  Also, where did the "L" come from
in "Lcom.wily..."?

--
View this message in context: http://apache-avro.679487.n3.nabble.com/Serializing-nested-records-tp4025992.html
Sent from the Avro - Users mailing list archive at Nabble.com.