Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Schema evolution and projection


+
Chris Laws 2013-02-28, 13:21
+
Doug Cutting 2013-02-28, 17:53
+
Douglas Creager 2013-02-28, 21:01
+
Chris Laws 2013-02-28, 21:13
+
Douglas Creager 2013-02-28, 22:38
+
Chris Laws 2013-03-01, 01:50
Copy link to this message
-
Re: Schema evolution and projection
Chris,

If you want a field in your reader schema that is not present in your
writer schema, you have to set a default value — otherwise the reader
has no way of knowing how to fill in that Field_3! If no particular
default value makes sense, a standard technique is to make the field
type a union with null, and to make null the default value
(effectively making the field optional).

For example:

const char  EXTENDED_SCHEMA[] "{\"type\":\"record\",\
  \"name\":\"SimpleScehma\",\
  \"fields\":[\
     {\"name\": \"Field_1\", \"type\": \"int\"},\
     {\"name\": \"Field_2\", \"type\": \"int\"},\
     {\"name\": \"Field_3\", \"type\": [\"null\", \"int\"],
\"default\": null}]}";

To build your intuitive understanding of how schema evolution works,
you might find this post useful:
http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

Best,
Martin

On 1 March 2013 01:50, Chris Laws <[EMAIL PROTECTED]> wrote:
> Doug,
>
> I have updated my test code in line with your excellent example and I now
> have the projection aspect working well.
>
> Now... I'm stuck on a schema evolution test. Basically if I use your example
> as the foundation and I create a new schema based on the WRITER_SCHEMA in
> which I add a new field to the end (to model schema evolution) I receive an
> error when trying to create the writer_iface.
>
> writer_iface = avro_resolved_writer_new(writer_schema, reader_schema);
>
> "Reader field Field_3 doesn't appear in writer"
>
> Any chance you could extending your example to show the ability of Avro to
> read data from a data file using an evolved schema (say in a simple
> situation were a new field is added to the schema)?
>
> Regards,
> Chris
>
>
>
> On Fri, Mar 1, 2013 at 9:08 AM, Douglas Creager <[EMAIL PROTECTED]>
> wrote:
>>
>> > Thanks for the informative reply. I look forward to the example code,
>> > that is exactly what I'm after.
>> >
>> > I'm really struggling with my schema evolution testing. I thought I'd
>> > post a question about schema projection because it seemed simpler but I
>> > guess it also rests on creating a resolver. I have not found a clear and
>> > simple example of how to do it using avro-c. I've trawled the test code
>> > for examples but as I mention I can't find a clear and simple example.
>>
>> Alrighty, here you go:
>>
>> http://dcreager.github.com/avro-examples/resolved-writer.html
>>
>> And a git repo with the source code:
>>
>> https://github.com/dcreager/avro-examples/tree/master/resolved-writer
>>
>> I hope this helps — please let me know if you have any other questions.
>>
>> cheers
>> –doug
>>
>
+
Chris Laws 2013-03-01, 22:26
+
Douglas Creager 2013-03-01, 23:34
+
Chris Laws 2013-03-02, 02:05
+
Doug Cutting 2013-03-01, 22:47
+
Doug Cutting 2013-03-01, 01:30
+
Chris Laws 2013-03-01, 01:41
+
Chris Laws 2013-03-01, 01:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB