Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Schema evolution and projection


+
Chris Laws 2013-02-28, 13:21
+
Doug Cutting 2013-02-28, 17:53
+
Douglas Creager 2013-02-28, 21:01
+
Chris Laws 2013-02-28, 21:13
+
Douglas Creager 2013-02-28, 22:38
+
Chris Laws 2013-03-01, 01:50
Copy link to this message
-
Re: Schema evolution and projection
Chris,

If you want a field in your reader schema that is not present in your
writer schema, you have to set a default value — otherwise the reader
has no way of knowing how to fill in that Field_3! If no particular
default value makes sense, a standard technique is to make the field
type a union with null, and to make null the default value
(effectively making the field optional).

For example:

const char  EXTENDED_SCHEMA[] "{\"type\":\"record\",\
  \"name\":\"SimpleScehma\",\
  \"fields\":[\
     {\"name\": \"Field_1\", \"type\": \"int\"},\
     {\"name\": \"Field_2\", \"type\": \"int\"},\
     {\"name\": \"Field_3\", \"type\": [\"null\", \"int\"],
\"default\": null}]}";

To build your intuitive understanding of how schema evolution works,
you might find this post useful:
http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

Best,
Martin

On 1 March 2013 01:50, Chris Laws <[EMAIL PROTECTED]> wrote:
> Doug,
>
> I have updated my test code in line with your excellent example and I now
> have the projection aspect working well.
>
> Now... I'm stuck on a schema evolution test. Basically if I use your example
> as the foundation and I create a new schema based on the WRITER_SCHEMA in
> which I add a new field to the end (to model schema evolution) I receive an
> error when trying to create the writer_iface.
>
> writer_iface = avro_resolved_writer_new(writer_schema, reader_schema);
>
> "Reader field Field_3 doesn't appear in writer"
>
> Any chance you could extending your example to show the ability of Avro to
> read data from a data file using an evolved schema (say in a simple
> situation were a new field is added to the schema)?
>
> Regards,
> Chris
>
>
>
> On Fri, Mar 1, 2013 at 9:08 AM, Douglas Creager <[EMAIL PROTECTED]>
> wrote:
>>
>> > Thanks for the informative reply. I look forward to the example code,
>> > that is exactly what I'm after.
>> >
>> > I'm really struggling with my schema evolution testing. I thought I'd
>> > post a question about schema projection because it seemed simpler but I
>> > guess it also rests on creating a resolver. I have not found a clear and
>> > simple example of how to do it using avro-c. I've trawled the test code
>> > for examples but as I mention I can't find a clear and simple example.
>>
>> Alrighty, here you go:
>>
>> http://dcreager.github.com/avro-examples/resolved-writer.html
>>
>> And a git repo with the source code:
>>
>> https://github.com/dcreager/avro-examples/tree/master/resolved-writer
>>
>> I hope this helps — please let me know if you have any other questions.
>>
>> cheers
>> –doug
>>
>
+
Chris Laws 2013-03-01, 22:26
+
Douglas Creager 2013-03-01, 23:34
+
Chris Laws 2013-03-02, 02:05
+
Doug Cutting 2013-03-01, 22:47
+
Doug Cutting 2013-03-01, 01:30
+
Chris Laws 2013-03-01, 01:41
+
Chris Laws 2013-03-01, 01:14