Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Properties on union schema?


+
Christophe Taton 2013-07-14, 02:01
+
Connor Doyle 2013-07-14, 02:15
+
Christophe Taton 2013-07-15, 17:57
+
Doug Cutting 2013-07-15, 18:25
+
Christophe Taton 2013-07-16, 07:40
+
Doug Cutting 2013-07-16, 18:13
+
Christophe Taton 2013-07-18, 17:23
Copy link to this message
-
Re: Properties on union schema?
A concern is that folks might write data, then later find they cannot
read it using tools they might otherwise expect to read it with.  They
might not initially use a feature that triggers the incompatibility,
only to find it later, when they've already committed to the newer
version.

To reduce the likelihood of such scenarios we should alter the file's
magic number when incompatible changes are included.  We can also
increment the project and specification's major version number.  If
such changes are optional then we might only increment the file's
magic number when they're enabled.  Then the project's major version
number might not be incremented until these are made default.

Since there are multiple implementations of Avro it behooves us not to
introduce too many magic numbers that will need long-term support.
(Once a format is used in production, it probably needs to be
supported forever.)  Perhaps we should start considering a list of
changes to be made in Avro 2.0's schema language.  We might use an
"experimental" magic number during development that prints obnoxious
warnings when features are enabled so that folks are well warned not
to use them in production until the full set of new features is
determined.

What do others think?

Doug

On Thu, Jul 18, 2013 at 10:23 AM, Christophe Taton <[EMAIL PROTECTED]> wrote:
> Hi Doug,
>
> I realize I have strong interests in several issues that would imply
> forward-incompatible changes (eg. AVRO-248 and AVRO-530).
> Do you believe it is possible to introduce part of these as experimental
> features that would be disabled by default, or would you rather isolate such
> changes in completely separate releases? I would happily invest energy in
> either approach.
>
> Thanks,
> C.
>
>
> On Tue, Jul 16, 2013 at 11:13 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>>
>> We've promised to maintain forward and backward compatibility until
>> version 2.0.  There have been a number of incompatible schema changes
>> requested over the history of Avro, but none have yet been made.  If
>> we ever decide to make any, we should perhaps try to make them all at
>> once and provide good conversion tools.
>>
>> Doug
>>
>> On Tue, Jul 16, 2013 at 12:40 AM, Christophe Taton <[EMAIL PROTECTED]>
>> wrote:
>> > Hi Doug,
>> >
>> > Thanks for the explanation.
>> > How far back is it reasonable to maintain backward and/or forward
>> > compatibility?
>> >
>> > Assuming there is a limit to the forward compatibility requirement,
>> > could we
>> > introduce the ability to read schemas with extended union descriptors,
>> > without the ability to write such descriptors, and introduce the ability
>> > to
>> > write after enough releases have passed?
>> >
>> > Thanks,
>> > C.
>> >
>> >
>> > On Mon, Jul 15, 2013 at 11:25 AM, Doug Cutting <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> The reason is simply that the JSON syntax for union schemas doesn't
>> >> permit properties.  To support properties we'd need to add an
>> >> alternate syntax for unions, but I don't see how to do that
>> >> compatibly, so that an old client seeing one of the new union schemas
>> >> would still be able to process the data.
>> >>
>> >> For example, the syntax might be something like:
>> >>
>> >> {"type":"union", "branches":[...], "prop1":"val1", ...}
>> >>
>> >> But that would cause errors in every existing implementation.
>> >>
>> >> Doug
>> >>
>> >> On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi,
>> >> > I'm toying with a few changes to provide alternative representations
>> >> > of
>> >> > union fields in Java (somewhat related to
>> >> > https://issues.apache.org/jira/browse/AVRO-248).
>> >> > To experiment with this, I'd like to set properties on union schemas,
>> >> > but
>> >> > properties are currently disabled on unions.
>> >> > Is there a particular reason for this, or is it a reasonable change
>> >> > to
>> >> > allow
>> >> > properties on unions?
>> >> > Thanks,
>> >> > C.
>> >