Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # dev - Re: Effort towards Avro 2.0?

Copy link to this message
Re: Effort towards Avro 2.0?
Christophe Taton 2013-12-05, 07:40
On Wed, Dec 4, 2013 at 12:18 PM, Douglas Creager <[EMAIL PROTECTED]>wrote:

> >    - unwieldy because as a user, I'll have to encode and decode the bytes
>    field manually everytime I want to access this field from the original
> >    record, unless I keep track of the decoded extension externally to the
> >    Avro record.
> Can you handle this in the middleware?  I.e., have the middleware decode
> the bytes field before passing control to the user code.  That's better
> from a decoupling standpoint anyway, since the user code shouldn't care
> what middleware is wrapping it.

Well, I guess one can always handle such things externally to Avro.

I don't necessarily want to dissociate the generic part of a record and its
In some cases, the extension in isolation doesn't make sense without the
context defined by the generic part of the record.

> When you write a middleware that lets users define custom types,
> > extensions are pretty much required.
> I guess my main point is that we already have two mechanisms for dealing
> with user extensions (schema resolution and Doug's bytes field
> proposal), both of which work just fine at runtime without rebuilding or
> restarting your code.  In general, I think it's better if we can solve a
> problem at the library or application level, without having to update
> the spec.

One use-case I have involves recursive records that can be extended by
Think of a tree of operations (eg. math/logic operations) that can be
extended by users with new operations.
Without support from Avro, I have to manually decode each node,
recursively, dealing with bytes at every step.
With extensions built-in, Avro could decode the entire tree in one single

Or maybe I misunderstand what you are suggesting I do instead?