Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # dev >> Re: Effort towards Avro 2.0?


Copy link to this message
-
Re: Effort towards Avro 2.0?
On Wed, Dec 4, 2013 at 12:18 PM, Douglas Creager <[EMAIL PROTECTED]>wrote:

> >    - unwieldy because as a user, I'll have to encode and decode the bytes
>
>    field manually everytime I want to access this field from the original
> >    record, unless I keep track of the decoded extension externally to the
> >    Avro record.
>
> Can you handle this in the middleware?  I.e., have the middleware decode
> the bytes field before passing control to the user code.  That's better
> from a decoupling standpoint anyway, since the user code shouldn't care
> what middleware is wrapping it.
>

Well, I guess one can always handle such things externally to Avro.

I don't necessarily want to dissociate the generic part of a record and its
extensions.
In some cases, the extension in isolation doesn't make sense without the
context defined by the generic part of the record.

> When you write a middleware that lets users define custom types,
> > extensions are pretty much required.
>
> I guess my main point is that we already have two mechanisms for dealing
> with user extensions (schema resolution and Doug's bytes field
> proposal), both of which work just fine at runtime without rebuilding or
> restarting your code.  In general, I think it's better if we can solve a
> problem at the library or application level, without having to update
> the spec.
>

One use-case I have involves recursive records that can be extended by
users.
Think of a tree of operations (eg. math/logic operations) that can be
extended by users with new operations.
Without support from Avro, I have to manually decode each node,
recursively, dealing with bytes at every step.
With extensions built-in, Avro could decode the entire tree in one single
step.

Or maybe I misunderstand what you are suggesting I do instead?

C.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB