Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # dev - C library

Douglas Creager 2010-10-14, 01:45
Bruce Mitchener 2010-10-14, 02:01
Douglas Creager 2010-10-14, 02:43
Copy link to this message
Re: C library
Bruce Mitchener 2010-10-14, 02:56
On Thu, Oct 14, 2010 at 9:43 AM, Douglas Creager <[EMAIL PROTECTED]>wrote:

> > I didn't write the existing C library, but I've used it and done some
> work
> > on it.  I'm currently writing my own more minimal and more streamlined
> > implementation of Avro in C ...
> >
> > The issues with glib specifically would be:
> >
> >
> >    - The license is not acceptable for use here. (LGPL)
> >    - It is much bigger than what is needed here.
> >    - Many of the things that make it more general would also make it
> slower
> >    than necessary. The existing C code isn't a speed demon either, but
> the C
> >    implementation should aim for solid performance.
> Ha!  Well you're certainly right that glib's not small.  Are you sure
> about the speed claims, though?  Would it be worth banging out a LGPL,
> glib-based prototype to do some initial tests?

Not to me. :)  I'm assuming that you mean something that uses GValue and so

I don't want the overhead of that sort of thing at all in my C code.  I'm
supporting resource constrained platforms, so I just want to go from my C
struct straight to a buffer without building an intermediate data structure.

Along those lines, you mention a new C implementation you're working on.
>  Is that something that you plan to fold back into the main libavro?  Or
> will it be separate?  The spec provides a good basis for defining how
> well different implementations interoperate, but so far it seems like
> everything has been folded into the single, Apache-sponsored project.
> Is there interest in having independent implementations?

I'm not sure what will happen with my implementation yet. I'm inclined to
say that it'll be opened up (Apache 2 licensed) but it will depend on the
quality of the code with respect to use outside of my product and other
factors.  More likely is that what I'm doing is going to serve as a test bed
for ideas and an implementation approach that can be merged into the Apache
Avro C implementation in the future.

As part of my own implementation of Avro in C, I'm also working on a binary
RPC protocol for talking with Cloudera Flume, so I have a bit more
motivation to get it opened up ...

 - Bruce
Douglas Creager 2010-10-14, 03:49
Bruce Mitchener 2010-10-14, 03:58
Matt Massie 2010-10-14, 15:59
Douglas Creager 2010-10-15, 22:02