Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # dev >> C library

Douglas Creager 2010-10-14, 01:45
Bruce Mitchener 2010-10-14, 02:01
Douglas Creager 2010-10-14, 02:43
Bruce Mitchener 2010-10-14, 02:56
Douglas Creager 2010-10-14, 03:49
On Thu, Oct 14, 2010 at 10:49 AM, Douglas Creager <[EMAIL PROTECTED]>wrote:

> > Not to me. :)  I'm assuming that you mean something that uses GValue and
> so
> > on?
> Ah, whoops.  No, I'm not suggesting GValue.  *shudder*

> I was thinking more like using:
>  • GObject for the schema/datum subclassing
>  • GHashTable or GTree to store a record schema's fields, etc.
>  • GIO for the generic I/O interfaces
>  • GQuark instead of the atom implementation that was checked in and
>   then reverted
Okay, I see ... but that can't happen within the Apache implementation due
to licensing issues.  (It also doesn't work for my usages because it isn't
clear that LGPL code can be shipped at all legally on some of my target
> > I don't want the overhead of that sort of thing at all in my C code.  I'm
> > supporting resource constrained platforms, so I just want to go from my C
> > struct straight to a buffer without building an intermediate data
> structure.
> We're in violent agreement.  One thing I've started experimenting with
> is a “streaming” API, so that instead of creating a tree of avro_datum_t
> instances, the file reader calls a series of callback functions as each
> bit of data is encountered.  We're generating Avro files from an
> existing C network sensor application, and it's a bit of overhead (in
> both code and speed) to have to move between our actual data types and
> the avro_datum_t instances.

Okay, then we're talking about similar things.  But you can also just
generate code and then you don't need schemas or anything else at runtime,

What I'm doing is just a low level API that I can use from generated code. I
don't need (or want) schemas or anything else in the way.

Maybe we should talk more off-list.

 - Bruce
Matt Massie 2010-10-14, 15:59
Douglas Creager 2010-10-15, 22:02