Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Avro-c with libhdfs


Copy link to this message
-
Re: Avro-c with libhdfs
Douglas Creager 2013-10-21, 12:57
> I would like to write Avro files to HDFS using C as the programming
> language. I have a simple queston: Can I use the avro-c API with
> libhdfs? I tried the hdfs-nfs-proxy, but it doesn't support the append
> operation.

I don't have any experience with libhdfs, so I'm not sure.  The Avro C
API lets you write Avro data files to a file on the local filesystem,
identified by a `const char *` filename, or a `FILE *` instance that
you've already opened, so my guess is that, out of the box, the only way
it would work would be to create the file locally and then transfer the
existing local file to HDFS.

To get it to work directly with libhdfs, we'd probably need to add a new
API for providing a callback function, which the Avro bindings would
call whenever data needs to be written.  (This is already what happens
under the covers, so it would really just be exposing parts of the
existing functionality in the public API.)

cheers
–doug