Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Avro-c with libhdfs


Copy link to this message
-
Re: Avro-c with libhdfs
> I would like to write Avro files to HDFS using C as the programming
> language. I have a simple queston: Can I use the avro-c API with
> libhdfs? I tried the hdfs-nfs-proxy, but it doesn't support the append
> operation.

I don't have any experience with libhdfs, so I'm not sure.  The Avro C
API lets you write Avro data files to a file on the local filesystem,
identified by a `const char *` filename, or a `FILE *` instance that
you've already opened, so my guess is that, out of the box, the only way
it would work would be to create the file locally and then transfer the
existing local file to HDFS.

To get it to work directly with libhdfs, we'd probably need to add a new
API for providing a callback function, which the Avro bindings would
call whenever data needs to be written.  (This is already what happens
under the covers, so it would really just be exposing parts of the
existing functionality in the public API.)

cheers
–doug

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB