Flume, mail # dev - Flume Avro Source and Avro C api - 2014-08-25, 21:22
Solr & Elasticsearch trainings in New York & San Fransisco [more info]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Flume Avro Source and Avro C api
Hi there,
I'm trying to write a small pice of code using Avro C api and then use a
Flume avro sink to write the collected and serialized data to hadoop HDFS.

I would like to use Avro C api as all the other lib/code I'm using to
generate/collect my data is in c language and it's already written. So I'm
adding Avro output to an already existing software.

As I'm reading from avro C API does not support RPC call yet.

So Is there a way to seamless integrate a Flume avro source with avro C api?
Is there any already written example?

Or I have to create my avro file with avro C api (i.e. my_data_file.avro)
on the file system
and then use the already provided avro client

bin/flume-ng avro-client -H localhost -p 41414 -F my_data_file.avro

in order to have the file "read" through the Flume avro source and sent to
the Flume sink on hdfs?

Isn't it possible only to configure the Flume Avro source to listen on a
particulary port and then use Avro C api to send the message?

(I did a quick search on avro C api, but i couln't find any function I
could call to answer my previous question)

Kind regard.

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB