Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Flume C Client

Copy link to this message
Re: Flume C Client
Hari Shreedharan 2013-04-25, 17:05
Actually, the Avro C API is not likely to be compatible with Flume, so you should use Thrift or HTTP. Thrift is a mechanism designed for IPC, so it is likely to perform better than HTTP.

Hari Shreedharan
On Thursday, April 25, 2013 at 4:25 AM, Israel Ekpo wrote:

> Flume does not have any of its components or plugins implemented in C.
> So there is no C client for it. It is not really designed in a
> client-server architecture.
> It was designed intentionally for collecting, aggregating, and transporting
> large amounts of data as events within its internal set of components.
> Nevertheless, there are components within this architecture (such as
> sources, channels and sinks) that interact with external endpoints in a
> client-server manner.
> Be that as it may, since your objective is to stream data to Flume, you can
> implement your own custom event-driven source for Flume and then connect to
> this source from your code to bridge in and move data from your system into
> flume.
> You could set up HTTPSource as the source and then use libcurl within your
> C code to send events to the HTTPSource for Flume.
> http://flume.apache.org/FlumeUserGuide.html#http-source
> http://curl.haxx.se/libcurl/c/
> Apache Avro has a C api documented here
> http://avro.apache.org/docs/current/api/c/index.html
> I hope this helps.
> On 24 April 2013 19:22, Neil Collier <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I have setup flume in our organization using Java clients with the SDK.
> > There is a need for us to stream data to flume from our existing C code
> > into our flume avro source.
> >
> > Is there an available C client or documentation somewhere? Note: I am not
> > a C programmer.
> >
> > Thanks,
> >
> > Neil
> >
> >
> >
> >
> > This email and any files included with it may contain privileged,
> > proprietary and/or confidential information that is for the sole use
> > of the intended recipient(s). Any disclosure, copying, distribution,
> > posting, or use of the information contained in or attached to this
> > email is prohibited unless permitted by the sender. If you have
> > received this email in error, please immediately notify the sender
> > via return email, telephone, or fax and destroy this original transmission
> > and its included files without reading or saving it in any manner.
> > Thank you.
> >