Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Using Python and Flume to store avro data

Copy link to this message
Re: Using Python and Flume to store avro data
The next release of Flume-1.3.0 adds support for an HTTP source, which will allow you to send data to Flume via HTTP/JSON(the representation of the data is pluggable - but a JSON representation is default). You could use this to write data to Flume from Python, which I believe has good http and json support.

Hari Shreedharan
On Thursday, November 8, 2012 at 10:45 AM, Bart Verwilst wrote:

> Hi,
> I've been spending quite a few hours trying to push avro data to Flume
> so i can store it on HDFS, this all with Python.
> It seems like something that is impossible for now, since the only way
> to push avro data to Flume is by the use of deprecated thrift binding
> that look pretty cumbersome to get working.
> I would like to know what's the best way to import avro data into Flume
> with Python? Maybe Flume isnt the right tool and I should use something
> else? My goal is to have multiple python workers pushing data to HDFS
> which ( by means of Flume in this case ) consolidates this all in 1 file
> there.
> Any thoughts?
> Thanks!
> Bart