Flume, mail # dev - Customr Sink with bulk processing - 2014-07-17, 15:44
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Customr Sink with bulk processing
Hi, I'm a newbie in a flume world, and I have a question about the custom
sink.

I would like to use flume process to gather log files and push them into
the database.
I know that I have to implement the sink because we're interested in very
specific database with a very specific table structure which is not
supported by flume.

So, I need to create a custom Sink which is well explained in flume
documentation. However I would like to process the events from the channel
in bulk (use bulk insert into db).

What is the best approach to implement such a code?
I'm not sure how to implement this logic given the fact that the event
taken from the channel can't be put back/we don't have something like
peek() in the channel.

Thanks a lot in advance and have a nice day :)

Mark

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB