Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Customr Sink with bulk processing


Copy link to this message
-
Customr Sink with bulk processing
Hi, I'm a newbie in a flume world, and I have a question about the custom
sink.

I would like to use flume process to gather log files and push them into
the database.
I know that I have to implement the sink because we're interested in very
specific database with a very specific table structure which is not
supported by flume.

So, I need to create a custom Sink which is well explained in flume
documentation. However I would like to process the events from the channel
in bulk (use bulk insert into db).

What is the best approach to implement such a code?
I'm not sure how to implement this logic given the fact that the event
taken from the channel can't be put back/we don't have something like
peek() in the channel.

Thanks a lot in advance and have a nice day :)

Mark

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB