Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Customr Sink with bulk processing


Copy link to this message
-
Customr Sink with bulk processing
Hi, I'm a newbie in a flume world, and I have a question about the custom
sink.

I would like to use flume process to gather log files and push them into
the database.
I know that I have to implement the sink because we're interested in very
specific database with a very specific table structure which is not
supported by flume.

So, I need to create a custom Sink which is well explained in flume
documentation. However I would like to process the events from the channel
in bulk (use bulk insert into db).

What is the best approach to implement such a code?
I'm not sure how to implement this logic given the fact that the event
taken from the channel can't be put back/we don't have something like
peek() in the channel.

Thanks a lot in advance and have a nice day :)

Mark