Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Design problem while monitoring Flume

Copy link to this message
Design problem while monitoring Flume

I want to get some statistics out of Flume (For example, how many records
were collected, How many files etc.).
I've written my own interceptor that updates an MBean whenever records

I've also written a MonitorServices that collects the data from the MBean
every X minutes and send it to a database.

My problem is that sometimes events are resent again from the source, I saw
that while debugging.
Not sure why... maybe because of a timeout while sending to the sink?

Anyway, if this happens in production it will corrupt my statistics.

Is there any way I can know that an event have failed reaching the sink
eventhough it passed the interceptor?
Is there a better place to collect such statistics than an interceptor?