Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Design problem while monitoring Flume


+
Anat Rozenzon 2013-08-28, 07:09
+
Juhani Connolly 2013-08-28, 08:45
+
Anat Rozenzon 2013-08-28, 12:59
+
Israel Ekpo 2013-08-28, 14:00
+
Juhani Connolly 2013-08-29, 02:09
Copy link to this message
-
Re: Design problem while monitoring Flume
Have you tried using the built-in counters that Flume exposes?  All of
these counters are published in JXM under the "org.apache.flume" namespace.
You could use counters from the first source in the pipeline to measure how
many true accepted events were taken up by Flume.

Also, related discussion is going in FLUME-2173 in case you are interested.

https://issues.apache.org/jira/browse/FLUME-2173

Regards,
Arvind Prabhakar
On Wed, Aug 28, 2013 at 12:09 AM, Anat Rozenzon <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I want to get some statistics out of Flume (For example, how many records
> were collected, How many files etc.).
> I've written my own interceptor that updates an MBean whenever records
> arrive.
>
> I've also written a MonitorServices that collects the data from the MBean
> every X minutes and send it to a database.
>
> My problem is that sometimes events are resent again from the source, I
> saw that while debugging.
> Not sure why... maybe because of a timeout while sending to the sink?
>
> Anyway, if this happens in production it will corrupt my statistics.
>
> Is there any way I can know that an event have failed reaching the sink
> eventhough it passed the interceptor?
> Is there a better place to collect such statistics than an interceptor?
>
> Thanks
> Anat
>