Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> how to print the channel capacity


+
liuyongbo 2013-05-15, 05:37
+
Nitin Pawar 2013-05-15, 08:20
+
Nitin Pawar 2013-05-15, 08:46
+
Nitin Pawar 2013-05-15, 08:49
+
liuyongbo 2013-05-15, 11:39
+
Nitin Pawar 2013-05-15, 11:48
+
Paul Chavez 2013-05-15, 16:36
Copy link to this message
-
Re: how to print the channel capacity
We do the same thing, but with Collectd as our graphing/collection mechanism. I am actually going to do a blog post in the next day or two with the code to our flume data collection script, and some example graphs/etc. We've done a similar thing with Zookeeper monitoring (http://engblog.nextdoor.com/post/49942956311/apache-zookeeper-performance-monitoring).

--Matt

On May 15, 2013, at 9:36 AM, Paul Chavez <[EMAIL PROTECTED]> wrote:

> There are a few ways to monitor flume in operation. We use the JSON reporting, which is available via 'http://<agent address>:<port>/metrics'. You need to start the agent with the following parameters to get this interface:
> -Dflume.monitoring.type=http -Dflume.monitoring.port=34545
> We use cacti to graph channel size both as a percentage of maximum and absolute number of events in channel. This provides warning if the sinks cannot keep up with the sources.
>  
> We also graph ingress/egress event counts, much like a network bandwidth graph, for some channels to get an idea of the throughput and to see if sources/sinks are running at same speed.
> From: liuyongbo [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 14, 2013 10:38 PM
> To: [EMAIL PROTECTED]
> Subject: how to print the channel capacity
>
> Hi:
>          I’m using flume to pass log data to mongodb, but I find that some data lose when the pressure is in high level, so I want to know the max request that flume can hold and need to print the capacity.but I can not find the proper way to do this instead of change the source code. Any ideas?
>          thanks

+
Matt Wise 2013-05-15, 18:05
+
liuyongbo 2013-05-16, 08:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB