Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Metrics: via Broker vs. Producer vs. Consumer


+
Otis Gospodnetic 2013-07-24, 14:29
+
Jay Kreps 2013-07-24, 15:22
+
Scott Clasen 2013-07-24, 16:25
+
Otis Gospodnetic 2013-07-24, 17:04
Copy link to this message
-
Re: Metrics: via Broker vs. Producer vs. Consumer
Yeah totally that was what I thought you are saying. So what we have is a
yammer metrics which allows you to plug in any metrics reporter (it will do
jmx or whatever). But all this does it let you customize how the individual
java processes output metrics. We don't solve the problem of putting all
these together into a central dashboard. But these can be hooked up to a
standard monitoring thingy that pulls in metrics from all the java
processes in your environment (brokers, consumers, non-kafka related
things, etc) and let's you see these stats in aggregate across all machines.

I'm actually kind of ignorant of the state of open source monitoring
thingies because at linkedin we have a pretty badass in-house thing that
does this and is used for everything. I guess a lot of people used to use
ganglia or nagios for this but nowadays I hear all the cool kids use
graphite?

At various times we have tossed around the idea of having a "Kafka
Monitoring Dashboard" that had a lot of stats on it. This would allow us to
make out-of-the-box monitoring a lot easier for Kafka. The feedback we have
gotten previously is that everyone has some inhouse tool they already use
for graphing/alerting/etc and they don't really want a one-off solution for
Kafka. This is definitely true for our own usage.

-Jay

On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:
 
+
Otis Gospodnetic 2013-07-25, 09:35
+
Jun Rao 2013-07-26, 05:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB