Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Metrics: via Broker vs. Producer vs. Consumer


Copy link to this message
-
Re: Metrics: via Broker vs. Producer vs. Consumer
On the broker side, we have jmx beans for producer/consumer request rate
and time from all clients. Each producer/consumer client has jmx beans that
tracks its own request rate and time.

Thanks,

Jun
On Thu, Jul 25, 2013 at 2:35 AM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Oh, I know about Yammer metrics, Ganglia, Graphite, and friends....
>
> My main Q was: can I get all Consumer and all Producer stats via Broker's
> JMX.  I think the answer is no :(
> So I'm forced to look at the JMX of individual applications that act as
> Kafka Consumers or Producers.
>
> Btw. this is for SPM -  http://sematext.com/spm - support for monitoring
> Kafka should be in the early August release.
>
> As a matter of fact, there is an SPM Reporter from Yammer metrics, so
> people should be able to get their Kafka 0.8 metrics graphed as custom
> metrics in SPM already.
>
> Thanks for the patience and the info, Jay!
>
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>
>
> >________________________________
> > From: Jay Kreps <[EMAIL PROTECTED]>
> >To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Otis Gospodnetic <
> [EMAIL PROTECTED]>
> >Sent: Thursday, July 25, 2013 1:07 AM
> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
> >
> >
> >Yeah totally that was what I thought you are saying. So what we have is a
> >yammer metrics which allows you to plug in any metrics reporter (it will
> do
> >jmx or whatever). But all this does it let you customize how the
> individual
> >java processes output metrics. We don't solve the problem of putting all
> >these together into a central dashboard. But these can be hooked up to a
> >standard monitoring thingy that pulls in metrics from all the java
> >processes in your environment (brokers, consumers, non-kafka related
> >things, etc) and let's you see these stats in aggregate across all
> machines.
> >
> >I'm actually kind of ignorant of the state of open source monitoring
> >thingies because at linkedin we have a pretty badass in-house thing that
> >does this and is used for everything. I guess a lot of people used to use
> >ganglia or nagios for this but nowadays I hear all the cool kids use
> >graphite?
> >
> >At various times we have tossed around the idea of having a "Kafka
> >Monitoring Dashboard" that had a lot of stats on it. This would allow us
> to
> >make out-of-the-box monitoring a lot easier for Kafka. The feedback we
> have
> >gotten previously is that everyone has some inhouse tool they already use
> >for graphing/alerting/etc and they don't really want a one-off solution
> for
> >Kafka. This is definitely true for our own usage.
> >
> >-Jay
> >
> >
> >
> >On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic <
> >[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> Ah, I think I didn't ask my question clearly.  Another try:
> >> * If I have a javaagent attached to the Kafka process, I'll be able to
> >> connect to its JMX and get all the Broker metrics for that Broker
> process.
> >> * If I have another Broker process, I'll need to attach my agent to this
> >> process, too, to get all Broker metrics associated with this second
> Broker
> >> process.
> >>
> >>
> >> So far OK - like you said, I can sum, average, etc.
> >>
> >> But what if I want to get all Producer metrics?  What do I need to do? I
> >> *believe* I would have to attach the javaagent to whichever app is
> acting
> >> as a Kafka Consumer and get Consumer stats from the JMX associated with
> the
> >> JVM process running that app.
> >> Is this correct?
> >> Is there any way to avoid that and get all Consumer and all Producer
> >> metrics using the javaagent attached to one of the Broker processes?
> >>
> >> Thanks,
> >> Otis
> >> ----
> >> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> >> http://sematext.com/spm
> >>
> >>
> >>
> >>
> >> >________________________________
> >> > From: Jay Kreps <[EMAIL PROTECTED]>