Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Metrics: via Broker vs. Producer vs. Consumer

Copy link to this message
Re: Metrics: via Broker vs. Producer vs. Consumer
Valid assumptions IMO

Sent from my iPhone

On Jul 24, 2013, at 8:22 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yeah all our monitoring is just of the local process (basically just a
> counter exposed through yammer metrics which support jmx and other
> outputs). If I understand what you want instead of having a counter that
> tracks, say, produce requests per second for a single broker you want one
> that covers the whole cluster. Obviously this would require collecting the
> local count and aggregating across all the brokers.
> Our assumption is that you already have a separate monitoring system which
> can slurp all these up, aggregate them, graph them, and alert off them.
> There are a number of open source thingies like this and I think most
> bigger shops have something they use. Our assumption is that trying to do a
> kafka-specific monitoring system wouldn't work for most people because they
> are wedded to their current setup and just want to integrate with that.
> I'm not sure how valid any of those assumptions actually are.
> -Jay
> On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
>> Hi,
>> I was looking at
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> and noticed there is no information about which metrics are available
>> in which process/JVM/JMX.
>> Some are available in the Broker process, but some are only available
>> from the JVM running Consumer and some only from the JVM running
>> Producer.  And yet some Producer and Consumer metrics are, I *believe*
>> available from Broker's JMX.
>> Would it be possible for somebody in the know to mark the metrics in
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> so one can tell where to get it?
>> Also, why is it that the Broker process doesn't have *all* metrics,
>> including Producer and Consumer one?  Is that because there can be N
>> Brokers and each P or C talk to one Broker at a time and thus there is
>> no single process/JMX that can know *all* stats for *all* Brokers and
>> for *all* Ps and Cs?
>> Thank you!
>> Otis
>> --
>> Performance Monitoring -- http://sematext.com/spm
>> Solr & ElasticSearch Support -- http://sematext.com/