Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Understanding how to monitor using JMX attributes


+
Mike Heffner 2012-11-20, 03:10
Copy link to this message
-
Re: Understanding how to monitor using JMX attributes
Instead of using jmx, have you tried using ConsumerOffsetChecker to figure
out the consumer lag?

Thanks,

Jun

On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to write a custom monitoring script for our Kafka setup and
> would like some help understanding how to interpret the JMX attributes.
>
> In our setup, the consumers are writing their current offset to a path in
> ZK. This is the value they are getting back from a call
> to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
>
> {"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
> "port"=>9092}, "topic"=>"mcommits"}
>
> Using the MX4J interface, I poll the
> bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> attribute values:
>
> {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
>
> At the time both of these values were snapshotted, this consumer was close
> to the end of the log file. In that case, I would expect both offsets to be
> fairly similar, however the consumer offset is >> the producer log offset,
> which doesn't make sense.
>
> Clearly there is something I'm not understanding. How do I use the JMX
> attributes to calculate how far behind the consumer is from the end of the
> log file? In this scenario the consumer offset is >> both the CurrentOffset
> value and the Size value. Is there a way of interpreting these values that
> I'm not seeing?
>
>
> Thanks,
>
> Mike
>
> --
>
>   Mike Heffner <[EMAIL PROTECTED]>
>   Librato, Inc.
>
+
Mike Heffner 2012-11-20, 16:03
+
Evan Chan 2012-11-20, 17:11
+
Mike Heffner 2012-11-20, 21:07
+
Jun Rao 2012-11-20, 17:12
+
Mike Heffner 2012-11-20, 21:12
+
Jun Rao 2012-11-21, 05:08
+
Mike Heffner 2012-11-21, 15:57
+
Joel Koshy 2012-11-21, 18:47
+
Mike Heffner 2012-11-21, 22:01
+
Joel Koshy 2012-11-21, 22:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB