Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Understanding how to monitor using JMX attributes

Copy link to this message
Re: Understanding how to monitor using JMX attributes
Instead of using jmx, have you tried using ConsumerOffsetChecker to figure
out the consumer lag?



On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Hi,
> I am trying to write a custom monitoring script for our Kafka setup and
> would like some help understanding how to interpret the JMX attributes.
> In our setup, the consumers are writing their current offset to a path in
> ZK. This is the value they are getting back from a call
> to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
> {"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
> "port"=>9092}, "topic"=>"mcommits"}
> Using the MX4J interface, I poll the
> bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> attribute values:
> {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
> At the time both of these values were snapshotted, this consumer was close
> to the end of the log file. In that case, I would expect both offsets to be
> fairly similar, however the consumer offset is >> the producer log offset,
> which doesn't make sense.
> Clearly there is something I'm not understanding. How do I use the JMX
> attributes to calculate how far behind the consumer is from the end of the
> log file? In this scenario the consumer offset is >> both the CurrentOffset
> value and the Size value. Is there a way of interpreting these values that
> I'm not seeing?
> Thanks,
> Mike
> --
>   Mike Heffner <[EMAIL PROTECTED]>
>   Librato, Inc.