Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Understanding how to monitor using JMX attributes


Copy link to this message
-
Re: Understanding how to monitor using JMX attributes
Jun Rao 2012-11-20, 05:34
Instead of using jmx, have you tried using ConsumerOffsetChecker to figure
out the consumer lag?

Thanks,

Jun

On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to write a custom monitoring script for our Kafka setup and
> would like some help understanding how to interpret the JMX attributes.
>
> In our setup, the consumers are writing their current offset to a path in
> ZK. This is the value they are getting back from a call
> to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
>
> {"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
> "port"=>9092}, "topic"=>"mcommits"}
>
> Using the MX4J interface, I poll the
> bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> attribute values:
>
> {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
>
> At the time both of these values were snapshotted, this consumer was close
> to the end of the log file. In that case, I would expect both offsets to be
> fairly similar, however the consumer offset is >> the producer log offset,
> which doesn't make sense.
>
> Clearly there is something I'm not understanding. How do I use the JMX
> attributes to calculate how far behind the consumer is from the end of the
> log file? In this scenario the consumer offset is >> both the CurrentOffset
> value and the Size value. Is there a way of interpreting these values that
> I'm not seeing?
>
>
> Thanks,
>
> Mike
>
> --
>
>   Mike Heffner <[EMAIL PROTECTED]>
>   Librato, Inc.
>