Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Understanding how to monitor using JMX attributes


+
Mike Heffner 2012-11-20, 03:10
Copy link to this message
-
Re: Understanding how to monitor using JMX attributes
Instead of using jmx, have you tried using ConsumerOffsetChecker to figure
out the consumer lag?

Thanks,

Jun

On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to write a custom monitoring script for our Kafka setup and
> would like some help understanding how to interpret the JMX attributes.
>
> In our setup, the consumers are writing their current offset to a path in
> ZK. This is the value they are getting back from a call
> to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
>
> {"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
> "port"=>9092}, "topic"=>"mcommits"}
>
> Using the MX4J interface, I poll the
> bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> attribute values:
>
> {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
>
> At the time both of these values were snapshotted, this consumer was close
> to the end of the log file. In that case, I would expect both offsets to be
> fairly similar, however the consumer offset is >> the producer log offset,
> which doesn't make sense.
>
> Clearly there is something I'm not understanding. How do I use the JMX
> attributes to calculate how far behind the consumer is from the end of the
> log file? In this scenario the consumer offset is >> both the CurrentOffset
> value and the Size value. Is there a way of interpreting these values that
> I'm not seeing?
>
>
> Thanks,
>
> Mike
>
> --
>
>   Mike Heffner <[EMAIL PROTECTED]>
>   Librato, Inc.
>
+
Mike Heffner 2012-11-20, 16:03
+
Evan Chan 2012-11-20, 17:11
+
Mike Heffner 2012-11-20, 21:07
+
Jun Rao 2012-11-20, 17:12
+
Mike Heffner 2012-11-20, 21:12
+
Jun Rao 2012-11-21, 05:08
+
Mike Heffner 2012-11-21, 15:57
+
Joel Koshy 2012-11-21, 18:47
+
Mike Heffner 2012-11-21, 22:01
+
Joel Koshy 2012-11-21, 22:16