Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Where is broker 'current offset' stored in ZooKeeper?


+
Chris Curtin 2013-02-18, 15:55
Copy link to this message
-
Re: Where is broker 'current offset' stored in ZooKeeper?
Jun Rao 2013-02-18, 17:04
All zk paths in 0.8 are documented in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

In 0.8, there is a jmx bean (ConsumerLag) in the consumer under
kafka.server that monitors the lag of each partition in terms of messages.
We also have a command line tool ConsumerOffsetChecker.

Thanks,

Jun

On Mon, Feb 18, 2013 at 7:54 AM, Chris Curtin <[EMAIL PROTECTED]>wrote:

> Hi,
>
> A few items our operations teams monitor today in our
> JMS infrastructure are 'number of messages processed' and 'number of
> messages in queue'.
>
> Since Kafka changes the paradigm away from 'messages in queue' how do we
> get operations an idea if our consumers are running behind?
>
> One thought is to query the zookeeper storage and get the current offset
> for each topic/partition/consumer group and compare it to the latest offset
> created by the broker. That will tell us if one or more consumer groups and
> specific partition consumers are not keeping up.
>
> Only problem, I can't figure out where the broker is storing the high water
> mark. I looked at the 0.8 Zookeeper document on the wiki and didn't see it
> there (which is where I see the consumer group information).
>
> First question, is the watermark stored in zookeeper?
>
> Second, is there a different way of monitoring how consumers are doing
> relative to what has been stored by the broker?
>
> Thanks,
>
> Chris
>

 
+
Chris Curtin 2013-02-18, 17:57