A few items our operations teams monitor today in our JMS infrastructure are 'number of messages processed' and 'number of messages in queue'.
Since Kafka changes the paradigm away from 'messages in queue' how do we get operations an idea if our consumers are running behind?
One thought is to query the zookeeper storage and get the current offset for each topic/partition/consumer group and compare it to the latest offset created by the broker. That will tell us if one or more consumer groups and specific partition consumers are not keeping up.
Only problem, I can't figure out where the broker is storing the high water mark. I looked at the 0.8 Zookeeper document on the wiki and didn't see it there (which is where I see the consumer group information).
First question, is the watermark stored in zookeeper?
Second, is there a different way of monitoring how consumers are doing relative to what has been stored by the broker?
Chris Curtin 2013-02-18, 15:55
Re: Where is broker 'current offset' stored in ZooKeeper?
Looks like a couple of ways to do this based on how operations wants to manage things.
Chris On Mon, Feb 18, 2013 at 12:04 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
Chris Curtin 2013-02-18, 17:57
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext