A few items our operations teams monitor today in our
JMS infrastructure are 'number of messages processed' and 'number of
messages in queue'.
Since Kafka changes the paradigm away from 'messages in queue' how do we
get operations an idea if our consumers are running behind?
One thought is to query the zookeeper storage and get the current offset
for each topic/partition/consumer group and compare it to the latest offset
created by the broker. That will tell us if one or more consumer groups and
specific partition consumers are not keeping up.
Only problem, I can't figure out where the broker is storing the high water
mark. I looked at the 0.8 Zookeeper document on the wiki and didn't see it
there (which is where I see the consumer group information).
First question, is the watermark stored in zookeeper?
Second, is there a different way of monitoring how consumers are doing
relative to what has been stored by the broker?