at Quantifind, we are big users of Kafka and we like it a lot! In a few use cases, we had to figure out if a queue was growing and how its consumers were behaving. There are a few command-line tools to try to figure out what's going on, but it's not always easy to debug and to see what has happened while everyone was sleeping.
To be able to monitor our kafka queues and consumers, we thus developed a tiny web app that could tell us the log size of each topic in the brokers and the offsets of each consumers. That's very similar to what the kafka ConsumerOffsetChecker tool* is doing, but instead of having a one off snapshot, our app keeps an history and displays a nice graph of what's going on.
Hi - am I right in that for this tool to be effective, consumers must be using the high level consumer or otherwise keeping their offsets in zookeeper? Is there any way to track performance without that? On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <[EMAIL PROTECTED]> wrote:
On Sat, Mar 8, 2014 at 12:17 AM, Dan Hoffman <[EMAIL PROTECTED]> wrote:
yes indeed, this will look into zookeeper for the consumers and their offsets.
I have to say that we haven't been using any other type of consumers, so I haven't explored how to track performance there. The code is pretty simple and could be made modular to grab offsets and log sizes form other sources I guess.
I don't know how kafka-spout works. Right now, the monitor uses the value defined in kafka.utils.ZkUtils.ConsumersPath to grab the list of consumers in ZK. Again, that could be a parameter to the app somewhere to be more flexible and deal with separate zk path.