Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Is there a way to get the offset of a consumer of a topic?


Copy link to this message
-
Re: Is there a way to get the offset of a consumer of a topic?
David DeMaagd 2013-12-04, 20:50
You can use either the MaxLag MBean (0.8):

http://kafka.apache.org/documentation.html#monitoring

Or the ConsumerOffsetChecker (0.7 or 0.8, can't seem to find a doc
reference for it):

./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker ...

--
Dave DeMaagd | S'aite Reliability Engineering, Y'all
[EMAIL PROTECTED] | 818 262 7958

([EMAIL PROTECTED] - Wed, Dec 04, 2013 at 12:40:59PM -0800)
> Hi,
> I was wondering if there's an easy way to get the current offset of a
> specified consumer?  What we need is something to check the consumer offset
> every 5 min.  If it's changing at a certain rate every time we check, good.
>  If not, then we have a problem.  I was going to write a monitoring program
> to check the broker or consumer for this offset to see if it changed and
> how much it changed since the last time I checked it.
>
> I ran into this pg on the wiki which looks like it's what I need.
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit%2FFetchAPI
>
> However, it says the API calls are not functional yet.  Are there some
> other methods or APIs or JMX bean/object/etc that we can use in the
> meantime?  If I were to ping zookeeper, where do I look for this offset?
>  Thanks in advance for your help.