Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2


Copy link to this message
-
Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Otis Gospodnetic 2014-01-27, 17:58
Hi Xuyen,

SPM for Kafka gets all this stuff already.  I didn't look into whether/how
exactly one can spot consumer lag, but if you spot a way to do it, please
share.

There is a demo of SPM for Kafka if you go to
https://apps.sematext.com/demoso you can see all Kafka performance
graphs and see if any of the metrics
there show consumer lag directly or indirectly.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jan 27, 2014 at 12:49 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Oh, I should note that the built in consumer offsetchecker tool works for
> me but I was hoping to use something like jmxtrans so that I could easily
> export the data to ganglia, graphite or some other graphing tool. Jmxtrans
> was recommended from the Kafka wiki and there's another project called
> kafka-ganglia but it doesn't support 0.7.2 Kafka which is what we're using
> right now. Any suggestions to get real time reporting of consumer lag
> through a tool that can export to ganglia would be appreciated.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Xuyen On [mailto:[EMAIL PROTECTED]]
> Sent: Monday, January 27, 2014 9:33 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> Thanks for your response Joel.
>
> I am currently trying out JMXTrans to get the stats from MBean and I can
> read attributes fine but it doesn't support JMX Operations yet. What tool
> do you use for your reporting? Is there another tool that supports JMX
> operations so that I can use the getOffsetLag operation?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 24, 2014 5:48 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> > kafka.logs.eventdata-0
> >   Attributes
> >       Name                                          // Name of partition
> >       Size                                               //Is this the
> current number of messages?
> Size -> in bytes
>
> >       NumberofSegments             // Don't know what this is
> Each partition contains multiple segment (files) on disk.
>
> >       CurrentOffset                           //Is this the current
> offset of a consumer?
>
> No - it is the (byte) offset up to which we have flushed. It is unrelated
> to the current consumption point of consumers.
>
> >       NumAppendedMessages   // Don't know what this is
> Simple counter of number of messages sent so far to this partition.
>
> > Can someone please tell me what these attributes mean? I am trying to
> find out if I can use these values to calculate the consumer lag ie: Size -
> CurrentOffset and report it to a chart or dashboard somewhere for real-time
> analysis.
>
> So the above would be insufficient to measure consumer lag. You can use
> the consumer offset checker tool and extract the lag from its output. There
> is also an mbean operation on the consumers
> (getOffsetLag) that you can use on a per-partition basis.
>
> Thanks,
>
> Joel
>
>
>