Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2


Copy link to this message
-
Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Hi Xuyen,

SPM for Kafka gets all this stuff already.  I didn't look into whether/how
exactly one can spot consumer lag, but if you spot a way to do it, please
share.

There is a demo of SPM for Kafka if you go to
https://apps.sematext.com/demoso you can see all Kafka performance
graphs and see if any of the metrics
there show consumer lag directly or indirectly.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jan 27, 2014 at 12:49 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Oh, I should note that the built in consumer offsetchecker tool works for
> me but I was hoping to use something like jmxtrans so that I could easily
> export the data to ganglia, graphite or some other graphing tool. Jmxtrans
> was recommended from the Kafka wiki and there's another project called
> kafka-ganglia but it doesn't support 0.7.2 Kafka which is what we're using
> right now. Any suggestions to get real time reporting of consumer lag
> through a tool that can export to ganglia would be appreciated.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Xuyen On [mailto:[EMAIL PROTECTED]]
> Sent: Monday, January 27, 2014 9:33 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> Thanks for your response Joel.
>
> I am currently trying out JMXTrans to get the stats from MBean and I can
> read attributes fine but it doesn't support JMX Operations yet. What tool
> do you use for your reporting? Is there another tool that supports JMX
> operations so that I can use the getOffsetLag operation?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 24, 2014 5:48 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> > kafka.logs.eventdata-0
> >   Attributes
> >       Name                                          // Name of partition
> >       Size                                               //Is this the
> current number of messages?
> Size -> in bytes
>
> >       NumberofSegments             // Don't know what this is
> Each partition contains multiple segment (files) on disk.
>
> >       CurrentOffset                           //Is this the current
> offset of a consumer?
>
> No - it is the (byte) offset up to which we have flushed. It is unrelated
> to the current consumption point of consumers.
>
> >       NumAppendedMessages   // Don't know what this is
> Simple counter of number of messages sent so far to this partition.
>
> > Can someone please tell me what these attributes mean? I am trying to
> find out if I can use these values to calculate the consumer lag ie: Size -
> CurrentOffset and report it to a chart or dashboard somewhere for real-time
> analysis.
>
> So the above would be insufficient to measure consumer lag. You can use
> the consumer offset checker tool and extract the lag from its output. There
> is also an mbean operation on the consumers
> (getOffsetLag) that you can use on a per-partition basis.
>
> Thanks,
>
> Joel
>
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB