Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Understanding how to monitor using JMX attributes


Copy link to this message
-
Re: Understanding how to monitor using JMX attributes
Joel Koshy 2012-11-21, 18:47
Hi Mike,

What Kafka version are you using? I tried the latest trunk (0.7) and the
attributes I see are:
Broker/partition,fetch offset,consumer offset

The values seem to be correct - and if you want to get the latest available
offset/consumer lag there are mbean operations (not attributes) that also
seem to work correctly.
On Wed, Nov 21, 2012 at 7:57 AM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Well, I've currently "solved" the monitoring problem on our end by skipping
> the JMX attributes and using the 'kafka-rb' Ruby gem to read the latest
> offset for the topic. This is the comparison of offsets read from the
> consumer, JMX, and the Ruby gem:
>
> Storm Consumer offset:
> ---------------------------------
> "offset"=>2847272176, "partition"=>2, "broker"=>{"host"=>"10.120.x.x",
> "port"=>9092}, "topic"=>"mcommits"}
>
> JMX Attributes:
> ---------------------
> attrs: {"CurrentOffset"=>162919578, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>7285958, "NumberOfSegments"=>3, "Size"=>1236661577}
>
> OFFSETS command sent from Ruby gem:
> ------------------------------------------------------------
> 10.120.x.x:mcommits:2: latest offset: 2847274728
>
> This is the output of the log file directory for this topic:partition:
>
> -rw-r--r-- 1 root root 536871010 2012-11-16 05:36
> 00000000001610613151.kafka
> -rw-r--r-- 1 root root 536870989 2012-11-20 14:28
> 00000000002147484161.kafka
> -rw-r--r-- 1 root root 162919578 2012-11-21 15:48
> 00000000002684355150.kafka
>
> So, the "latest offset" from the Ruby gem matches what I would expect to
> see -- only slightly ahead of the active consumer. AFAICT, the values from
> JMX aren't usable to monitor this. Should I file a bug or feature request
> to publish an offset in the JMX attributes that matches the 'latest offset'
> read from the Kakfa server?
>
> Cheers,
>
> Mike
>
>
> On Wed, Nov 21, 2012 at 12:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > The attribute getCurrentOffset gives the log end offset. It's not
> > necessarily the log size though since older segments could be deleted.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 20, 2012 at 1:12 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:
> >
> > > Jun,
> > >
> > > Do you have any idea on what the JMX attribute values on the beans "
> > > kafka:type=kafka.logs.{topic name}-{partition idx}" represent then? It
> > > seems like these should correctly represent the current offsets of the
> > > producer logs? They appeared to track correctly for a while, but once
> the
> > > log size grew, they seemed to no longer be correct. Is there
> potentially
> > a
> > > bug in these values are large log sizes?
> > >
> > > I can try the other interface, but it would be nice to know what's
> wrong
> > > with the current JMX values.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > >
> > > On Tue, Nov 20, 2012 at 12:12 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > The tool gets the end offset of the log using getOffsetBefore and the
> > > > consumer offset from ZK. It then calculates the lag.
> > > >
> > > > We do have a JMX for lag in ZookeeperConsumerConnector. The api is
> the
> > > > following, but you need to provide topic/brokerid/partitionid.
> > > >
> > > > /**
> > > >  *  JMX interface for monitoring consumer
> > > >  */
> > > > trait ZookeeperConsumerConnectorMBean {
> > > >   def getPartOwnerStats: String
> > > >   def getConsumerGroup: String
> > > >   def getOffsetLag(topic: String, brokerId: Int, partitionId: Int):
> > Long
> > > >   def getConsumedOffset(topic: String, brokerId: Int, partitionId:
> > Int):
> > > > Long
> > > >   def getLatestOffset(topic: String, brokerId: Int, partitionId:
> Int):
> > > Long
> > > > }
> > > >
> > > > Thanks
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 20, 2012 at 8:03 AM, Mike Heffner <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > I have not tried that yet, I was hoping to use an existing Ruby
> > > > monitoring
> > > > > process that we use to monitor several other existing resources.  I