Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Understanding how to monitor using JMX attributes


Copy link to this message
-
Re: Understanding how to monitor using JMX attributes
Hi Mike,

What Kafka version are you using? I tried the latest trunk (0.7) and the
attributes I see are:
Broker/partition,fetch offset,consumer offset

The values seem to be correct - and if you want to get the latest available
offset/consumer lag there are mbean operations (not attributes) that also
seem to work correctly.
On Wed, Nov 21, 2012 at 7:57 AM, Mike Heffner <[EMAIL PROTECTED]> wrote:

> Well, I've currently "solved" the monitoring problem on our end by skipping
> the JMX attributes and using the 'kafka-rb' Ruby gem to read the latest
> offset for the topic. This is the comparison of offsets read from the
> consumer, JMX, and the Ruby gem:
>
> Storm Consumer offset:
> ---------------------------------
> "offset"=>2847272176, "partition"=>2, "broker"=>{"host"=>"10.120.x.x",
> "port"=>9092}, "topic"=>"mcommits"}
>
> JMX Attributes:
> ---------------------
> attrs: {"CurrentOffset"=>162919578, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>7285958, "NumberOfSegments"=>3, "Size"=>1236661577}
>
> OFFSETS command sent from Ruby gem:
> ------------------------------------------------------------
> 10.120.x.x:mcommits:2: latest offset: 2847274728
>
> This is the output of the log file directory for this topic:partition:
>
> -rw-r--r-- 1 root root 536871010 2012-11-16 05:36
> 00000000001610613151.kafka
> -rw-r--r-- 1 root root 536870989 2012-11-20 14:28
> 00000000002147484161.kafka
> -rw-r--r-- 1 root root 162919578 2012-11-21 15:48
> 00000000002684355150.kafka
>
> So, the "latest offset" from the Ruby gem matches what I would expect to
> see -- only slightly ahead of the active consumer. AFAICT, the values from
> JMX aren't usable to monitor this. Should I file a bug or feature request
> to publish an offset in the JMX attributes that matches the 'latest offset'
> read from the Kakfa server?
>
> Cheers,
>
> Mike
>
>
> On Wed, Nov 21, 2012 at 12:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > The attribute getCurrentOffset gives the log end offset. It's not
> > necessarily the log size though since older segments could be deleted.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 20, 2012 at 1:12 PM, Mike Heffner <[EMAIL PROTECTED]> wrote:
> >
> > > Jun,
> > >
> > > Do you have any idea on what the JMX attribute values on the beans "
> > > kafka:type=kafka.logs.{topic name}-{partition idx}" represent then? It
> > > seems like these should correctly represent the current offsets of the
> > > producer logs? They appeared to track correctly for a while, but once
> the
> > > log size grew, they seemed to no longer be correct. Is there
> potentially
> > a
> > > bug in these values are large log sizes?
> > >
> > > I can try the other interface, but it would be nice to know what's
> wrong
> > > with the current JMX values.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > >
> > > On Tue, Nov 20, 2012 at 12:12 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > The tool gets the end offset of the log using getOffsetBefore and the
> > > > consumer offset from ZK. It then calculates the lag.
> > > >
> > > > We do have a JMX for lag in ZookeeperConsumerConnector. The api is
> the
> > > > following, but you need to provide topic/brokerid/partitionid.
> > > >
> > > > /**
> > > >  *  JMX interface for monitoring consumer
> > > >  */
> > > > trait ZookeeperConsumerConnectorMBean {
> > > >   def getPartOwnerStats: String
> > > >   def getConsumerGroup: String
> > > >   def getOffsetLag(topic: String, brokerId: Int, partitionId: Int):
> > Long
> > > >   def getConsumedOffset(topic: String, brokerId: Int, partitionId:
> > Int):
> > > > Long
> > > >   def getLatestOffset(topic: String, brokerId: Int, partitionId:
> Int):
> > > Long
> > > > }
> > > >
> > > > Thanks
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 20, 2012 at 8:03 AM, Mike Heffner <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > I have not tried that yet, I was hoping to use an existing Ruby
> > > > monitoring
> > > > > process that we use to monitor several other existing resources.  I
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB