Hi there,

Is there any way to know from kafka's JMX interface whether there are
unread messages on a topic?  I think the answer is 'no', based on my
read of documentation (esp. the 'high water mark'), and my own
experiments with producing test messages, looking at the stats, and
then consuming messages (both with/without committing messages
consumed).

I thought I'd double check on the users list anyhow, though, in case
I'm missing something.  The KAFKA-260 work looks useful in this area
too.

Thanks for any help.
...joe

Joe,

We have the following bean name and type registered for consumers. There is
also a CheckOffsetLag tool.

kafka:type=kafka.ConsumerStats
/**
 *  JMX interface for monitoring consumer
 */
trait ZookeeperConsumerConnectorMBean {
  def getPartOwnerStats: String
  def getConsumerGroup: String
  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
  def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
Long
  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int): Long
}

Thanks,

Jun

On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <[EMAIL PROTECTED]
> wrote:

> Hi there,
>
> Is there any way to know from kafka's JMX interface whether there are
> unread messages on a topic?  I think the answer is 'no', based on my
> read of documentation (esp. the 'high water mark'), and my own
> experiments with producing test messages, looking at the stats, and
> then consuming messages (both with/without committing messages
> consumed).
>
> I thought I'd double check on the users list anyhow, though, in case
> I'm missing something.  The KAFKA-260 work looks useful in this area
> too.
>
> Thanks for any help.
>
>
> ...joe
>

It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
seeing it.

My read of JIRA / mailing list archives shows KAFKA-136 went in
shortly after the 0.7.1 release was rolled up.
Thanks for the pointer, though.
...joe
On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Joe,
>
> We have the following bean name and type registered for consumers. There is
> also a CheckOffsetLag tool.
>
> kafka:type=kafka.ConsumerStats
> /**
>  *  JMX interface for monitoring consumer
>  */
> trait ZookeeperConsumerConnectorMBean {
>   def getPartOwnerStats: String
>   def getConsumerGroup: String
>   def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>   def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
> Long
>   def getLatestOffset(topic: String, brokerId: Int, partitionId: Int): Long
> }
>
> Thanks,
>
> Jun
>
> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <[EMAIL PROTECTED]
>> wrote:
>
>> Hi there,
>>
>> Is there any way to know from kafka's JMX interface whether there are
>> unread messages on a topic?  I think the answer is 'no', based on my
>> read of documentation (esp. the 'high water mark'), and my own
>> experiments with producing test messages, looking at the stats, and
>> then consuming messages (both with/without committing messages
>> consumed).
>>
>> I thought I'd double check on the users list anyhow, though, in case
>> I'm missing something.  The KAFKA-260 work looks useful in this area
>> too.
>>
>> Thanks for any help.
>>
>>
>> ...joe
>>

Joe,

It should be there. 0.7.1 was released last month and that jira was closed
last year.

Thanks,

Jun

On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <[EMAIL PROTECTED]
> wrote:

> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
> seeing it.
>
> My read of JIRA / mailing list archives shows KAFKA-136 went in
> shortly after the 0.7.1 release was rolled up.
>
>
> Thanks for the pointer, though.
>
>
> ...joe
>
>
>
>
> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > Joe,
> >
> > We have the following bean name and type registered for consumers. There
> is
> > also a CheckOffsetLag tool.
> >
> > kafka:type=kafka.ConsumerStats
> > /**
> >  *  JMX interface for monitoring consumer
> >  */
> > trait ZookeeperConsumerConnectorMBean {
> >   def getPartOwnerStats: String
> >   def getConsumerGroup: String
> >   def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
> >   def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
> > Long
> >   def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
> Long
> > }
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
> [EMAIL PROTECTED]
> >> wrote:
> >
> >> Hi there,
> >>
> >> Is there any way to know from kafka's JMX interface whether there are
> >> unread messages on a topic?  I think the answer is 'no', based on my
> >> read of documentation (esp. the 'high water mark'), and my own
> >> experiments with producing test messages, looking at the stats, and
> >> then consuming messages (both with/without committing messages
> >> consumed).
> >>
> >> I thought I'd double check on the users list anyhow, though, in case
> >> I'm missing something.  The KAFKA-260 work looks useful in this area
> >> too.
> >>
> >> Thanks for any help.
> >>
> >>
> >> ...joe
> >>
>

Thanks!  I was (wrongly) looking for this information on the kafka
process, and not on the consumer process.  Once I looked there I found
kafka.ConsumerStats.

(still not sure how I read dates wrong for the JIRA, but that happens)
...joe
On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Joe,
>
> It should be there. 0.7.1 was released last month and that jira was closed
> last year.
>
> Thanks,
>
> Jun
>
> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <[EMAIL PROTECTED]
>> wrote:
>
>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>> seeing it.
>>
>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>> shortly after the 0.7.1 release was rolled up.
>>
>>
>> Thanks for the pointer, though.
>>
>>
>> ...joe
>>
>>
>>
>>
>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > Joe,
>> >
>> > We have the following bean name and type registered for consumers. There
>> is
>> > also a CheckOffsetLag tool.
>> >
>> > kafka:type=kafka.ConsumerStats
>> > /**
>> >  *  JMX interface for monitoring consumer
>> >  */
>> > trait ZookeeperConsumerConnectorMBean {
>> >   def getPartOwnerStats: String
>> >   def getConsumerGroup: String
>> >   def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>> >   def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>> > Long
>> >   def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>> Long
>> > }
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>> [EMAIL PROTECTED]
>> >> wrote:
>> >
>> >> Hi there,
>> >>
>> >> Is there any way to know from kafka's JMX interface whether there are
>> >> unread messages on a topic?  I think the answer is 'no', based on my
>> >> read of documentation (esp. the 'high water mark'), and my own
>> >> experiments with producing test messages, looking at the stats, and
>> >> then consuming messages (both with/without committing messages
>> >> consumed).
>> >>
>> >> I thought I'd double check on the users list anyhow, though, in case
>> >> I'm missing something.  The KAFKA-260 work looks useful in this area
>> >> too.
>> >>
>> >> Thanks for any help.
>> >>
>> >>
>> >> ...joe
>> >>
>>

Along these lines is there a list of JMX beans and what values are exposed w/some description of the metric they communicate?
I am trying to understand how the system is doing and don't know what I don't know yet :)

On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:

> Thanks!  I was (wrongly) looking for this information on the kafka
> process, and not on the consumer process.  Once I looked there I found
> kafka.ConsumerStats.
>
> (still not sure how I read dates wrong for the JIRA, but that happens)
>
>
> ...joe
>
>
> On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> Joe,
>>
>> It should be there. 0.7.1 was released last month and that jira was closed
>> last year.
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <[EMAIL PROTECTED]
>>> wrote:
>>
>>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>>> seeing it.
>>>
>>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>>> shortly after the 0.7.1 release was rolled up.
>>>
>>>
>>> Thanks for the pointer, though.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>>
>>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>> Joe,
>>>>
>>>> We have the following bean name and type registered for consumers. There
>>> is
>>>> also a CheckOffsetLag tool.
>>>>
>>>> kafka:type=kafka.ConsumerStats
>>>> /**
>>>> *  JMX interface for monitoring consumer
>>>> */
>>>> trait ZookeeperConsumerConnectorMBean {
>>>>  def getPartOwnerStats: String
>>>>  def getConsumerGroup: String
>>>>  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>>>>  def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>>>> Long
>>>>  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>>> Long
>>>> }
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>>> [EMAIL PROTECTED]
>>>>> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there any way to know from kafka's JMX interface whether there are
>>>>> unread messages on a topic?  I think the answer is 'no', based on my
>>>>> read of documentation (esp. the 'high water mark'), and my own
>>>>> experiments with producing test messages, looking at the stats, and
>>>>> then consuming messages (both with/without committing messages
>>>>> consumed).
>>>>>
>>>>> I thought I'd double check on the users list anyhow, though, in case
>>>>> I'm missing something.  The KAFKA-260 work looks useful in this area
>>>>> too.
>>>>>
>>>>> Thanks for any help.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>

Corbin,

The JMX beans and their methods can be found at
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring

Thanks,

Jun

On Sun, Aug 5, 2012 at 8:08 AM, Corbin Hoenes <[EMAIL PROTECTED]> wrote:

> Along these lines is there a list of JMX beans and what values are exposed
> w/some description of the metric they communicate?
> I am trying to understand how the system is doing and don't know what I
> don't know yet :)
>
> On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:
>
> > Thanks!  I was (wrongly) looking for this information on the kafka
> > process, and not on the consumer process.  Once I looked there I found
> > kafka.ConsumerStats.
> >
> > (still not sure how I read dates wrong for the JIRA, but that happens)
> >
> >
> > ...joe
> >
> >
> > On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> Joe,
> >>
> >> It should be there. 0.7.1 was released last month and that jira was
> closed
> >> last year.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <
> [EMAIL PROTECTED]
> >>> wrote:
> >>
> >>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
> >>> seeing it.
> >>>
> >>> My read of JIRA / mailing list archives shows KAFKA-136 went in
> >>> shortly after the 0.7.1 release was rolled up.
> >>>
> >>>
> >>> Thanks for the pointer, though.
> >>>
> >>>
> >>> ...joe
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>>> Joe,
> >>>>
> >>>> We have the following bean name and type registered for consumers.
> There
> >>> is
> >>>> also a CheckOffsetLag tool.
> >>>>
> >>>> kafka:type=kafka.ConsumerStats
> >>>> /**
> >>>> *  JMX interface for monitoring consumer
> >>>> */
> >>>> trait ZookeeperConsumerConnectorMBean {
> >>>>  def getPartOwnerStats: String
> >>>>  def getConsumerGroup: String
> >>>>  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int):
> Long
> >>>>  def getConsumedOffset(topic: String, brokerId: Int, partitionId:
> Int):
> >>>> Long
> >>>>  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
> >>> Long
> >>>> }
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
> >>> [EMAIL PROTECTED]
> >>>>> wrote:
> >>>>
> >>>>> Hi there,
> >>>>>
> >>>>> Is there any way to know from kafka's JMX interface whether there are
> >>>>> unread messages on a topic?  I think the answer is 'no', based on my
> >>>>> read of documentation (esp. the 'high water mark'), and my own
> >>>>> experiments with producing test messages, looking at the stats, and
> >>>>> then consuming messages (both with/without committing messages
> >>>>> consumed).
> >>>>>
> >>>>> I thought I'd double check on the users list anyhow, though, in case
> >>>>> I'm missing something.  The KAFKA-260 work looks useful in this area
> >>>>> too.
> >>>>>
> >>>>> Thanks for any help.
> >>>>>
> >>>>>
> >>>>> ...joe
> >>>>>
> >>>
>
>