Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Any way to know from kafka's JMX interface whether there are unread messages on a topic


Copy link to this message
-
Re: Any way to know from kafka's JMX interface whether there are unread messages on a topic
Along these lines is there a list of JMX beans and what values are exposed w/some description of the metric they communicate?
I am trying to understand how the system is doing and don't know what I don't know yet :)

On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:

> Thanks!  I was (wrongly) looking for this information on the kafka
> process, and not on the consumer process.  Once I looked there I found
> kafka.ConsumerStats.
>
> (still not sure how I read dates wrong for the JIRA, but that happens)
>
>
> ...joe
>
>
> On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> Joe,
>>
>> It should be there. 0.7.1 was released last month and that jira was closed
>> last year.
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <[EMAIL PROTECTED]
>>> wrote:
>>
>>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>>> seeing it.
>>>
>>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>>> shortly after the 0.7.1 release was rolled up.
>>>
>>>
>>> Thanks for the pointer, though.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>>
>>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>> Joe,
>>>>
>>>> We have the following bean name and type registered for consumers. There
>>> is
>>>> also a CheckOffsetLag tool.
>>>>
>>>> kafka:type=kafka.ConsumerStats
>>>> /**
>>>> *  JMX interface for monitoring consumer
>>>> */
>>>> trait ZookeeperConsumerConnectorMBean {
>>>>  def getPartOwnerStats: String
>>>>  def getConsumerGroup: String
>>>>  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>>>>  def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>>>> Long
>>>>  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>>> Long
>>>> }
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>>> [EMAIL PROTECTED]
>>>>> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there any way to know from kafka's JMX interface whether there are
>>>>> unread messages on a topic?  I think the answer is 'no', based on my
>>>>> read of documentation (esp. the 'high water mark'), and my own
>>>>> experiments with producing test messages, looking at the stats, and
>>>>> then consuming messages (both with/without committing messages
>>>>> consumed).
>>>>>
>>>>> I thought I'd double check on the users list anyhow, though, in case
>>>>> I'm missing something.  The KAFKA-260 work looks useful in this area
>>>>> too.
>>>>>
>>>>> Thanks for any help.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB