Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Any way to know from kafka's JMX interface whether there are unread messages on a topic


+
Joseph Boyd 2012-08-01, 17:10
+
Jun Rao 2012-08-01, 17:19
+
Joseph Boyd 2012-08-01, 17:51
+
Jun Rao 2012-08-02, 05:22
+
Joseph Boyd 2012-08-02, 17:20
Copy link to this message
-
Re: Any way to know from kafka's JMX interface whether there are unread messages on a topic
Along these lines is there a list of JMX beans and what values are exposed w/some description of the metric they communicate?
I am trying to understand how the system is doing and don't know what I don't know yet :)

On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:

> Thanks!  I was (wrongly) looking for this information on the kafka
> process, and not on the consumer process.  Once I looked there I found
> kafka.ConsumerStats.
>
> (still not sure how I read dates wrong for the JIRA, but that happens)
>
>
> ...joe
>
>
> On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> Joe,
>>
>> It should be there. 0.7.1 was released last month and that jira was closed
>> last year.
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <[EMAIL PROTECTED]
>>> wrote:
>>
>>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>>> seeing it.
>>>
>>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>>> shortly after the 0.7.1 release was rolled up.
>>>
>>>
>>> Thanks for the pointer, though.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>>
>>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>> Joe,
>>>>
>>>> We have the following bean name and type registered for consumers. There
>>> is
>>>> also a CheckOffsetLag tool.
>>>>
>>>> kafka:type=kafka.ConsumerStats
>>>> /**
>>>> *  JMX interface for monitoring consumer
>>>> */
>>>> trait ZookeeperConsumerConnectorMBean {
>>>>  def getPartOwnerStats: String
>>>>  def getConsumerGroup: String
>>>>  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>>>>  def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>>>> Long
>>>>  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>>> Long
>>>> }
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>>> [EMAIL PROTECTED]
>>>>> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there any way to know from kafka's JMX interface whether there are
>>>>> unread messages on a topic?  I think the answer is 'no', based on my
>>>>> read of documentation (esp. the 'high water mark'), and my own
>>>>> experiments with producing test messages, looking at the stats, and
>>>>> then consuming messages (both with/without committing messages
>>>>> consumed).
>>>>>
>>>>> I thought I'd double check on the users list anyhow, though, in case
>>>>> I'm missing something.  The KAFKA-260 work looks useful in this area
>>>>> too.
>>>>>
>>>>> Thanks for any help.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>
+
Jun Rao 2012-08-07, 01:45