Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - API to to query messages amount under one topic


Copy link to this message
-
Re: API to to query messages amount under one topic
Rob Withers 2013-05-17, 23:52
Found out about the following:

JMX -> “kafka.server” -> (various FeatcherThread)-ConsumerLag

this sounds like the latency, in time.  is it so?

thanks,
rob

On May 17, 2013, at 11:49 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:

> Could you add some JMX stats for us, then?  
>
> - Queue length, by group offset vs lastOffset
> - latency between produce and consume, again by group
>
> Thanks,
>
>
> Rob Withers
> Staff Analyst/Developer
> o: (720) 514-8963  
> c:  (571) 262-1873
>
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 8:52 AM
> To: [EMAIL PROTECTED]
> Subject: Re: API to to query messages amount under one topic
>
> For monitoring, we have jmxs on the broker for both the message and the byte rate.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 16, 2013 at 10:04 PM, Rob Withers <[EMAIL PROTECTED]> wrote:
>
>> Immediately monitoring.  Later, possible thresholding to evoke a
>> reconfiguration of the number of partitions into a new topic and
>> migrate message index/logs and redirect pubs/subs to the new topic,
>> during a traffic spike.
>>
>>> -----Original Message-----
>>> From: Jun Rao [mailto:[EMAIL PROTECTED]]
>>> Sent: Thursday, May 16, 2013 10:00 PM
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: API to to query messages amount under one topic
>>>
>>> What do you plan to use this information for?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> Any possibility for a queuedMessageCount(topic, partitionNumber)
>>> protocol?
>>>>
>>>> thanks,
>>>> rob
>>>> ________________________________________
>>>> From: Jun Rao [[EMAIL PROTECTED]]
>>>> Sent: Wednesday, May 15, 2013 10:59 PM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: Re: API to to query messages amount under one topic
>>>>
>>>> In 0.8, you can get the earliest and the latest offset using the
>>>> getOffsetBefore api. The difference btw the two gives the number
>>>> of messages on the broker.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi,
>>>>> Is there any APIs in kafka that I can use to query how many
>>>>> messages
>>>> there
>>>>> are under one topic?
>>>>> This topic already exist in Kafka server, and producer is
>>>>> sending
>>>> messages
>>>>> to one topic in kafka server
>>>>>
>>>>>
>>>>> -- Regards
>>>>>   Sining Ma
>>>>>
>>>>>
>>>>>
>>>>
>>
>>