Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> API to to query messages amount under one topic


Copy link to this message
-
Re: API to to query messages amount under one topic
Yes, that's probably what you are looking for. It tells you the # of
unconsumed messages per partition, for a particular consumer.

Thanks,

Jun
On Fri, May 17, 2013 at 4:52 PM, Rob Withers <[EMAIL PROTECTED]> wrote:

> Found out about the following:
>
> JMX -> “kafka.server” -> (various FeatcherThread)-ConsumerLag
>
> this sounds like the latency, in time.  is it so?
>
> thanks,
> rob
>
> On May 17, 2013, at 11:49 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > Could you add some JMX stats for us, then?
> >
> > - Queue length, by group offset vs lastOffset
> > - latency between produce and consume, again by group
> >
> > Thanks,
> >
> >
> > Rob Withers
> > Staff Analyst/Developer
> > o: (720) 514-8963
> > c:  (571) 262-1873
> >
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, May 17, 2013 8:52 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: API to to query messages amount under one topic
> >
> > For monitoring, we have jmxs on the broker for both the message and the
> byte rate.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 16, 2013 at 10:04 PM, Rob Withers <[EMAIL PROTECTED]>
> wrote:
> >
> >> Immediately monitoring.  Later, possible thresholding to evoke a
> >> reconfiguration of the number of partitions into a new topic and
> >> migrate message index/logs and redirect pubs/subs to the new topic,
> >> during a traffic spike.
> >>
> >>> -----Original Message-----
> >>> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> >>> Sent: Thursday, May 16, 2013 10:00 PM
> >>> To: [EMAIL PROTECTED]
> >>> Subject: Re: API to to query messages amount under one topic
> >>>
> >>> What do you plan to use this information for?
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> >>> <[EMAIL PROTECTED]>wrote:
> >>>
> >>>> Any possibility for a queuedMessageCount(topic, partitionNumber)
> >>> protocol?
> >>>>
> >>>> thanks,
> >>>> rob
> >>>> ________________________________________
> >>>> From: Jun Rao [[EMAIL PROTECTED]]
> >>>> Sent: Wednesday, May 15, 2013 10:59 PM
> >>>> To: [EMAIL PROTECTED]
> >>>> Subject: Re: API to to query messages amount under one topic
> >>>>
> >>>> In 0.8, you can get the earliest and the latest offset using the
> >>>> getOffsetBefore api. The difference btw the two gives the number
> >>>> of messages on the broker.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Is there any APIs in kafka that I can use to query how many
> >>>>> messages
> >>>> there
> >>>>> are under one topic?
> >>>>> This topic already exist in Kafka server, and producer is
> >>>>> sending
> >>>> messages
> >>>>> to one topic in kafka server
> >>>>>
> >>>>>
> >>>>> -- Regards
> >>>>>   Sining Ma
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>