Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> API to to query messages amount under one topic


Copy link to this message
-
Re: API to to query messages amount under one topic
Yes, that's probably what you are looking for. It tells you the # of
unconsumed messages per partition, for a particular consumer.

Thanks,

Jun
On Fri, May 17, 2013 at 4:52 PM, Rob Withers <[EMAIL PROTECTED]> wrote:

> Found out about the following:
>
> JMX -> “kafka.server” -> (various FeatcherThread)-ConsumerLag
>
> this sounds like the latency, in time.  is it so?
>
> thanks,
> rob
>
> On May 17, 2013, at 11:49 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > Could you add some JMX stats for us, then?
> >
> > - Queue length, by group offset vs lastOffset
> > - latency between produce and consume, again by group
> >
> > Thanks,
> >
> >
> > Rob Withers
> > Staff Analyst/Developer
> > o: (720) 514-8963
> > c:  (571) 262-1873
> >
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, May 17, 2013 8:52 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: API to to query messages amount under one topic
> >
> > For monitoring, we have jmxs on the broker for both the message and the
> byte rate.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 16, 2013 at 10:04 PM, Rob Withers <[EMAIL PROTECTED]>
> wrote:
> >
> >> Immediately monitoring.  Later, possible thresholding to evoke a
> >> reconfiguration of the number of partitions into a new topic and
> >> migrate message index/logs and redirect pubs/subs to the new topic,
> >> during a traffic spike.
> >>
> >>> -----Original Message-----
> >>> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> >>> Sent: Thursday, May 16, 2013 10:00 PM
> >>> To: [EMAIL PROTECTED]
> >>> Subject: Re: API to to query messages amount under one topic
> >>>
> >>> What do you plan to use this information for?
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> >>> <[EMAIL PROTECTED]>wrote:
> >>>
> >>>> Any possibility for a queuedMessageCount(topic, partitionNumber)
> >>> protocol?
> >>>>
> >>>> thanks,
> >>>> rob
> >>>> ________________________________________
> >>>> From: Jun Rao [[EMAIL PROTECTED]]
> >>>> Sent: Wednesday, May 15, 2013 10:59 PM
> >>>> To: [EMAIL PROTECTED]
> >>>> Subject: Re: API to to query messages amount under one topic
> >>>>
> >>>> In 0.8, you can get the earliest and the latest offset using the
> >>>> getOffsetBefore api. The difference btw the two gives the number
> >>>> of messages on the broker.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Is there any APIs in kafka that I can use to query how many
> >>>>> messages
> >>>> there
> >>>>> are under one topic?
> >>>>> This topic already exist in Kafka server, and producer is
> >>>>> sending
> >>>> messages
> >>>>> to one topic in kafka server
> >>>>>
> >>>>>
> >>>>> -- Regards
> >>>>>   Sining Ma
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB