Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Offset in high level consumer


Copy link to this message
-
Re: Offset in high level consumer
The other option is jmx bean that exposes the lag. Also Kafka provides at
least once guarantees so even if your consumer lags occasionally, you will
eventually receive all mesaages. You need to provision enough consumers so
that they don't fall behind.

Thanks,
Neha
On May 23, 2013 5:30 AM, "arathi maddula" <[EMAIL PROTECTED]> wrote:

> Hi Neha,
> Thanks for the quick reply. Could you tell me if there is some way of
> determining the offset for a consumer from a high level Java consumer class
> apart from ConsumerOffsetChecker tool? This tool can be run only from  the
> command line. Is it possible to use this in a Java class?
>
> I write streaming data from a website into Kafka topic and then read the
> same data using a servlet and serve it to a Java client. The problem is
> that the servlet sometimes does not consume data from the topic as quickly
> as it is produced. In order to give warning to the user saying data is
> lost, I will need some way of determining the consumer group offset from
> the servlet.
>
> The servlet is a high level Kafka consumer and it is sufficient for our
> needs, so I don't want to use a simple consumer.
>
> Any pointers will be of great help.
> Thanks
> Arathi
> On Wed, May 22, 2013 at 7:24 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > You can run the ConsumerOffsetChecker tool that ships with Kafka.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, May 22, 2013 at 2:02 PM, arathi maddula <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > Could you tell me how to find the offset in a high level  Java
> consumer ?
> > >
> > > Thanks
> > > Arathi
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB