Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Measuring remaining data


Copy link to this message
-
Re: Measuring remaining data
Hi Jonathan,

Have you checked out the ConsumerOffsetChecker tool?

Guozhang
On Thu, Jan 2, 2014 at 1:15 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:

> I need to provide a report that shows the total amount of unread data
> available to a consumer group for a particular set of topics.
>
> Right now I do this by adding custom offset tracking logic to the
> producers and consumers. This works but I think I can do better with
> one of the following ideas:
>
> * Running something alongside each broker that grabs the data from the
> log dir and zookeeper
> * Extending Kafka to do the same as above but expose it as a request
> through the protocol
>
> If I wanted to do these things, where would be a good place to start
> looking?
>

--
-- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB