Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Measuring remaining data


+
Jonathan Davey 2014-01-02, 21:15
+
Guozhang Wang 2014-01-02, 21:27
+
Jonathan Davey 2014-01-02, 22:27
Copy link to this message
-
Re: Measuring remaining data
Since 0.8 we have migrated from physical offset to logical offset, and that
is why we can no longer trace bytes in size. KAFKA-1197 is filed for part
of this requirement, does it satisfy what you need?

As for KAFKA-656, it is not abandoned but we are still trying to figure out
the correct way of doing so.

Guozhang
On Thu, Jan 2, 2014 at 2:26 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:

> That's certainly a step in the right direction. The thing is, I'd
> ideally like to use bytes rather than just messages and that data no
> longer seems to be available in Kafka 0.8. KAFKA-656 might give me
> what I need but that seems abandoned.
>
> On 2 January 2014 21:26, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> > Hi Jonathan,
> >
> > Have you checked out the ConsumerOffsetChecker tool?
> >
> > Guozhang
> >
> >
> > On Thu, Jan 2, 2014 at 1:15 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:
> >
> >> I need to provide a report that shows the total amount of unread data
> >> available to a consumer group for a particular set of topics.
> >>
> >> Right now I do this by adding custom offset tracking logic to the
> >> producers and consumers. This works but I think I can do better with
> >> one of the following ideas:
> >>
> >> * Running something alongside each broker that grabs the data from the
> >> log dir and zookeeper
> >> * Extending Kafka to do the same as above but expose it as a request
> >> through the protocol
> >>
> >> If I wanted to do these things, where would be a good place to start
> >> looking?
> >>
> >
> >
> >
> > --
> > -- Guozhang
>

--
-- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB