-Re: Measuring remaining data
Guozhang Wang 2014-01-02, 22:43
Since 0.8 we have migrated from physical offset to logical offset, and that
is why we can no longer trace bytes in size. KAFKA-1197 is filed for part
of this requirement, does it satisfy what you need?
As for KAFKA-656, it is not abandoned but we are still trying to figure out
the correct way of doing so.
On Thu, Jan 2, 2014 at 2:26 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:
> That's certainly a step in the right direction. The thing is, I'd
> ideally like to use bytes rather than just messages and that data no
> longer seems to be available in Kafka 0.8. KAFKA-656 might give me
> what I need but that seems abandoned.
> On 2 January 2014 21:26, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> > Hi Jonathan,
> > Have you checked out the ConsumerOffsetChecker tool?
> > Guozhang
> > On Thu, Jan 2, 2014 at 1:15 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:
> >> I need to provide a report that shows the total amount of unread data
> >> available to a consumer group for a particular set of topics.
> >> Right now I do this by adding custom offset tracking logic to the
> >> producers and consumers. This works but I think I can do better with
> >> one of the following ideas:
> >> * Running something alongside each broker that grabs the data from the
> >> log dir and zookeeper
> >> * Extending Kafka to do the same as above but expose it as a request
> >> through the protocol
> >> If I wanted to do these things, where would be a good place to start
> >> looking?
> > --
> > -- Guozhang