Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Measuring remaining data


+
Jonathan Davey 2014-01-02, 21:15
+
Guozhang Wang 2014-01-02, 21:27
+
Jonathan Davey 2014-01-02, 22:27
Copy link to this message
-
Re: Measuring remaining data
Since 0.8 we have migrated from physical offset to logical offset, and that
is why we can no longer trace bytes in size. KAFKA-1197 is filed for part
of this requirement, does it satisfy what you need?

As for KAFKA-656, it is not abandoned but we are still trying to figure out
the correct way of doing so.

Guozhang
On Thu, Jan 2, 2014 at 2:26 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:

> That's certainly a step in the right direction. The thing is, I'd
> ideally like to use bytes rather than just messages and that data no
> longer seems to be available in Kafka 0.8. KAFKA-656 might give me
> what I need but that seems abandoned.
>
> On 2 January 2014 21:26, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> > Hi Jonathan,
> >
> > Have you checked out the ConsumerOffsetChecker tool?
> >
> > Guozhang
> >
> >
> > On Thu, Jan 2, 2014 at 1:15 PM, Jonathan Davey <[EMAIL PROTECTED]> wrote:
> >
> >> I need to provide a report that shows the total amount of unread data
> >> available to a consumer group for a particular set of topics.
> >>
> >> Right now I do this by adding custom offset tracking logic to the
> >> producers and consumers. This works but I think I can do better with
> >> one of the following ideas:
> >>
> >> * Running something alongside each broker that grabs the data from the
> >> log dir and zookeeper
> >> * Extending Kafka to do the same as above but expose it as a request
> >> through the protocol
> >>
> >> If I wanted to do these things, where would be a good place to start
> >> looking?
> >>
> >
> >
> >
> > --
> > -- Guozhang
>

--
-- Guozhang