Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> It would be nice to be able to read Kafka topics in reverse.


Copy link to this message
-
Re: It would be nice to be able to read Kafka topics in reverse.
Hi,

We have a similar use case.  In our case it's about performance metrics and
situations where we plan catch up.  Instead of processing data
chronologically, it would make sense to process newest data first, so
performance metrics graphs for "now" have data.

Here is a related thread: http://search-hadoop.com/m/4TaT4HaM6f1

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Feb 5, 2014 at 6:45 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Technically this is possible with the existing server and protocol and
> could be implemented using the "low level" network client. The high-level
> client doesn't really allow this. This would be a good thing to think about
> as we start on the redesign of that client. I don't think it has to be
> terribly convenient because it is not a very common use case, but just
> being able to read chunks and reverse them in memory would allow this, so I
> think the only thing needed is the ability to set your offset easily.
>
> -Jay
>
>
> On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Here is a use case that we would like to see kafka's client support in
> the
> > future. Currently reading a topic is FIFO. It would be awesome to read a
> > topic in LIFO order. Put another way we would like to be able to read a
> > topic in reverse.
> >
> >
> > Why? Basically we have per user streams which we have to keep up to date.
> > When an event comes into the system it is determined to be cheap enough
> to
> > fan out the user stream updates at write time or to expensive and thus
> > deferred until read time. This is where kafka comes into play. We push
> all
> > the events that were to expensive to fan out at write time onto a kafka
> > topic. When a given user logs on we read through this topic and update
> the
> > users stream. If it has been a while since the user has logged in it
> takes
> > a while to catch up. If we could read a topic in reverse order we could
> > catch the most recent events in the stream first. Which in our case is
> what
> > the user cares about the most. Once the 20 most resent items have been
> > added to a user's stream we can return the head of the stream and
> continue
> > the catch up process in the background.
> >
> >
> > Thanks,
> >
> > Jonathan
> >
> >
>