-Re: It would be nice to be able to read Kafka topics in reverse.
Jay Kreps 2014-02-05, 23:46
Technically this is possible with the existing server and protocol and
could be implemented using the "low level" network client. The high-level
client doesn't really allow this. This would be a good thing to think about
as we start on the redesign of that client. I don't think it has to be
terribly convenient because it is not a very common use case, but just
being able to read chunks and reverse them in memory would allow this, so I
think the only thing needed is the ability to set your offset easily.
On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt <
[EMAIL PROTECTED]> wrote:
> Here is a use case that we would like to see kafka's client support in the
> future. Currently reading a topic is FIFO. It would be awesome to read a
> topic in LIFO order. Put another way we would like to be able to read a
> topic in reverse.
> Why? Basically we have per user streams which we have to keep up to date.
> When an event comes into the system it is determined to be cheap enough to
> fan out the user stream updates at write time or to expensive and thus
> deferred until read time. This is where kafka comes into play. We push all
> the events that were to expensive to fan out at write time onto a kafka
> topic. When a given user logs on we read through this topic and update the
> users stream. If it has been a while since the user has logged in it takes
> a while to catch up. If we could read a topic in reverse order we could
> catch the most recent events in the stream first. Which in our case is what
> the user cares about the most. Once the 20 most resent items have been
> added to a user's stream we can return the head of the stream and continue
> the catch up process in the background.