Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> It would be nice to be able to read Kafka topics in reverse.


Copy link to this message
-
Re: It would be nice to be able to read Kafka topics in reverse.
Technically this is possible with the existing server and protocol and
could be implemented using the "low level" network client. The high-level
client doesn't really allow this. This would be a good thing to think about
as we start on the redesign of that client. I don't think it has to be
terribly convenient because it is not a very common use case, but just
being able to read chunks and reverse them in memory would allow this, so I
think the only thing needed is the ability to set your offset easily.

-Jay
On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Here is a use case that we would like to see kafka's client support in the
> future. Currently reading a topic is FIFO. It would be awesome to read a
> topic in LIFO order. Put another way we would like to be able to read a
> topic in reverse.
>
>
> Why? Basically we have per user streams which we have to keep up to date.
> When an event comes into the system it is determined to be cheap enough to
> fan out the user stream updates at write time or to expensive and thus
> deferred until read time. This is where kafka comes into play. We push all
> the events that were to expensive to fan out at write time onto a kafka
> topic. When a given user logs on we read through this topic and update the
> users stream. If it has been a while since the user has logged in it takes
> a while to catch up. If we could read a topic in reverse order we could
> catch the most recent events in the stream first. Which in our case is what
> the user cares about the most. Once the 20 most resent items have been
> added to a user's stream we can return the head of the stream and continue
> the catch up process in the background.
>
>
> Thanks,
>
> Jonathan
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB