Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Offset commit api


Copy link to this message
-
Re: Offset commit api
Good point about compressed message sets. I think that that works and is
simpler. We might still need the txn id to be able to do the application to
the hashmap atomically, but this depends on a number of details that aren't
really spec'd out, in particular how the replicas keep their hashmap fed
and how we fail over when mastership changes.

-Jay
On Tue, Dec 18, 2012 at 8:05 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Thanks for the proposal. Added a couple of comments to the wiki.
>
> Thanks,
>
> Jun
>
> On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Hey Guys,
> >
> > David has made a bunch of progress on the offset commit api
> implementation.
> >
> > Since this is a public API it would be good to do as much thinking
> up-front
> > as possible to minimize future iterations.
> >
> > It would be great if folks could do the following:
> > 1. Read the wiki here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> > 2. Check out the code David wrote here:
> > https://issues.apache.org/jira/browse/KAFKA-657
> >
> > In particular our hope is that this API can act as the first step in
> > scaling the way we store offsets (ZK is not really very appropriate for
> > this). This of course requires having some plan in mind for offset
> storage.
> > I have written (and then after getting some initial feedback, rewritten)
> a
> > section in the above wiki on how this might work.
> >
> > If no one says anything I will be taking a slightly modified patch that
> > adds this functionality on trunk as soon as David gets in a few minor
> > tweaks.
> >
> > -Jay
> >
>