Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Offset commit api


Copy link to this message
-
Re: Offset commit api
Good point about compressed message sets. I think that that works and is
simpler. We might still need the txn id to be able to do the application to
the hashmap atomically, but this depends on a number of details that aren't
really spec'd out, in particular how the replicas keep their hashmap fed
and how we fail over when mastership changes.

-Jay
On Tue, Dec 18, 2012 at 8:05 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Thanks for the proposal. Added a couple of comments to the wiki.
>
> Thanks,
>
> Jun
>
> On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Hey Guys,
> >
> > David has made a bunch of progress on the offset commit api
> implementation.
> >
> > Since this is a public API it would be good to do as much thinking
> up-front
> > as possible to minimize future iterations.
> >
> > It would be great if folks could do the following:
> > 1. Read the wiki here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> > 2. Check out the code David wrote here:
> > https://issues.apache.org/jira/browse/KAFKA-657
> >
> > In particular our hope is that this API can act as the first step in
> > scaling the way we store offsets (ZK is not really very appropriate for
> > this). This of course requires having some plan in mind for offset
> storage.
> > I have written (and then after getting some initial feedback, rewritten)
> a
> > section in the above wiki on how this might work.
> >
> > If no one says anything I will be taking a slightly modified patch that
> > adds this functionality on trunk as soon as David gets in a few minor
> > tweaks.
> >
> > -Jay
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB