Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Update: RE: are commitOffsets botched to zookeeper?


Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
Have you considered abstracting offset storage away so people could implement their own?
Would you take a patch if I'd stabbed at it, and if yes, what's the process (pardon the n00b)?

KCBO,
--
"If you can't conceal it well, expose it with all your might"
Alex Zuzin
On Friday, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:

> There is no particular need for storing the offsets in zookeeper. In fact
> with Kafka 0.8, since partitions will be highly available, offsets could be
> stored in Kafka topics. However, we haven't ironed out the design for this
> yet.
>
> Thanks,
> Neha
>
>
> On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > afaik you dont 'have' to store the consumed offsets in zk right, this is
> > only automatic with some of the clients?
> >
> > why not store them in a data store that can write at the rate that you
> > require?
> >
> >
> > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > wrote:
> >
> >
> > > Update from our OPS team, regarding zookeeper 3.4.x. Given stability,
> > > adoption of offset batching would be the only remaining bit of work to
> > >
> >
> > do.
> > > Still, I totally understand the restraint for 0.8...
> > >
> > >
> > > "As exercise in upgradability of zookeeper, I did a "out-of-the"box"
> > > upgrade on Zookeeper. I downloaded a generic distribution of Apache
> > > Zookeeper and used it for the upgrade.
> > >
> > > Kafka included version of Zookeeper 3.3.3.
> > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > >
> > > Running, working great. I did *not* have to wipe out the zookeeper
> > > databases. All data stayed intact.
> > >
> > > I got a new feature, which allows auto-purging of logs. This keeps OPS
> > > maintenance to a minimum."
> > >
> > >
> > > thanks,
> > > rob
> > >
> > >
> > > -----Original Message-----
> > > From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, May 17, 2013 7:38 AM
> > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > Subject: RE: are commitOffsets botched to zookeeper?
> > >
> > > Fair enough, this is something to look forward to. I appreciate the
> > > restraint you show to stay out of troubled waters. :)
> > >
> > > thanks,
> > > rob
> > >
> > > ________________________________________
> > > From: Neha Narkhede [[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])]
> > > Sent: Friday, May 17, 2013 7:35 AM
> > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > Subject: RE: are commitOffsets botched to zookeeper?
> > >
> > > Upgrading to a new zookeeper version is not an easy change. Also
> > zookeeper
> > > 3.3.4 is much more stable compared to 3.4.x. We think it is better not to
> > > club 2 big changes together. So most likely this will be a post 08 item
> > >
> >
> > for
> > > stability purposes.
> > >
> > > Thanks,
> > > Neha
> > > On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > > wrote:
> > >
> > > > Awesome! Thanks for the clarification. I would like to offer my
> > > > strong vote that this get tackled before a beta, to get it firmly into
> > > >
> > >
> > > 0.8.
> > > > Stabilize everything else to the existing use, but make offset updates
> > > > batched.
> > > >
> > > > thanks,
> > > > rob
> > > > ________________________________________
> > > > From: Neha Narkhede [[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])]
> > > > Sent: Friday, May 17, 2013 7:17 AM
> > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > >
> > > > Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08
> > > > is stable and released it will be worth looking into when we can use
> > > > zookeeper 3.4.x.
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 10:32 PM, "Rob Withers" <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: