Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Update: RE: are commitOffsets botched to zookeeper?


Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
Neha,

apologies, I just re-read what I sent and realized my "you" wasn't specific enough - it meant the Kafka team ;).

--
"If you can't conceal it well, expose it with all your might"
Alex Zuzin
On Friday, May 17, 2013 at 2:25 PM, Alex Zuzin wrote:

> Have you considered abstracting offset storage away so people could implement their own?
> Would you take a patch if I'd stabbed at it, and if yes, what's the process (pardon the n00b)?
>
> KCBO,
> --
> "If you can't conceal it well, expose it with all your might"
> Alex Zuzin
>
>
> On Friday, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:
>
> > There is no particular need for storing the offsets in zookeeper. In fact
> > with Kafka 0.8, since partitions will be highly available, offsets could be
> > stored in Kafka topics. However, we haven't ironed out the design for this
> > yet.
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> >
> > > afaik you dont 'have' to store the consumed offsets in zk right, this is
> > > only automatic with some of the clients?
> > >
> > > why not store them in a data store that can write at the rate that you
> > > require?
> > >
> > >
> > > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > wrote:
> > >
> > >
> > > > Update from our OPS team, regarding zookeeper 3.4.x. Given stability,
> > > > adoption of offset batching would be the only remaining bit of work to
> > > >
> > >
> > > do.
> > > > Still, I totally understand the restraint for 0.8...
> > > >
> > > >
> > > > "As exercise in upgradability of zookeeper, I did a "out-of-the"box"
> > > > upgrade on Zookeeper. I downloaded a generic distribution of Apache
> > > > Zookeeper and used it for the upgrade.
> > > >
> > > > Kafka included version of Zookeeper 3.3.3.
> > > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > > >
> > > > Running, working great. I did *not* have to wipe out the zookeeper
> > > > databases. All data stayed intact.
> > > >
> > > > I got a new feature, which allows auto-purging of logs. This keeps OPS
> > > > maintenance to a minimum."
> > > >
> > > >
> > > > thanks,
> > > > rob
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> > > > Sent: Friday, May 17, 2013 7:38 AM
> > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > >
> > > > Fair enough, this is something to look forward to. I appreciate the
> > > > restraint you show to stay out of troubled waters. :)
> > > >
> > > > thanks,
> > > > rob
> > > >
> > > > ________________________________________
> > > > From: Neha Narkhede [[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])]
> > > > Sent: Friday, May 17, 2013 7:35 AM
> > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > >
> > > > Upgrading to a new zookeeper version is not an easy change. Also
> > > zookeeper
> > > > 3.3.4 is much more stable compared to 3.4.x. We think it is better not to
> > > > club 2 big changes together. So most likely this will be a post 08 item
> > > >
> > >
> > > for
> > > > stability purposes.
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > > > wrote:
> > > >
> > > > > Awesome! Thanks for the clarification. I would like to offer my
> > > > > strong vote that this get tackled before a beta, to get it firmly into
> > > > >
> > > >
> > > > 0.8.
> > > > > Stabilize everything else to the existing use, but make offset updates
> > > > > batched.
> > > > >
> > > > > thanks,
> > > > > rob
> > > > > ________________________________________
> > > > > From: Neha Narkhede [[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])]