Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Update: RE: are commitOffsets botched to zookeeper?


Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
No problem. You can take a look at some of the thoughts we had on improving
the offset storage here -
https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management.
Suggestions are welcome.

Thanks,
Neha
On Fri, May 17, 2013 at 2:40 PM, Alex Zuzin <[EMAIL PROTECTED]> wrote:

> Neha,
>
> apologies, I just re-read what I sent and realized my "you" wasn't
> specific enough - it meant the Kafka team ;).
>
> --
> "If you can't conceal it well, expose it with all your might"
> Alex Zuzin
>
>
> On Friday, May 17, 2013 at 2:25 PM, Alex Zuzin wrote:
>
> > Have you considered abstracting offset storage away so people could
> implement their own?
> > Would you take a patch if I'd stabbed at it, and if yes, what's the
> process (pardon the n00b)?
> >
> > KCBO,
> > --
> > "If you can't conceal it well, expose it with all your might"
> > Alex Zuzin
> >
> >
> > On Friday, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:
> >
> > > There is no particular need for storing the offsets in zookeeper. In
> fact
> > > with Kafka 0.8, since partitions will be highly available, offsets
> could be
> > > stored in Kafka topics. However, we haven't ironed out the design for
> this
> > > yet.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])> wrote:
> > >
> > > > afaik you dont 'have' to store the consumed offsets in zk right,
> this is
> > > > only automatic with some of the clients?
> > > >
> > > > why not store them in a data store that can write at the rate that
> you
> > > > require?
> > > >
> > > >
> > > > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > wrote:
> > > >
> > > >
> > > > > Update from our OPS team, regarding zookeeper 3.4.x. Given
> stability,
> > > > > adoption of offset batching would be the only remaining bit of
> work to
> > > > >
> > > >
> > > > do.
> > > > > Still, I totally understand the restraint for 0.8...
> > > > >
> > > > >
> > > > > "As exercise in upgradability of zookeeper, I did a
> "out-of-the"box"
> > > > > upgrade on Zookeeper. I downloaded a generic distribution of Apache
> > > > > Zookeeper and used it for the upgrade.
> > > > >
> > > > > Kafka included version of Zookeeper 3.3.3.
> > > > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > > > >
> > > > > Running, working great. I did *not* have to wipe out the zookeeper
> > > > > databases. All data stayed intact.
> > > > >
> > > > > I got a new feature, which allows auto-purging of logs. This keeps
> OPS
> > > > > maintenance to a minimum."
> > > > >
> > > > >
> > > > > thanks,
> > > > > rob
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> > > > > Sent: Friday, May 17, 2013 7:38 AM
> > > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > > >
> > > > > Fair enough, this is something to look forward to. I appreciate the
> > > > > restraint you show to stay out of troubled waters. :)
> > > > >
> > > > > thanks,
> > > > > rob
> > > > >
> > > > > ________________________________________
> > > > > From: Neha Narkhede [[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])]
> > > > > Sent: Friday, May 17, 2013 7:35 AM
> > > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > > >
> > > > > Upgrading to a new zookeeper version is not an easy change. Also
> > > > zookeeper
> > > > > 3.3.4 is much more stable compared to 3.4.x. We think it is better
> not to
> > > > > club 2 big changes together. So most likely this will be a post 08
> item
> > > > >
> > > >
> > > > for
> > > > > stability purposes.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > > On May 17, 2013 6:31 AM, "Withers, Robert" <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> >