Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Re: Update: RE: are commitOffsets botched to zookeeper?


+
Scott Clasen 2013-05-17, 21:19
+
Neha Narkhede 2013-05-17, 21:23
+
Scott Clasen 2013-05-17, 21:26
+
Alex Zuzin 2013-05-17, 21:26
+
Alex Zuzin 2013-05-17, 21:40
+
Neha Narkhede 2013-05-20, 17:25
+
Alex Zuzin 2013-05-20, 17:37
Copy link to this message
-
RE: Update: RE: are commitOffsets botched to zookeeper?
Rob Withers 2013-05-21, 01:35
Yes, it looks spot on.

Thanks,
rob

> -----Original Message-----
> From: Alex Zuzin [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 20, 2013 11:37 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Update: RE: are commitOffsets botched to zookeeper?
>
> Did so. The proposal looks perfectly sensible on first reading.
>
> I understand that the patches in
> https://issues.apache.org/jira/browse/KAFKA-657 are already in the trunk
> and scheduled for 0.8.1? Are they going out with 0.8? If not, what's ETA for
> 0.8.1?
>
> Either way, I'm going to try my hand at backing this with MySQL and report
> the results here shortly.
>
> --
> "If you can't conceal it well, expose it with all your might"
> Alex Zuzin
>
>
> On Monday, May 20, 2013 at 10:24 AM, Neha Narkhede wrote:
>
> > No problem. You can take a look at some of the thoughts we had on
> improving
> > the offset storage here -
> > https://cwiki.apache.org/confluence/display/KAFKA/ffset+Management
> (https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management).
> > Suggestions are welcome.
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, May 17, 2013 at 2:40 PM, Alex Zuzin <[EMAIL PROTECTED]
> (mailto:[EMAIL PROTECTED])> wrote:
> >
> > > Neha,
> > >
> > > apologies, I just re-read what I sent and realized my "you" wasn't
> > > specific enough - it meant the Kafka team ;).
> > >
> > > --
> > > "If you can't conceal it well, expose it with all your might"
> > > Alex Zuzin
> > >
> > >
> > > On Friday, May 17, 2013 at 2:25 PM, Alex Zuzin wrote:
> > >
> > > > Have you considered abstracting offset storage away so people could
> > > implement their own?
> > > > Would you take a patch if I'd stabbed at it, and if yes, what's the
> > >
> > > process (pardon the n00b)?
> > > >
> > > > KCBO,
> > > > --
> > > > "If you can't conceal it well, expose it with all your might"
> > > > Alex Zuzin
> > > >
> > > >
> > > > On Friday, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:
> > > >
> > > > > There is no particular need for storing the offsets in zookeeper. In
> > > fact
> > > > > with Kafka 0.8, since partitions will be highly available, offsets
> > > >
> > >
> > > could be
> > > > > stored in Kafka topics. However, we haven't ironed out the design for
> > > >
> > >
> > > this
> > > > > yet.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED]
> (mailto:[EMAIL PROTECTED])(mailto:
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > > >
> > > > > > afaik you dont 'have' to store the consumed offsets in zk right,
> > > this is
> > > > > > only automatic with some of the clients?
> > > > > >
> > > > > > why not store them in a data store that can write at the rate that
> > > you
> > > > > > require?
> > > > > >
> > > > > >
> > > > > > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > > > wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Update from our OPS team, regarding zookeeper 3.4.x. Given
> > > stability,
> > > > > > > adoption of offset batching would be the only remaining bit of
> > > > > >
> > > > >
> > > >
> > >
> > > work to
> > > > > > >
> > > > > >
> > > > > >
> > > > > > do.
> > > > > > > Still, I totally understand the restraint for 0.8...
> > > > > > >
> > > > > > >
> > > > > > > "As exercise in upgradability of zookeeper, I did a
> > > "out-of-the"box"
> > > > > > > upgrade on Zookeeper. I downloaded a generic distribution of
> Apache
> > > > > > > Zookeeper and used it for the upgrade.
> > > > > > >
> > > > > > > Kafka included version of Zookeeper 3.3.3.
> > > > > > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > > > > > >
> > > > > > > Running, working great. I did *not* have to wipe out the
> zookeeper
> > > > > > > databases. All data stayed intact.
> > > > > > >
> > > > > > > I got a new feature, which allows auto-purging of logs. This keeps
> > > OPS
> > > > > > > maintenance to a minimum."