Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: Update: RE: are commitOffsets botched to zookeeper?


Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
Scott Clasen 2013-05-17, 21:19
afaik you dont 'have' to store the consumed offsets in zk right, this is
only automatic with some of the clients?

why not store them in a data store that can write at the rate that you
require?
On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <[EMAIL PROTECTED]>wrote:

> Update from our OPS team, regarding zookeeper 3.4.x.  Given stability,
> adoption of offset batching would be the only remaining bit of work to do.
>  Still, I totally understand the restraint for 0.8...
>
>
> "As exercise in upgradability of zookeeper, I did a "out-of-the"box"
> upgrade on Zookeeper. I downloaded a generic distribution of Apache
> Zookeeper and used it for the upgrade.
>
> Kafka included version of Zookeeper 3.3.3.
> Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
>
> Running, working great. I did *not* have to wipe out the zookeeper
> databases. All data stayed intact.
>
> I got a new feature, which allows auto-purging of logs. This keeps OPS
> maintenance to a minimum."
>
>
> thanks,
> rob
>
>
> -----Original Message-----
> From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:38 AM
> To: [EMAIL PROTECTED]
> Subject: RE: are commitOffsets botched to zookeeper?
>
> Fair enough, this is something to look forward to.  I appreciate the
> restraint you show to stay out of troubled waters.  :)
>
> thanks,
> rob
>
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:35 AM
> To: [EMAIL PROTECTED]
> Subject: RE: are commitOffsets botched to zookeeper?
>
> Upgrading to a new zookeeper version is not an easy change. Also zookeeper
> 3.3.4 is much more stable compared to 3.4.x. We think it is better not to
> club 2 big changes together. So most likely this will be a post 08 item for
> stability purposes.
>
> Thanks,
> Neha
> On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > Awesome!  Thanks for the clarification.  I would like to offer my
> > strong vote that this get tackled before a beta, to get it firmly into
> 0.8.
> > Stabilize everything else to the existing use, but make offset updates
> > batched.
> >
> > thanks,
> > rob
> > ________________________________________
> > From: Neha Narkhede [[EMAIL PROTECTED]]
> > Sent: Friday, May 17, 2013 7:17 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: are commitOffsets botched to zookeeper?
> >
> > Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08
> > is stable and released it will be worth looking into when we can use
> > zookeeper 3.4.x.
> >
> > Thanks,
> > Neha
> > On May 16, 2013 10:32 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> >
> > > Can a request be made to zookeeper for this feature?
> > >
> > > Thanks,
> > > rob
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > Sent: Thursday, May 16, 2013 9:53 PM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: Re: are commitOffsets botched to zookeeper?
> > > >
> > > > Currently Kafka depends on zookeeper 3.3.4 that doesn't have a
> > > > batch
> > > write
> > > > api. So if you commit after every message at a high rate, it will
> > > > be
> > slow
> > > and
> > > > inefficient. Besides it will cause zookeeper performance to degrade.
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 6:54 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > We are calling commitOffsets after every message consumption.
> > > > > It looks to be ~60% slower, with 29 partitions.  If a single
> > > > > KafkaStream thread is from a connector, and there are 29
> > > > > partitions, then commitOffsets sends 29 offset updates, correct?
> > > > > Are these offset updates batched in one send to zookeeper?
> > > > >
> > > > > thanks,
> > > > > rob
> > >
> > >
> >
>