Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - are commitOffsets botched to zookeeper?


+
Rob Withers 2013-05-17, 01:54
+
Neha Narkhede 2013-05-17, 03:53
+
Rob Withers 2013-05-17, 05:32
+
Neha Narkhede 2013-05-17, 13:18
Copy link to this message
-
RE: are commitOffsets botched to zookeeper?
Neha Narkhede 2013-05-17, 13:35
Upgrading to a new zookeeper version is not an easy change. Also zookeeper
3.3.4 is much more stable compared to 3.4.x. We think it is better not to
club 2 big changes together. So most likely this will be a post 08 item for
stability purposes.

Thanks,
Neha
On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:

> Awesome!  Thanks for the clarification.  I would like to offer my strong
> vote that this get tackled before a beta, to get it firmly into 0.8.
> Stabilize everything else to the existing use, but make offset updates
> batched.
>
> thanks,
> rob
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:17 AM
> To: [EMAIL PROTECTED]
> Subject: RE: are commitOffsets botched to zookeeper?
>
> Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08 is
> stable and released it will be worth looking into when we can use zookeeper
> 3.4.x.
>
> Thanks,
> Neha
> On May 16, 2013 10:32 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
>
> > Can a request be made to zookeeper for this feature?
> >
> > Thanks,
> > rob
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > Sent: Thursday, May 16, 2013 9:53 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: are commitOffsets botched to zookeeper?
> > >
> > > Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch
> > write
> > > api. So if you commit after every message at a high rate, it will be
> slow
> > and
> > > inefficient. Besides it will cause zookeeper performance to degrade.
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 6:54 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > >
> > > > We are calling commitOffsets after every message consumption.  It
> > > > looks to be ~60% slower, with 29 partitions.  If a single KafkaStream
> > > > thread is from a connector, and there are 29 partitions, then
> > > > commitOffsets sends 29 offset updates, correct?  Are these offset
> > > > updates batched in one send to zookeeper?
> > > >
> > > > thanks,
> > > > rob
> >
> >
>

 
+
Neha Narkhede 2013-05-20, 16:40