Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> are commitOffsets botched to zookeeper?


+
Rob Withers 2013-05-17, 01:54
+
Neha Narkhede 2013-05-17, 03:53
+
Rob Withers 2013-05-17, 05:32
+
Neha Narkhede 2013-05-17, 13:18
Copy link to this message
-
RE: are commitOffsets botched to zookeeper?
Upgrading to a new zookeeper version is not an easy change. Also zookeeper
3.3.4 is much more stable compared to 3.4.x. We think it is better not to
club 2 big changes together. So most likely this will be a post 08 item for
stability purposes.

Thanks,
Neha
On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:

> Awesome!  Thanks for the clarification.  I would like to offer my strong
> vote that this get tackled before a beta, to get it firmly into 0.8.
> Stabilize everything else to the existing use, but make offset updates
> batched.
>
> thanks,
> rob
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:17 AM
> To: [EMAIL PROTECTED]
> Subject: RE: are commitOffsets botched to zookeeper?
>
> Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08 is
> stable and released it will be worth looking into when we can use zookeeper
> 3.4.x.
>
> Thanks,
> Neha
> On May 16, 2013 10:32 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
>
> > Can a request be made to zookeeper for this feature?
> >
> > Thanks,
> > rob
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > Sent: Thursday, May 16, 2013 9:53 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: are commitOffsets botched to zookeeper?
> > >
> > > Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch
> > write
> > > api. So if you commit after every message at a high rate, it will be
> slow
> > and
> > > inefficient. Besides it will cause zookeeper performance to degrade.
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 6:54 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > >
> > > > We are calling commitOffsets after every message consumption.  It
> > > > looks to be ~60% slower, with 29 partitions.  If a single KafkaStream
> > > > thread is from a connector, and there are 29 partitions, then
> > > > commitOffsets sends 29 offset updates, correct?  Are these offset
> > > > updates batched in one send to zookeeper?
> > > >
> > > > thanks,
> > > > rob
> >
> >
>

 
+
Neha Narkhede 2013-05-20, 16:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB