Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: Update: RE: are commitOffsets botched to zookeeper?


+
Scott Clasen 2013-05-17, 21:19
+
Neha Narkhede 2013-05-17, 21:23
Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
to clarify, I meant that Robert could/should store the offsets in a faster
store not that kafka should default to that :)

Thanks Neha
On Fri, May 17, 2013 at 2:22 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> There is no particular need for storing the offsets in zookeeper. In fact
> with Kafka 0.8, since partitions will be highly available, offsets could be
> stored in Kafka topics. However, we haven't ironed out the design for this
> yet.
>
> Thanks,
> Neha
>
>
> On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED]> wrote:
>
> > afaik you dont 'have' to store the consumed offsets in zk right, this is
> > only automatic with some of the clients?
> >
> > why not store them in a data store that can write at the rate that you
> > require?
> >
> >
> > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Update from our OPS team, regarding zookeeper 3.4.x.  Given stability,
> > > adoption of offset batching would be the only remaining bit of work to
> > do.
> > >  Still, I totally understand the restraint for 0.8...
> > >
> > >
> > > "As exercise in upgradability of zookeeper, I did a "out-of-the"box"
> > > upgrade on Zookeeper. I downloaded a generic distribution of Apache
> > > Zookeeper and used it for the upgrade.
> > >
> > > Kafka included version of Zookeeper 3.3.3.
> > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > >
> > > Running, working great. I did *not* have to wipe out the zookeeper
> > > databases. All data stayed intact.
> > >
> > > I got a new feature, which allows auto-purging of logs. This keeps OPS
> > > maintenance to a minimum."
> > >
> > >
> > > thanks,
> > > rob
> > >
> > >
> > > -----Original Message-----
> > > From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, May 17, 2013 7:38 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: are commitOffsets botched to zookeeper?
> > >
> > > Fair enough, this is something to look forward to.  I appreciate the
> > > restraint you show to stay out of troubled waters.  :)
> > >
> > > thanks,
> > > rob
> > >
> > > ________________________________________
> > > From: Neha Narkhede [[EMAIL PROTECTED]]
> > > Sent: Friday, May 17, 2013 7:35 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: are commitOffsets botched to zookeeper?
> > >
> > > Upgrading to a new zookeeper version is not an easy change. Also
> > zookeeper
> > > 3.3.4 is much more stable compared to 3.4.x. We think it is better not
> to
> > > club 2 big changes together. So most likely this will be a post 08 item
> > for
> > > stability purposes.
> > >
> > > Thanks,
> > > Neha
> > > On May 17, 2013 6:31 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Awesome!  Thanks for the clarification.  I would like to offer my
> > > > strong vote that this get tackled before a beta, to get it firmly
> into
> > > 0.8.
> > > > Stabilize everything else to the existing use, but make offset
> updates
> > > > batched.
> > > >
> > > > thanks,
> > > > rob
> > > > ________________________________________
> > > > From: Neha Narkhede [[EMAIL PROTECTED]]
> > > > Sent: Friday, May 17, 2013 7:17 AM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > >
> > > > Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08
> > > > is stable and released it will be worth looking into when we can use
> > > > zookeeper 3.4.x.
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 10:32 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Can a request be made to zookeeper for this feature?
> > > > >
> > > > > Thanks,
> > > > > rob
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > > > Sent: Thursday, May 16, 2013 9:53 PM
> > > > > > To: [EMAIL PROTECTED]
> > > > > > Subject: Re: are commitOffsets botched to zookeeper?
> > > > > >
> > > > > > Currently Kafka depends on zookeeper 3.3.4 that doesn't have a

 
+
Alex Zuzin 2013-05-17, 21:26
+
Alex Zuzin 2013-05-17, 21:40
+
Neha Narkhede 2013-05-20, 17:25
+
Alex Zuzin 2013-05-20, 17:37
+
Rob Withers 2013-05-21, 01:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB