Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Update: RE: are commitOffsets botched to zookeeper?


Copy link to this message
-
Re: Update: RE: are commitOffsets botched to zookeeper?
Did so. The proposal looks perfectly sensible on first reading.

I understand that the patches in https://issues.apache.org/jira/browse/KAFKA-657 are already in the trunk and scheduled for 0.8.1? Are they going out with 0.8? If not, what's ETA for 0.8.1?

Either way, I'm going to try my hand at backing this with MySQL and report the results here shortly.

--
"If you can't conceal it well, expose it with all your might"
Alex Zuzin
On Monday, May 20, 2013 at 10:24 AM, Neha Narkhede wrote:

> No problem. You can take a look at some of the thoughts we had on improving
> the offset storage here -
> https://cwiki.apache.org/confluence/display/KAFKA/ffset+Management (https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management).
> Suggestions are welcome.
>
> Thanks,
> Neha
>
>
> On Fri, May 17, 2013 at 2:40 PM, Alex Zuzin <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > Neha,
> >
> > apologies, I just re-read what I sent and realized my "you" wasn't
> > specific enough - it meant the Kafka team ;).
> >
> > --
> > "If you can't conceal it well, expose it with all your might"
> > Alex Zuzin
> >
> >
> > On Friday, May 17, 2013 at 2:25 PM, Alex Zuzin wrote:
> >
> > > Have you considered abstracting offset storage away so people could
> > implement their own?
> > > Would you take a patch if I'd stabbed at it, and if yes, what's the
> >
> > process (pardon the n00b)?
> > >
> > > KCBO,
> > > --
> > > "If you can't conceal it well, expose it with all your might"
> > > Alex Zuzin
> > >
> > >
> > > On Friday, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:
> > >
> > > > There is no particular need for storing the offsets in zookeeper. In
> > fact
> > > > with Kafka 0.8, since partitions will be highly available, offsets
> > >
> >
> > could be
> > > > stored in Kafka topics. However, we haven't ironed out the design for
> > >
> >
> > this
> > > > yet.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Fri, May 17, 2013 at 2:19 PM, Scott Clasen <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > >
> > > > > afaik you dont 'have' to store the consumed offsets in zk right,
> > this is
> > > > > only automatic with some of the clients?
> > > > >
> > > > > why not store them in a data store that can write at the rate that
> > you
> > > > > require?
> > > > >
> > > > >
> > > > > On Fri, May 17, 2013 at 2:15 PM, Withers, Robert <
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > > Update from our OPS team, regarding zookeeper 3.4.x. Given
> > stability,
> > > > > > adoption of offset batching would be the only remaining bit of
> > > > >
> > > >
> > >
> >
> > work to
> > > > > >
> > > > >
> > > > >
> > > > > do.
> > > > > > Still, I totally understand the restraint for 0.8...
> > > > > >
> > > > > >
> > > > > > "As exercise in upgradability of zookeeper, I did a
> > "out-of-the"box"
> > > > > > upgrade on Zookeeper. I downloaded a generic distribution of Apache
> > > > > > Zookeeper and used it for the upgrade.
> > > > > >
> > > > > > Kafka included version of Zookeeper 3.3.3.
> > > > > > Out of the box Apache Zookeeper 3.4.5 (which I upgraded to)
> > > > > >
> > > > > > Running, working great. I did *not* have to wipe out the zookeeper
> > > > > > databases. All data stayed intact.
> > > > > >
> > > > > > I got a new feature, which allows auto-purging of logs. This keeps
> > OPS
> > > > > > maintenance to a minimum."
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > > rob
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Withers, Robert [mailto:[EMAIL PROTECTED]]
> > > > > > Sent: Friday, May 17, 2013 7:38 AM
> > > > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > > > Subject: RE: are commitOffsets botched to zookeeper?
> > > > > >
> > > > > > Fair enough, this is something to look forward to. I appreciate the

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB