Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: only-once consumer groups


+
Neha Narkhede 2013-05-17, 04:03
+
Rob Withers 2013-05-17, 05:29
+
Neha Narkhede 2013-05-17, 13:32
+
Rob Withers 2013-05-21, 01:45
+
Neha Narkhede 2013-05-21, 02:00
+
Rob Withers 2013-05-21, 02:21
Copy link to this message
-
RE: only-once consumer groups
Sorry for being unclear Neha.  I meant that I had forgotten that the
introduction of replicas is happening in 0.8 and I was confusing the two.  

Thanks,
rob

> -----Original Message-----
> From: Rob Withers [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 20, 2013 8:21 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: only-once consumer groups
>
> Right, Neha, consumer groups are from 7, while replicas are in 8.  Does
this
> mean the simple consumer in 8 can recognize a leader change?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, May 20, 2013 8:00 PM
> > To: [EMAIL PROTECTED]
> > Subject: RE: only-once consumer groups
> >
> > Rob,
> >
> > A consumer rebalances whenever a consumer process dies or a new
> > consumer process joins the group. The details of the algorithm can be
> > found here http://kafka.apache.org/07/design.html
> >
> > Thanks,
> > Neha
> > On May 20, 2013 6:45 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> >
> > > That page is packed full of super design!  Many of these features we
> > > would find useful, I think.  One thing I found myself not knowing is
> > > what a consumer rebalance actually is.
> > >
> > > Is a rebalance when the thread that is consuming a particular
> > > partition dies, this is detected and the partition is reassigned to
> > > a new thread, thus violating manual partitioning?
> > >
> > > Or is a rebalance when a broker dies and new leader partitions are
> elected?
> > > The simple consumer must get told of a leader election and this is
> > > nothing to do with rebalance, is it so?
> > >
> > > Thanks,
> > > rob
> > >
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > Sent: Friday, May 17, 2013 7:32 AM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: RE: only-once consumer groups
> > > >
> > > > We spent some time thinking about consolidating the high level and
> > > > low
> > > level
> > > > consumer APIs. It will be great if you can read the wiki and
> > > > provide
> > > feedback
> > > > -
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re
> > > > -
> > > > Design
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > We want to ensure only-once message processing, but we also want
> > > > > the benefit of rebalancing.  commitOffsets updates all
> > > > > partitions from out of a connector instance.  We want to commit
> > > > > the offset for just the partition that delivered a message to
> > > > > the iterator, even if several fetchers are feeding a thread.
> > > > > Perhaps the message metadata contains the partition a msg came
> > > > > from; could you not just update the offset for that partition if
> > > > > the property "only.once=true" is sent to the consumer connector?
> > > > >
> > > > > Thanks,
> > > > > rob
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > > > To: [EMAIL PROTECTED]
> > > > > > Subject: Re: only-once consumer groups
> > > > > >
> > > > > > Can you describe your requirements in a little more detail?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > > > <[EMAIL PROTECTED]>
> > > > > > wrote:
> > > > > >
> > > > > > > is it technically feasible to use an only-once simple
> > > > > > > consumer within a consumer group?
> > > > > > >
> > > > > > > thanks,
> > > > > > > rob
> > > > >
> > > > >
> > >
> > >
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB