Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
Sorry for being unclear Neha.  I meant that I had forgotten that the
introduction of replicas is happening in 0.8 and I was confusing the two.  

Thanks,
rob

> -----Original Message-----
> From: Rob Withers [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 20, 2013 8:21 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: only-once consumer groups
>
> Right, Neha, consumer groups are from 7, while replicas are in 8.  Does
this
> mean the simple consumer in 8 can recognize a leader change?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, May 20, 2013 8:00 PM
> > To: [EMAIL PROTECTED]
> > Subject: RE: only-once consumer groups
> >
> > Rob,
> >
> > A consumer rebalances whenever a consumer process dies or a new
> > consumer process joins the group. The details of the algorithm can be
> > found here http://kafka.apache.org/07/design.html
> >
> > Thanks,
> > Neha
> > On May 20, 2013 6:45 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> >
> > > That page is packed full of super design!  Many of these features we
> > > would find useful, I think.  One thing I found myself not knowing is
> > > what a consumer rebalance actually is.
> > >
> > > Is a rebalance when the thread that is consuming a particular
> > > partition dies, this is detected and the partition is reassigned to
> > > a new thread, thus violating manual partitioning?
> > >
> > > Or is a rebalance when a broker dies and new leader partitions are
> elected?
> > > The simple consumer must get told of a leader election and this is
> > > nothing to do with rebalance, is it so?
> > >
> > > Thanks,
> > > rob
> > >
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > Sent: Friday, May 17, 2013 7:32 AM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: RE: only-once consumer groups
> > > >
> > > > We spent some time thinking about consolidating the high level and
> > > > low
> > > level
> > > > consumer APIs. It will be great if you can read the wiki and
> > > > provide
> > > feedback
> > > > -
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re
> > > > -
> > > > Design
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > We want to ensure only-once message processing, but we also want
> > > > > the benefit of rebalancing.  commitOffsets updates all
> > > > > partitions from out of a connector instance.  We want to commit
> > > > > the offset for just the partition that delivered a message to
> > > > > the iterator, even if several fetchers are feeding a thread.
> > > > > Perhaps the message metadata contains the partition a msg came
> > > > > from; could you not just update the offset for that partition if
> > > > > the property "only.once=true" is sent to the consumer connector?
> > > > >
> > > > > Thanks,
> > > > > rob
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > > > To: [EMAIL PROTECTED]
> > > > > > Subject: Re: only-once consumer groups
> > > > > >
> > > > > > Can you describe your requirements in a little more detail?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > > > <[EMAIL PROTECTED]>
> > > > > > wrote:
> > > > > >
> > > > > > > is it technically feasible to use an only-once simple
> > > > > > > consumer within a consumer group?
> > > > > > >
> > > > > > > thanks,
> > > > > > > rob
> > > > >
> > > > >
> > >
> > >