Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Re: only-once consumer groups


+
Neha Narkhede 2013-05-17, 04:03
+
Rob Withers 2013-05-17, 05:29
+
Neha Narkhede 2013-05-17, 13:32
+
Rob Withers 2013-05-21, 01:45
+
Neha Narkhede 2013-05-21, 02:00
Copy link to this message
-
RE: only-once consumer groups
Rob Withers 2013-05-21, 02:21
Right, Neha, consumer groups are from 7, while replicas are in 8.  Does this
mean the simple consumer in 8 can recognize a leader change?

Thanks,
rob

> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 20, 2013 8:00 PM
> To: [EMAIL PROTECTED]
> Subject: RE: only-once consumer groups
>
> Rob,
>
> A consumer rebalances whenever a consumer process dies or a new
> consumer process joins the group. The details of the algorithm can be
found
> here http://kafka.apache.org/07/design.html
>
> Thanks,
> Neha
> On May 20, 2013 6:45 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
>
> > That page is packed full of super design!  Many of these features we
> > would find useful, I think.  One thing I found myself not knowing is
> > what a consumer rebalance actually is.
> >
> > Is a rebalance when the thread that is consuming a particular
> > partition dies, this is detected and the partition is reassigned to a
> > new thread, thus violating manual partitioning?
> >
> > Or is a rebalance when a broker dies and new leader partitions are
elected?
> > The simple consumer must get told of a leader election and this is
> > nothing to do with rebalance, is it so?
> >
> > Thanks,
> > rob
> >
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, May 17, 2013 7:32 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: only-once consumer groups
> > >
> > > We spent some time thinking about consolidating the high level and
> > > low
> > level
> > > consumer APIs. It will be great if you can read the wiki and provide
> > feedback
> > > -
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re
> > > -
> > > Design
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > >
> > > > We want to ensure only-once message processing, but we also want
> > > > the benefit of rebalancing.  commitOffsets updates all partitions
> > > > from out of a connector instance.  We want to commit the offset
> > > > for just the partition that delivered a message to the iterator,
> > > > even if several fetchers are feeding a thread.  Perhaps the
> > > > message metadata contains the partition a msg came from; could you
> > > > not just update the offset for that partition if the property
> > > > "only.once=true" is sent to the consumer connector?
> > > >
> > > > Thanks,
> > > > rob
> > > >
> > > > > -----Original Message-----
> > > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > > To: [EMAIL PROTECTED]
> > > > > Subject: Re: only-once consumer groups
> > > > >
> > > > > Can you describe your requirements in a little more detail?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > > <[EMAIL PROTECTED]>
> > > > > wrote:
> > > > >
> > > > > > is it technically feasible to use an only-once simple consumer
> > > > > > within a consumer group?
> > > > > >
> > > > > > thanks,
> > > > > > rob
> > > >
> > > >
> >
> >
 
+
Rob Withers 2013-05-21, 02:55