-RE: only-once consumer groups
Rob Withers 2013-05-21, 01:45
That page is packed full of super design! Many of these features we would
find useful, I think. One thing I found myself not knowing is what a
consumer rebalance actually is.
Is a rebalance when the thread that is consuming a particular partition
dies, this is detected and the partition is reassigned to a new thread, thus
violating manual partitioning?
Or is a rebalance when a broker dies and new leader partitions are elected?
The simple consumer must get told of a leader election and this is nothing
to do with rebalance, is it so?
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:32 AM
> To: [EMAIL PROTECTED]
> Subject: RE: only-once consumer groups
> We spent some time thinking about consolidating the high level and low
> consumer APIs. It will be great if you can read the wiki and provide
> - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-
> On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> > We want to ensure only-once message processing, but we also want the
> > benefit of rebalancing. commitOffsets updates all partitions from out
> > of a connector instance. We want to commit the offset for just the
> > partition that delivered a message to the iterator, even if several
> > fetchers are feeding a thread. Perhaps the message metadata contains
> > the partition a msg came from; could you not just update the offset
> > for that partition if the property "only.once=true" is sent to the
> > consumer connector?
> > Thanks,
> > rob
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > Sent: Thursday, May 16, 2013 10:03 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: only-once consumer groups
> > >
> > > Can you describe your requirements in a little more detail?
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 6:11 AM, "Withers, Robert"
> <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > is it technically feasible to use an only-once simple consumer
> > > > within a consumer group?
> > > >
> > > > thanks,
> > > > rob