Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
That page is packed full of super design!  Many of these features we would
find useful, I think.  One thing I found myself not knowing is what a
consumer rebalance actually is.  

Is a rebalance when the thread that is consuming a particular partition
dies, this is detected and the partition is reassigned to a new thread, thus
violating manual partitioning?  

Or is a rebalance when a broker dies and new leader partitions are elected?
The simple consumer must get told of a leader election and this is nothing
to do with rebalance, is it so?

Thanks,
rob
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:32 AM
> To: [EMAIL PROTECTED]
> Subject: RE: only-once consumer groups
>
> We spent some time thinking about consolidating the high level and low
level
> consumer APIs. It will be great if you can read the wiki and provide
feedback
> - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-
> Design
>
> Thanks,
> Neha
> On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
>
> > We want to ensure only-once message processing, but we also want the
> > benefit of rebalancing.  commitOffsets updates all partitions from out
> > of a connector instance.  We want to commit the offset for just the
> > partition that delivered a message to the iterator, even if several
> > fetchers are feeding a thread.  Perhaps the message metadata contains
> > the partition a msg came from; could you not just update the offset
> > for that partition if the property "only.once=true" is sent to the
> > consumer connector?
> >
> > Thanks,
> > rob
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > Sent: Thursday, May 16, 2013 10:03 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: only-once consumer groups
> > >
> > > Can you describe your requirements in a little more detail?
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 6:11 AM, "Withers, Robert"
> <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > is it technically feasible to use an only-once simple consumer
> > > > within a consumer group?
> > > >
> > > > thanks,
> > > > rob
> >
> >