Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
Rob,

A consumer rebalances whenever a consumer process dies or a new consumer
process joins the group. The details of the algorithm can be found here
http://kafka.apache.org/07/design.html

Thanks,
Neha
On May 20, 2013 6:45 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:

> That page is packed full of super design!  Many of these features we would
> find useful, I think.  One thing I found myself not knowing is what a
> consumer rebalance actually is.
>
> Is a rebalance when the thread that is consuming a particular partition
> dies, this is detected and the partition is reassigned to a new thread,
> thus
> violating manual partitioning?
>
> Or is a rebalance when a broker dies and new leader partitions are elected?
> The simple consumer must get told of a leader election and this is nothing
> to do with rebalance, is it so?
>
> Thanks,
> rob
>
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, May 17, 2013 7:32 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: only-once consumer groups
> >
> > We spent some time thinking about consolidating the high level and low
> level
> > consumer APIs. It will be great if you can read the wiki and provide
> feedback
> > - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-
> > Design
> >
> > Thanks,
> > Neha
> > On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:
> >
> > > We want to ensure only-once message processing, but we also want the
> > > benefit of rebalancing.  commitOffsets updates all partitions from out
> > > of a connector instance.  We want to commit the offset for just the
> > > partition that delivered a message to the iterator, even if several
> > > fetchers are feeding a thread.  Perhaps the message metadata contains
> > > the partition a msg came from; could you not just update the offset
> > > for that partition if the property "only.once=true" is sent to the
> > > consumer connector?
> > >
> > > Thanks,
> > > rob
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: Re: only-once consumer groups
> > > >
> > > > Can you describe your requirements in a little more detail?
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > is it technically feasible to use an only-once simple consumer
> > > > > within a consumer group?
> > > > >
> > > > > thanks,
> > > > > rob
> > >
> > >
>
>