Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
Neha Narkhede 2013-05-17, 13:32
We spent some time thinking about consolidating the high level and low
level consumer APIs. It will be great if you can read the wiki and provide
feedback -
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Thanks,
Neha
On May 16, 2013 10:29 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:

> We want to ensure only-once message processing, but we also want the
> benefit
> of rebalancing.  commitOffsets updates all partitions from out of a
> connector instance.  We want to commit the offset for just the partition
> that delivered a message to the iterator, even if several fetchers are
> feeding a thread.  Perhaps the message metadata contains the partition a
> msg
> came from; could you not just update the offset for that partition if the
> property "only.once=true" is sent to the consumer connector?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, May 16, 2013 10:03 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: only-once consumer groups
> >
> > Can you describe your requirements in a little more detail?
> >
> > Thanks,
> > Neha
> > On May 16, 2013 6:11 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> > wrote:
> >
> > > is it technically feasible to use an only-once simple consumer within
> > > a consumer group?
> > >
> > > thanks,
> > > rob
>
>