Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
We want to ensure only-once message processing, but we also want the benefit
of rebalancing.  commitOffsets updates all partitions from out of a
connector instance.  We want to commit the offset for just the partition
that delivered a message to the iterator, even if several fetchers are
feeding a thread.  Perhaps the message metadata contains the partition a msg
came from; could you not just update the offset for that partition if the
property "only.once=true" is sent to the consumer connector?

Thanks,
rob

> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 16, 2013 10:03 PM
> To: [EMAIL PROTECTED]
> Subject: Re: only-once consumer groups
>
> Can you describe your requirements in a little more detail?
>
> Thanks,
> Neha
> On May 16, 2013 6:11 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > is it technically feasible to use an only-once simple consumer within
> > a consumer group?
> >
> > thanks,
> > rob