Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: only-once consumer groups


Copy link to this message
-
RE: only-once consumer groups
We want to ensure only-once message processing, but we also want the benefit
of rebalancing.  commitOffsets updates all partitions from out of a
connector instance.  We want to commit the offset for just the partition
that delivered a message to the iterator, even if several fetchers are
feeding a thread.  Perhaps the message metadata contains the partition a msg
came from; could you not just update the offset for that partition if the
property "only.once=true" is sent to the consumer connector?

Thanks,
rob

> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 16, 2013 10:03 PM
> To: [EMAIL PROTECTED]
> Subject: Re: only-once consumer groups
>
> Can you describe your requirements in a little more detail?
>
> Thanks,
> Neha
> On May 16, 2013 6:11 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > is it technically feasible to use an only-once simple consumer within
> > a consumer group?
> >
> > thanks,
> > rob
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB