Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Duplicate messages after new consumer introduction


Copy link to this message
-
Re: Duplicate messages after new consumer introduction
I downloaded the tar from the download link provided in quickstart page.
Almost more than a month back.

I trunk maintaining different code than the tar?

Can number of partitions cause this problem, beacuse i am using 2
partitions on each of the two brokers.?

Thanks,
Navneet Sharma

On Wed, May 16, 2012 at 10:00 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Technically this is the guarantee we provide--at least once delivery.
> It is very expensive to completely eliminate this possibility in the
> general case as you need to co-ordinate any state changes the consumer
> makes with committing the offset that marks the position. But we have
> improved the common cases for normal rebalancing so if you are using
> trunk the only time this would happen is when there is a hard crash of
> a process.
>
> -Jay
>
> On Wed, May 16, 2012 at 2:41 AM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I tried a scenario wherein:
> > 1) i had 1 producer and 3 consumers subscribed for a topic - "cartTopic",
> > all in same group.
> > 2) Now, when everything is executing, i introduced another consumer for
> the
> > same topic and in the same group. So, overall there are 4 consumers.
> > 3) Ofcourse, it triggered re-balancing.
> >
> > But then final result is that few messages are duplicated.
> > In my example run, producer sent 800,000 records, but consumer received
> > 801,448 records.
> > I am using log4j to generate the output file.
> >
> > Is there any reasons for duplicacy?
> >
> > Thanks,
> > Navneet Sharma
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB