Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Duplicate messages after new consumer introduction


Copy link to this message
-
Duplicate messages after new consumer introduction
Hi,

I tried a scenario wherein:
1) i had 1 producer and 3 consumers subscribed for a topic - "cartTopic",
all in same group.
2) Now, when everything is executing, i introduced another consumer for the
same topic and in the same group. So, overall there are 4 consumers.
3) Ofcourse, it triggered re-balancing.

But then final result is that few messages are duplicated.
In my example run, producer sent 800,000 records, but consumer received
801,448 records.
I am using log4j to generate the output file.

Is there any reasons for duplicacy?

Thanks,
Navneet Sharma
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB