Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Duplicate Messages on the Consumer


Copy link to this message
-
Re: Duplicate Messages on the Consumer
Have you actually examined the Kafka files on disk, to make sure those
dupes are really there? Or is this a case of reading the same message
more than once?

Philip

On Thu, Jul 18, 2013 at 8:55 AM, Sybrandy, Casey
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> We recently started seeing duplicate messages appearing at our consumers.  Thankfully, the database is set up so that we don't store the dupes, but it is annoying.  It's not every message, only about 1% of them.  We are running 0.7.0 for the broker with Zookeeper 3.3.4 from Cloudera and 0.7.0 for the producer and consumer.  We tried upgrading the consumer to 0.7.2 to see if that worked, but we're still seeing the dupes.  Do we have to upgrade the broker as well to resolve this?  Is there something we can check to see what's going on because we're not seeing anything unusual in the logs.  I suspected that there may be significant rebalancing, but that does not appear to be the case at all.
>
> Casey Sybrandy
>