Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - ZK session kill makes high level consumer replay


Copy link to this message
-
Re: ZK session kill makes high level consumer replay
Neha Narkhede 2014-03-28, 03:33
When I call consumer.commitOffsets(); before killing session, unit test
succeeded. This problem would happen only with autoCommit enabled

That seems expected. If you call commitOffsets() explicitly before
simulating a GC pause on the consumer, there will be no duplicates since
the next consumer instance that picks up the same partitions will start
reading from the last checkpointed offset. On the other hand, autoCommit
will only commit at a certain interval. So if you decide to pause the
consumer between 2 intervals, then it will replay data since the last
interval.

Thanks,
Neha
On Thu, Mar 27, 2014 at 4:21 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]> wrote: