Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Unpacking "strong durability and fault-tolerance guarantees"


Copy link to this message
-
Re: Unpacking "strong durability and fault-tolerance guarantees"
Thanks Jay, this is helpful information. Controlling when offsets are committed is possible only with the SimpleConsumer, correct? I believe that the ConsumerConnector (the high level consumer) commits the offsets automatically. It would be nice to have a hook into it that I can override to let it know when I finished processing a message so it can commit an offset. Is there an improvement request submitted for this already?

Thanks!

F.

On Jul 7, 2013, at 2:35 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Consumer:
> The consumers position is controlled using a saved "offset" that marks its
> position in the topic/partition it is reading. This position is
> periodically updated. If you update the saved offset before processing
> messages you have the possibility of message loss if your consumer crashes
> before processing the messages. If you update the offset after processing
> the messages you have the possibility of duplicate messages when your
> consumer restarts as it will reprocess a few message it has already seen.
> You can control when the position is saved by calling commit().
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB