Kafka, mail # user - Re: Unpacking "strong durability and fault-tolerance guarantees" - 2013-07-07, 21:36
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
David James 2013-07-07, 15:36
Philip OToole 2013-07-07, 19:51
Copy link to this message
Re: Unpacking "strong durability and fault-tolerance guarantees"
I think there may be some confusion as we significantly strengthened the
guarantees in the latest release (0.8) and since this is pretty recent we
haven't updated all the documentation. Prior to this we did not support

As of 0.8:
1. If your replication factor is 3 you can tolerate 2 failures before you
lose data.
2. This question is a little confusing--there are guarantees for both the
producer and the consumer.
Kafka does have a configurable acknowledgement policy for the producer as
of 0.8 using the request.required.acks setting. If this is set to 0 it does
not wait. If it is set to 1 it waits only for the broker to which the data
was written. If it is set to -1 it waits for all caught up replicas.
The consumers position is controlled using a saved "offset" that marks its
position in the topic/partition it is reading. This position is
periodically updated. If you update the saved offset before processing
messages you have the possibility of message loss if your consumer crashes
before processing the messages. If you update the offset after processing
the messages you have the possibility of duplicate messages when your
consumer restarts as it will reprocess a few message it has already seen.
You can control when the position is saved by calling commit().
On Sun, Jul 7, 2013 at 8:35 AM, David James <[EMAIL PROTECTED]> wrote:
Florin Trofin 2013-07-13, 01:39
Eric Sites 2013-07-13, 02:58
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB