I know this is a really old thread, but it looked like the only pertinent one that came up when searching for ‘exactly once’ in the archives. I just want to confirm my understanding of the 0.8 version in that it still doesn’t completely support exactly once semantics. With the producer configured in sync mode and quorum commits there are still some edge case failure modes where the producer won’t receive the ack and resends the message(s). I think I read that the consumers don’t see uncommitted messages in the log, but I don’t think that addresses this producer case. Please correct me if I am missing something here. Don’t get me wrong we are very thankful for the 0.8 features. It offers by far the best message delivery guarantees out of the products we evaluated like Rabbit and ActiveMQ. We attempt to make are downstream consumer processes idempotent to mitigate this edge case, but it isn’t always feasible. Also the suggestion by Milind in this thread of using Storm for exactly once guarantees makes a lot of sense. Trident State seems to address this very issue ( https://github.com/nathanmarz/storm/wiki/Trident-state) so we could just have it mediate our topics that required exactly once. -Jonathan
On Sat, Nov 3, 2012 at 1:53 PM, Milind Parikh <[EMAIL PROTECTED]>wrote:
With a single writer, the producer can achieve exact once write. If a send request fails, the producer first checks the end of the log to see if the previous write succeeded or not. The producer will only resend if the previous write fails.
To do this, the producer needs the offset of appended messages. In 0.8, such offsets are not returned in our high level producer API yet. We plan to extend our producer API post 0.8 to expose this information.
On Wed, Mar 27, 2013 at 2:41 PM, Jonathan Hodges <[EMAIL PROTECTED]> wrote:
Awesome! Thanks for confirmation and continued great work on Kafka! On Thu, Mar 28, 2013 at 9:22 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext