We want to ensure only-once message processing, but we also want the benefit of rebalancing. commitOffsets updates all partitions from out of a connector instance. We want to commit the offset for just the partition that delivered a message to the iterator, even if several fetchers are feeding a thread. Perhaps the message metadata contains the partition a msg came from; could you not just update the offset for that partition if the property "only.once=true" is sent to the consumer connector?
Sorry for being unclear Neha. I meant that I had forgotten that the introduction of replicas is happening in 0.8 and I was confusing the two.
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext