In our environment we use currently use Kafka 0.7.1.
The core features I am looking for in a client are this:
1. Provide confirmation of produce requests (or notification of disconnection during requests). 2. Uses asynchronous IO so that: A. Multiple ops can be queued/in-flight at once. B. Fetch #2 can be in-flight while fetch #1 is being processed (without requiring an extra thread per consumer) 3. Allow precise control of offsets (like the SimpleConsumer).
I have been working on a version of the Kafka client that meets those requirements. However, I recall hearing talk of rewriting the client. Did this happen as part of 0.8.x? If so, how different is it from the 0.7.x clients, and how well would it support the requirements listed above?
Produce requests in 0.7 are fire-and-forget since there is no acknowledgements being returned by the broker. 0.8 fixes this by the optional use of required.acks.
I dont know if a non-JVM client is relevant for your needs, but librdkafka implements all of your listed requirements for 0.8:
1. Message delivery reports (success or specific error) per message. 2.A Prodcuer is fully async, requests are queued locally and sent in batches to the broker. Multiple requests are usually in-flight at any given time. 2.B Consumer is fully async. The consumer strives to keep at least queued.min.messages in the local queue of fetched message ready for the application to consume. 3. A bit vague, but the application may state what offset to start consuming from.
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext