I sincerely apologize if this has been covered before, I haven't quite found a similar situation.
We are using Kafka 0.7.2 in production, and we are using the ZK high level Scala consumer. However, we find the ZK consumer very unstable. It would work for one or two weeks, then suddenly it would complain about ZK nodes disappearing, and one consumer would die, then another, then another, until our pipeline is no longer pulling any data. There are multiple NullPointerExceptions, and other problems. We can restart it, but it does not stay up predictably.
On the other hand, I have a simple app which I wrote using the simple consumer to mirror select partitions (will blog about this later) and it just works flawlessly.
So we are faced with a dilemma to get back on track: 1) Use SimpleConsumer, and write our own balancing code (but honestly our boxes almost never go down, compared to the rate of ZK mishaps) 2) Upgrade to Kafka 0.8 and hope that that resolves the issue.
There seem to be so many improvements in 0.8 that that seems to be the biggest win long-term, so I am wondering if people can comment on: - has anyone tried using 0.8 in production? Is it stable yet? - How much more stable is the ZK consumer in 0.8? - will it be possible to change the offset in the 0.8 consumer? That was the other reason why we wanted to move to SimpleConsumer.
Which version of ZK are you using ? On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext