On our current project we want to have a single shell script that: first starts kafka with zookeeper and second strarts tomcat server with our webapp. We run kafka and tomcat on single machine, use kafka 0.8 We set num.partitions=4 in server.properties. In webapp we start 4 threads that do kafka.consumer.Consumer.createJavaConsumerConnector to create consumer connectors.
The logic of script is following: 1. run zookeeper-server-start.sh and wait until zk process appears(with linux pgrep command). 2. run kafka-server-start.sh and wait until kafka process appears. 3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize 4. start tomcat
If timeout in step#3 is smaller we get following warning and consumers do not receive messages: WARN [kafka.consumer.ZookeeperConsumerConnector] [test-1381405301679-580f9706], No broker partitions consumed by consumer thread test-1381405301679-580f9706-3 for topic test-topic
Is there a way to avoid this hardcoded timeout? How to check that partitions are initialized? Thanks, Viktor
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext