Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka startup, no broker partitions consumed by consumer thread

Copy link to this message
Re: Kafka startup, no broker partitions consumed by consumer thread
Could you try the latest code in the 0.8 branch? We recently fixed a
consumer side issue (KAFKA-1030) that seems related.



On Thu, Oct 10, 2013 at 6:36 AM, Viktor Kolodrevskiy <

> On our current project we want to have a single shell script that:
> first starts kafka with zookeeper and second strarts tomcat server
> with our webapp.
> We run kafka and tomcat on single machine, use kafka 0.8
> We set num.partitions=4 in server.properties.
> In webapp we start 4 threads that do
> kafka.consumer.Consumer.createJavaConsumerConnector to create consumer
> connectors.
> The logic of script is following:
> 1. run zookeeper-server-start.sh and wait until zk process
> appears(with linux pgrep command).
> 2. run kafka-server-start.sh and wait until kafka process appears.
> 3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize
> 4. start tomcat
> If timeout in step#3 is smaller we get following warning and consumers
> do not receive messages:
> WARN  [kafka.consumer.ZookeeperConsumerConnector]
> [test-1381405301679-580f9706], No broker partitions consumed by
> consumer thread test-1381405301679-580f9706-3 for topic test-topic
> Is there a way to avoid this hardcoded timeout? How to check that
> partitions are initialized?
> Thanks,
> Viktor