Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka startup, no broker partitions consumed by consumer thread


Copy link to this message
-
Re: Kafka startup, no broker partitions consumed by consumer thread
Jun Rao 2013-10-10, 14:46
Could you try the latest code in the 0.8 branch? We recently fixed a
consumer side issue (KAFKA-1030) that seems related.

Thanks,

Jun

On Thu, Oct 10, 2013 at 6:36 AM, Viktor Kolodrevskiy <
[EMAIL PROTECTED]> wrote:

> On our current project we want to have a single shell script that:
> first starts kafka with zookeeper and second strarts tomcat server
> with our webapp.
> We run kafka and tomcat on single machine, use kafka 0.8
> We set num.partitions=4 in server.properties.
> In webapp we start 4 threads that do
> kafka.consumer.Consumer.createJavaConsumerConnector to create consumer
> connectors.
>
> The logic of script is following:
> 1. run zookeeper-server-start.sh and wait until zk process
> appears(with linux pgrep command).
> 2. run kafka-server-start.sh and wait until kafka process appears.
> 3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize
> 4. start tomcat
>
> If timeout in step#3 is smaller we get following warning and consumers
> do not receive messages:
> WARN  [kafka.consumer.ZookeeperConsumerConnector]
> [test-1381405301679-580f9706], No broker partitions consumed by
> consumer thread test-1381405301679-580f9706-3 for topic test-topic
>
> Is there a way to avoid this hardcoded timeout? How to check that
> partitions are initialized?
>
>
> Thanks,
> Viktor
>