Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka startup, no broker partitions consumed by consumer thread


Copy link to this message
-
Re: Kafka startup, no broker partitions consumed by consumer thread
Could you try the latest code in the 0.8 branch? We recently fixed a
consumer side issue (KAFKA-1030) that seems related.

Thanks,

Jun

On Thu, Oct 10, 2013 at 6:36 AM, Viktor Kolodrevskiy <
[EMAIL PROTECTED]> wrote:

> On our current project we want to have a single shell script that:
> first starts kafka with zookeeper and second strarts tomcat server
> with our webapp.
> We run kafka and tomcat on single machine, use kafka 0.8
> We set num.partitions=4 in server.properties.
> In webapp we start 4 threads that do
> kafka.consumer.Consumer.createJavaConsumerConnector to create consumer
> connectors.
>
> The logic of script is following:
> 1. run zookeeper-server-start.sh and wait until zk process
> appears(with linux pgrep command).
> 2. run kafka-server-start.sh and wait until kafka process appears.
> 3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize
> 4. start tomcat
>
> If timeout in step#3 is smaller we get following warning and consumers
> do not receive messages:
> WARN  [kafka.consumer.ZookeeperConsumerConnector]
> [test-1381405301679-580f9706], No broker partitions consumed by
> consumer thread test-1381405301679-580f9706-3 for topic test-topic
>
> Is there a way to avoid this hardcoded timeout? How to check that
> partitions are initialized?
>
>
> Thanks,
> Viktor
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB