Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Dynamic broker discovery not working for me


Copy link to this message
-
Dynamic broker discovery not working for me
navneet sharma 2012-04-18, 09:00
Hi All,

I was trying the following scenario:
1) Start zookeeper
2) Start server 1 ie broker 1. It will connect with zookeeper
3) Start Producer stand alone java program. This will push string messages
read from a file line-by-line. Before pushing it will divide the messages
into 3 topics.
4) Start Consumer stand alone java program. This is a set of 3 consumers
each dedicated to 3 different topics.

After this, :
5) I started another broker server on a different port.
6) But this was never discovered by the producer and it kept on pushing
everything to the first broker only

I could see this on producer logs:::::
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Getting the number of
broker partitions registered for topic: cartTopic
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Broker partitions
registered for topic: cartTopic = List(1-0)
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Sending message to
broker 127.0.1.1:9095 on partition 0
13:42:43,227 [main] DEBUG kafka.producer.ProducerPool  - Fetching sync
producer for broker id: 1
13:42:43,227 [main] DEBUG kafka.message.ByteBufferMessageSet  - makeNext()
in deepIterator: innerDone = true
13:42:43,228 [main] DEBUG kafka.message.ByteBufferMessageSet  - Message is
uncompressed. Valid byte count = 0
13:42:43,228 [main] DEBUG kafka.message.ByteBufferMessageSet  - makeNext()
in deepIterator: innerDone = true
13:42:43,228 [main] DEBUG kafka.producer.ProducerPool  - Sending message to
broker 1

Infact i suspected that producer may sync up with zookeeper only at run
time. So, with zookeeper and both brokers up, i re-ran producer and
consumer but it gave me same result.

Am i missing anything?

Thanks,
Navneet Sharma