Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Dynamic broker discovery not working for me


Copy link to this message
-
Dynamic broker discovery not working for me
Hi All,

I was trying the following scenario:
1) Start zookeeper
2) Start server 1 ie broker 1. It will connect with zookeeper
3) Start Producer stand alone java program. This will push string messages
read from a file line-by-line. Before pushing it will divide the messages
into 3 topics.
4) Start Consumer stand alone java program. This is a set of 3 consumers
each dedicated to 3 different topics.

After this, :
5) I started another broker server on a different port.
6) But this was never discovered by the producer and it kept on pushing
everything to the first broker only

I could see this on producer logs:::::
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Getting the number of
broker partitions registered for topic: cartTopic
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Broker partitions
registered for topic: cartTopic = List(1-0)
13:42:43,227 [main] DEBUG kafka.producer.Producer  - Sending message to
broker 127.0.1.1:9095 on partition 0
13:42:43,227 [main] DEBUG kafka.producer.ProducerPool  - Fetching sync
producer for broker id: 1
13:42:43,227 [main] DEBUG kafka.message.ByteBufferMessageSet  - makeNext()
in deepIterator: innerDone = true
13:42:43,228 [main] DEBUG kafka.message.ByteBufferMessageSet  - Message is
uncompressed. Valid byte count = 0
13:42:43,228 [main] DEBUG kafka.message.ByteBufferMessageSet  - makeNext()
in deepIterator: innerDone = true
13:42:43,228 [main] DEBUG kafka.producer.ProducerPool  - Sending message to
broker 1

Infact i suspected that producer may sync up with zookeeper only at run
time. So, with zookeeper and both brokers up, i re-ran producer and
consumer but it gave me same result.

Am i missing anything?

Thanks,
Navneet Sharma
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB