Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?


Copy link to this message
-
Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?
(version 0.7.2)

For some reason, my producers are only picking up the partition on 1 of my
2 brokers. I've been digging through the code, and I don't see any issues
given the state of my ZK nodes. The producer never seems to locate a
partition on Broker0, even though ZK clearly states that it has 1 (just
like Broker1 does).

The result of this is that the producer works, but only sends data to one
Broker.

Has anyone seen something like this before? I'm stumped. Thanks.

# zk information follows:
[zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids
[1, 0]

[zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0
10.10.71.113-1365733477001:10.10.71.113:9092

[zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1
10.10.150.16-1369236663861:10.10.150.16:9092

[zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test
[1, 0]

# this is the most confusing one
[zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0
1

[zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1
1

# kafka producer ZK information DEBUG log, as you can see it finds 0
partitions on Broker0:
Broker Topic Path => /brokers/topics
DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo:
Broker ids and # of partitions on each for topic: test = ArrayBuffer((0,0),
(1,1))
DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo:
Sorted list of broker ids and partition ids on each for topic: test =
TreeSet(1-0)
DEBUG [2013-06-04 23:14:30,819]
kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
[BrokerTopicsListener] Creating broker topics listener to watch the
following paths -
/broker/topics, /broker/topics/topic, /broker/ids
DEBUG [2013-06-04 23:14:30,823]
kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
[BrokerTopicsListener] Initialized this broker topics listener with initial
mapping of broker id to partition id per topic with Map(test ->
TreeSet(1-0))
DEBUG [2013-06-04 23:14:30,904] kafka.producer.ZKBrokerPartitionInfo:
Registering listener on path: /brokers/topics/test

 
+
Neha Narkhede 2013-06-04, 23:24
+
Brett Hoerner 2013-06-13, 17:32
+
Brett Hoerner 2013-06-13, 19:39