Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?


Copy link to this message
-
Re: Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?
You are probably hitting https://issues.apache.org/jira/browse/KAFKA-278.
Can you please try the workaround mentioned in the JIRA description?

Thanks,
Neha
On Tue, Jun 4, 2013 at 4:21 PM, Brett Hoerner <[EMAIL PROTECTED]>wrote:

> (version 0.7.2)
>
> For some reason, my producers are only picking up the partition on 1 of my
> 2 brokers. I've been digging through the code, and I don't see any issues
> given the state of my ZK nodes. The producer never seems to locate a
> partition on Broker0, even though ZK clearly states that it has 1 (just
> like Broker1 does).
>
> The result of this is that the producer works, but only sends data to one
> Broker.
>
> Has anyone seen something like this before? I'm stumped. Thanks.
>
> # zk information follows:
> [zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids
> [1, 0]
>
> [zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0
> 10.10.71.113-1365733477001:10.10.71.113:9092
>
> [zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1
> 10.10.150.16-1369236663861:10.10.150.16:9092
>
> [zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test
> [1, 0]
>
> # this is the most confusing one
> [zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0
> 1
>
> [zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1
> 1
>
> # kafka producer ZK information DEBUG log, as you can see it finds 0
> partitions on Broker0:
> Broker Topic Path => /brokers/topics
> DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo:
> Broker ids and # of partitions on each for topic: test = ArrayBuffer((0,0),
> (1,1))
> DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo:
> Sorted list of broker ids and partition ids on each for topic: test =
> TreeSet(1-0)
> DEBUG [2013-06-04 23:14:30,819]
> kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> [BrokerTopicsListener] Creating broker topics listener to watch the
> following paths -
> /broker/topics, /broker/topics/topic, /broker/ids
> DEBUG [2013-06-04 23:14:30,823]
> kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> [BrokerTopicsListener] Initialized this broker topics listener with initial
> mapping of broker id to partition id per topic with Map(test ->
> TreeSet(1-0))
> DEBUG [2013-06-04 23:14:30,904] kafka.producer.ZKBrokerPartitionInfo:
> Registering listener on path: /brokers/topics/test
>