Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?


Copy link to this message
-
Re: Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?
You are probably hitting https://issues.apache.org/jira/browse/KAFKA-278.
Can you please try the workaround mentioned in the JIRA description?

Thanks,
Neha
On Tue, Jun 4, 2013 at 4:21 PM, Brett Hoerner <[EMAIL PROTECTED]>wrote:

> (version 0.7.2)
>
> For some reason, my producers are only picking up the partition on 1 of my
> 2 brokers. I've been digging through the code, and I don't see any issues
> given the state of my ZK nodes. The producer never seems to locate a
> partition on Broker0, even though ZK clearly states that it has 1 (just
> like Broker1 does).
>
> The result of this is that the producer works, but only sends data to one
> Broker.
>
> Has anyone seen something like this before? I'm stumped. Thanks.
>
> # zk information follows:
> [zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids
> [1, 0]
>
> [zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0
> 10.10.71.113-1365733477001:10.10.71.113:9092
>
> [zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1
> 10.10.150.16-1369236663861:10.10.150.16:9092
>
> [zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test
> [1, 0]
>
> # this is the most confusing one
> [zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0
> 1
>
> [zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1
> 1
>
> # kafka producer ZK information DEBUG log, as you can see it finds 0
> partitions on Broker0:
> Broker Topic Path => /brokers/topics
> DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo:
> Broker ids and # of partitions on each for topic: test = ArrayBuffer((0,0),
> (1,1))
> DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo:
> Sorted list of broker ids and partition ids on each for topic: test =
> TreeSet(1-0)
> DEBUG [2013-06-04 23:14:30,819]
> kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> [BrokerTopicsListener] Creating broker topics listener to watch the
> following paths -
> /broker/topics, /broker/topics/topic, /broker/ids
> DEBUG [2013-06-04 23:14:30,823]
> kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> [BrokerTopicsListener] Initialized this broker topics listener with initial
> mapping of broker id to partition id per topic with Map(test ->
> TreeSet(1-0))
> DEBUG [2013-06-04 23:14:30,904] kafka.producer.ZKBrokerPartitionInfo:
> Registering listener on path: /brokers/topics/test
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB