For some reason, my producers are only picking up the partition on 1 of my 2 brokers. I've been digging through the code, and I don't see any issues given the state of my ZK nodes. The producer never seems to locate a partition on Broker0, even though ZK clearly states that it has 1 (just like Broker1 does).
The result of this is that the producer works, but only sends data to one Broker.
Has anyone seen something like this before? I'm stumped. Thanks.
# zk information follows: [zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids [1, 0]
[zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0 10.10.71.113-1365733477001:10.10.71.113:9092
[zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1 10.10.150.16-1369236663861:10.10.150.16:9092
[zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test [1, 0]
# this is the most confusing one [zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0 1
[zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1 1
# kafka producer ZK information DEBUG log, as you can see it finds 0 partitions on Broker0: Broker Topic Path => /brokers/topics DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo: Broker ids and # of partitions on each for topic: test = ArrayBuffer((0,0), (1,1)) DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo: Sorted list of broker ids and partition ids on each for topic: test = TreeSet(1-0) DEBUG [2013-06-04 23:14:30,819] kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener: [BrokerTopicsListener] Creating broker topics listener to watch the following paths - /broker/topics, /broker/topics/topic, /broker/ids DEBUG [2013-06-04 23:14:30,823] kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener: [BrokerTopicsListener] Initialized this broker topics listener with initial mapping of broker id to partition id per topic with Map(test -> TreeSet(1-0)) DEBUG [2013-06-04 23:14:30,904] kafka.producer.ZKBrokerPartitionInfo: Registering listener on path: /brokers/topics/test
First I'd like to note ways in which my issues seems different than KAFKA-278,
* I did not add a new broker or a new topic, this topic has been in use on two existing brokers for months * The topic definitely exists on both brokers. The topic/data directory exists on both, and as noted above both brokers even show it in ZK
That said, I went ahead and did the "work around" from the ticket, which in my case basically means "restart the brokers" (because the topic/data directory already exists).
After bouncing both brokers I *did* get data to both brokers for a while. I'm not yet sure if this only lasts until I have to restart my *producers* (as I've had to update them a bit lately), but that is my current guess.
When I start a producer now (both brokers up, data looks exactly like ZK in original post), I get output like this:
2013-06-13_15:46:11.64496 Broker Topic Path => /brokers/topics 2013-06-13_15:46:11.99990 15:46:11.999 [run-main] INFO kafka.producer.ProducerPool - Creating async producer for broker id = 1 at 10.10.150.16:9092 2013-06-13_15:46:12.00109 15:46:12.001 [run-main] INFO kafka.producer.ProducerPool - Creating async producer for broker id = 0 at 10.10.71.113:9092 2013-06-13_15:46:16.77956 15:46:16.779 [ProducerSendThread-1375847990] INFO kafka.producer.SyncProducer - Connected to 10.10.150.16:9092 for producing
The last line repeats as the SyncProducer does its periodic reconnect thing, but note that it is *always* broker 1 at 10.10.150.16:9092, even though it seems like broker 0 is "seen."
Thanks for your help!
On Tue, Jun 4, 2013 at 6:24 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
You know what, it's likely this is all because I'm running a bad fork of Kafka 0.7.2 for Scala 2.10 (on the producers/consumers) since that's the version we've standardized on.
Behavior in 2.9.2 with the official Kafka 0.7.2 release seems much more normal -- I'm working on downgrading all our clients and I'll report back.
Sorry for the spam. On Thu, Jun 13, 2013 at 12:31 PM, Brett Hoerner <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext