Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?


+
Brett Hoerner 2013-06-04, 23:22
+
Neha Narkhede 2013-06-04, 23:24
Copy link to this message
-
Re: Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?
As an update, this continues to affect us.

First I'd like to note ways in which my issues seems different than
KAFKA-278,

* I did not add a new broker or a new topic, this topic has been in use on
two existing brokers for months
* The topic definitely exists on both brokers. The topic/data directory
exists on both, and as noted above both brokers even show it in ZK

That said, I went ahead and did the "work around" from the ticket, which in
my case basically means "restart the brokers" (because the topic/data
directory already exists).

After bouncing both brokers I *did* get data to both brokers for a while.
I'm not yet sure if this only lasts until I have to restart my *producers*
(as I've had to update them a bit lately), but that is my current guess.

When I start a producer now (both brokers up, data looks exactly like ZK in
original post), I get output like this:

2013-06-13_15:46:11.64496 Broker Topic Path => /brokers/topics
2013-06-13_15:46:11.99990 15:46:11.999 [run-main] INFO
kafka.producer.ProducerPool - Creating async producer for broker id = 1 at
10.10.150.16:9092
2013-06-13_15:46:12.00109 15:46:12.001 [run-main] INFO
kafka.producer.ProducerPool - Creating async producer for broker id = 0 at
10.10.71.113:9092
2013-06-13_15:46:16.77956 15:46:16.779 [ProducerSendThread-1375847990]
INFO  kafka.producer.SyncProducer - Connected to 10.10.150.16:9092 for
producing

The last line repeats as the SyncProducer does its periodic reconnect
thing, but note that it is *always* broker 1 at 10.10.150.16:9092, even
though it seems like broker 0 is "seen."

Thanks for your help!

On Tue, Jun 4, 2013 at 6:24 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> You are probably hitting https://issues.apache.org/jira/browse/KAFKA-278.
> Can you please try the workaround mentioned in the JIRA description?
>
> Thanks,
> Neha
>
>
> On Tue, Jun 4, 2013 at 4:21 PM, Brett Hoerner <[EMAIL PROTECTED]
> >wrote:
>
> > (version 0.7.2)
> >
> > For some reason, my producers are only picking up the partition on 1 of
> my
> > 2 brokers. I've been digging through the code, and I don't see any issues
> > given the state of my ZK nodes. The producer never seems to locate a
> > partition on Broker0, even though ZK clearly states that it has 1 (just
> > like Broker1 does).
> >
> > The result of this is that the producer works, but only sends data to one
> > Broker.
> >
> > Has anyone seen something like this before? I'm stumped. Thanks.
> >
> > # zk information follows:
> > [zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids
> > [1, 0]
> >
> > [zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0
> > 10.10.71.113-1365733477001:10.10.71.113:9092
> >
> > [zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1
> > 10.10.150.16-1369236663861:10.10.150.16:9092
> >
> > [zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test
> > [1, 0]
> >
> > # this is the most confusing one
> > [zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0
> > 1
> >
> > [zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1
> > 1
> >
> > # kafka producer ZK information DEBUG log, as you can see it finds 0
> > partitions on Broker0:
> > Broker Topic Path => /brokers/topics
> > DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo:
> > Broker ids and # of partitions on each for topic: test =
> ArrayBuffer((0,0),
> > (1,1))
> > DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo:
> > Sorted list of broker ids and partition ids on each for topic: test =
> > TreeSet(1-0)
> > DEBUG [2013-06-04 23:14:30,819]
> > kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> > [BrokerTopicsListener] Creating broker topics listener to watch the
> > following paths -
> > /broker/topics, /broker/topics/topic, /broker/ids
> > DEBUG [2013-06-04 23:14:30,823]
> > kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener:
> > [BrokerTopicsListener] Initialized this broker topics listener with
> initial
> > mapping of broker id to partition id per topic with Map(test ->

 
+
Brett Hoerner 2013-06-13, 19:39