Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?


Copy link to this message
-
Re: Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?
You know what, it's likely this is all because I'm running a bad fork of
Kafka 0.7.2 for Scala 2.10 (on the producers/consumers) since that's the
version we've standardized on.

Behavior in 2.9.2 with the official Kafka 0.7.2 release seems much more
normal -- I'm working on downgrading all our clients and I'll report back.

Sorry for the spam.
On Thu, Jun 13, 2013 at 12:31 PM, Brett Hoerner <[EMAIL PROTECTED]>wrote:

> As an update, this continues to affect us.
>
> First I'd like to note ways in which my issues seems different than
> KAFKA-278,
>
> * I did not add a new broker or a new topic, this topic has been in use on
> two existing brokers for months
> * The topic definitely exists on both brokers. The topic/data directory
> exists on both, and as noted above both brokers even show it in ZK
>
> That said, I went ahead and did the "work around" from the ticket, which
> in my case basically means "restart the brokers" (because the topic/data
> directory already exists).
>
> After bouncing both brokers I *did* get data to both brokers for a while.
> I'm not yet sure if this only lasts until I have to restart my *producers*
> (as I've had to update them a bit lately), but that is my current guess.
>
> When I start a producer now (both brokers up, data looks exactly like ZK
> in original post), I get output like this:
>
> 2013-06-13_15:46:11.64496 Broker Topic Path => /brokers/topics
> 2013-06-13_15:46:11.99990 15:46:11.999 [run-main] INFO
> kafka.producer.ProducerPool - Creating async producer for broker id = 1 at
> 10.10.150.16:9092
> 2013-06-13_15:46:12.00109 15:46:12.001 [run-main] INFO
> kafka.producer.ProducerPool - Creating async producer for broker id = 0 at
> 10.10.71.113:9092
> 2013-06-13_15:46:16.77956 15:46:16.779 [ProducerSendThread-1375847990]
> INFO  kafka.producer.SyncProducer - Connected to 10.10.150.16:9092 for
> producing
>
> The last line repeats as the SyncProducer does its periodic reconnect
> thing, but note that it is *always* broker 1 at 10.10.150.16:9092, even
> though it seems like broker 0 is "seen."
>
> Thanks for your help!
>
>
>
> On Tue, Jun 4, 2013 at 6:24 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> You are probably hitting https://issues.apache.org/jira/browse/KAFKA-278.
>> Can you please try the workaround mentioned in the JIRA description?
>>
>> Thanks,
>> Neha
>>
>>
>> On Tue, Jun 4, 2013 at 4:21 PM, Brett Hoerner <[EMAIL PROTECTED]
>> >wrote:
>>
>> > (version 0.7.2)
>> >
>> > For some reason, my producers are only picking up the partition on 1 of
>> my
>> > 2 brokers. I've been digging through the code, and I don't see any
>> issues
>> > given the state of my ZK nodes. The producer never seems to locate a
>> > partition on Broker0, even though ZK clearly states that it has 1 (just
>> > like Broker1 does).
>> >
>> > The result of this is that the producer works, but only sends data to
>> one
>> > Broker.
>> >
>> > Has anyone seen something like this before? I'm stumped. Thanks.
>> >
>> > # zk information follows:
>> > [zk: localhost:2181(CONNECTED) 5] ls /kafka/brokers/ids
>> > [1, 0]
>> >
>> > [zk: localhost:2181(CONNECTED) 6] get /kafka/brokers/ids/0
>> > 10.10.71.113-1365733477001:10.10.71.113:9092
>> >
>> > [zk: localhost:2181(CONNECTED) 7] get /kafka/brokers/ids/1
>> > 10.10.150.16-1369236663861:10.10.150.16:9092
>> >
>> > [zk: localhost:2181(CONNECTED) 10] ls /kafka/brokers/topics/test
>> > [1, 0]
>> >
>> > # this is the most confusing one
>> > [zk: localhost:2181(CONNECTED) 11] get /kafka/brokers/topics/test/0
>> > 1
>> >
>> > [zk: localhost:2181(CONNECTED) 12] get /kafka/brokers/topics/test/1
>> > 1
>> >
>> > # kafka producer ZK information DEBUG log, as you can see it finds 0
>> > partitions on Broker0:
>> > Broker Topic Path => /brokers/topics
>> > DEBUG [2013-06-04 23:14:30,689] kafka.producer.ZKBrokerPartitionInfo:
>> > Broker ids and # of partitions on each for topic: test =
>> ArrayBuffer((0,0),
>> > (1,1))
>> > DEBUG [2013-06-04 23:14:30,690] kafka.producer.ZKBrokerPartitionInfo: