Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Not balancing across multiple brokers


Copy link to this message
-
Re: Not balancing across multiple brokers
Neha Narkhede 2013-04-16, 16:10
Tom,

I think you maybe hitting
https://issues.apache.org/jira/browse/KAFKA-278. A workaround is to
create the topic-partition directories on the newly added broker and
restart the broker. After that the zookeeper based producer will load
balance correctly.

Thanks,
Neha

On Fri, Apr 12, 2013 at 4:02 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
> In the producer config, we use the zk connect string:
> zk001,zk002,zk003/kafka.
>
> Both brokers have registered themselves with zookeeper. Because only the
> first broker has ever received any writes, only the first broker is
> registered for the topic in question.
>
> --Tom
>
>
> On Fri, Apr 12, 2013 at 3:32 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Do you use a VIP or zookeeper for producer side load balancing ? In
>> other words, what are the values you override for "broker.list" and
>> "zk.connect" in the producer config ?
>>
>> Thanks,
>> Neha
>>
>> On Fri, Apr 12, 2013 at 12:16 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
>> > We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
>> > topic has 2 partitions per server. We have a two processes that that
>> write
>> > to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
>> >
>> > The problem is that the first process only writes to the first broker.
>> The
>> > second process (using the exact same code to perform the write)
>> > successfully writes to both brokers.
>> >
>> > How can I identify the cause of the imbalance in the first process? How
>> > does the Producer decide which broker is the recipient of each message?
>> >
>> > Thanks!
>> >
>> > --Tom
>>