Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Not balancing across multiple brokers


+
Tom Brown 2013-04-12, 19:17
+
Neha Narkhede 2013-04-12, 21:32
+
Tom Brown 2013-04-12, 23:03
Copy link to this message
-
Re: Not balancing across multiple brokers
Tom,

I think you maybe hitting
https://issues.apache.org/jira/browse/KAFKA-278. A workaround is to
create the topic-partition directories on the newly added broker and
restart the broker. After that the zookeeper based producer will load
balance correctly.

Thanks,
Neha

On Fri, Apr 12, 2013 at 4:02 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
> In the producer config, we use the zk connect string:
> zk001,zk002,zk003/kafka.
>
> Both brokers have registered themselves with zookeeper. Because only the
> first broker has ever received any writes, only the first broker is
> registered for the topic in question.
>
> --Tom
>
>
> On Fri, Apr 12, 2013 at 3:32 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Do you use a VIP or zookeeper for producer side load balancing ? In
>> other words, what are the values you override for "broker.list" and
>> "zk.connect" in the producer config ?
>>
>> Thanks,
>> Neha
>>
>> On Fri, Apr 12, 2013 at 12:16 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
>> > We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
>> > topic has 2 partitions per server. We have a two processes that that
>> write
>> > to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
>> >
>> > The problem is that the first process only writes to the first broker.
>> The
>> > second process (using the exact same code to perform the write)
>> > successfully writes to both brokers.
>> >
>> > How can I identify the cause of the imbalance in the first process? How
>> > does the Producer decide which broker is the recipient of each message?
>> >
>> > Thanks!
>> >
>> > --Tom
>>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB