Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> automatic producer balancing over brokers

Samuel García Martínez 2012-12-28, 12:29
Copy link to this message
Re: automatic producer balancing over brokers
This is a known bug in Kafka 0.7.x. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. For now, you have to manually create the topic on the new brokers
(e.g., by sending some data to the new broker directly).



On Fri, Dec 28, 2012 at 4:29 AM, Samuel García Martínez <

> Hi! I'm playing with kafka 0.7.2 using the following setup:
> 3 zk nodes ensemble
> 2 brokers:
>    * num_partitions:3
>    * topic.partition.count.map=test-topic:5
> My producer connects to brokers using "zk.connect" property. When the
> producer sends messages to the "test-topic" topic, the partitions are
> created on both brokers and messages are distributed across all
> broker-partition available (not evenly), however, when producer sends
> messages to another topic (not configured in topic.partition.count.map)
> only are created in one server (and all newly created topics are on the
> same server)
> The main question is, is there any way to distribute messages across all
> servers using default partitioner and zk configuration?
> Thanks for your help! :)
> --
> Regards
> Samuel García.