Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> automatic producer balancing over brokers


Copy link to this message
-
Re: automatic producer balancing over brokers
This is a known bug in Kafka 0.7.x. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. For now, you have to manually create the topic on the new brokers
(e.g., by sending some data to the new broker directly).

Thanks,

Jun

On Fri, Dec 28, 2012 at 4:29 AM, Samuel García Martínez <
[EMAIL PROTECTED]> wrote:

> Hi! I'm playing with kafka 0.7.2 using the following setup:
>
> 3 zk nodes ensemble
> 2 brokers:
>    * num_partitions:3
>    * topic.partition.count.map=test-topic:5
>
> My producer connects to brokers using "zk.connect" property. When the
> producer sends messages to the "test-topic" topic, the partitions are
> created on both brokers and messages are distributed across all
> broker-partition available (not evenly), however, when producer sends
> messages to another topic (not configured in topic.partition.count.map)
> only are created in one server (and all newly created topics are on the
> same server)
>
> The main question is, is there any way to distribute messages across all
> servers using default partitioner and zk configuration?
>
> Thanks for your help! :)
> --
> Regards
> Samuel García.
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB