Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How does a new broker decide the partitions of the topic?


Copy link to this message
-
Re: How does a new broker decide the partitions of the topic?
Thanks Yonghui.

But how does the new broker decide the partition id of the new topic? Or
each partion id contains a broker id and a sequence id?
On Fri, Dec 28, 2012 at 12:08 PM, 永辉 赵 <[EMAIL PROTECTED]> wrote:

> Here is my understanding, but may be wrong.
>
> Every topic in one broker has num.partitions partitions if we only set
> num.partitions
>
> When a new broker is added, if there is topic data folder under kafka-logs
> folder,
> then the broker will register num.partitions of this topic in zk, else the
> broker doesn't know which topic should be registered.
> If no brokers register the topic(topic data folder is not existed in any
> broker), consumer will fall in bootstrap, assuming each broker has 1
> partition.
> The broker will register num.partitions of this topic in zk, after it
> receive this topic message .
>
> If only a few brokers haven't register the topic in zk, I think these
> brokers may not actually work for this topic.
>
> Please correct me if I am wrong.
>
> Thanks,
> Yonghui
>
>
>
>
>
> On 12-12-28 上午10:30, "王国栋" <[EMAIL PROTECTED]> wrote:
>
> >Hi ,
> >
> >I am reading the design doc for Kafka. But  I can not find the algorithm
> >which illustrates such scenario. When a new broker is registered in
> >zookeeper, how do all the brokers negotiates with all the others? How does
> >a new broker decide which partitions should be stored ?
> >
> >By the way, I find num.partitions in server configure, but I can not find
> >any parameter defining the topics of this broker.
> >
> >Could you tell me how to configure brokers in the following example. For
> >example, I want to set "broker1" to store "topic1" and "topic2", each
> >topic
> >has 3 partitions?
> >I think setting topic.partition.count.map as "topic1:3, topic2:3" should
> >works. Is there any simpler configuration?
> >
> >
> >Thanks.
> >
> >--
> >Guodong Wang(王国栋)
> >Email:[EMAIL PROTECTED]
>
>
>
--
Guodong Wang
王国栋

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB