Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - How does a new broker decide the partitions of the topic?


Copy link to this message
-
Re: How does a new broker decide the partitions of the topic?
王国栋 2012-12-30, 15:45
Thanks Yonghui.

But how does the new broker decide the partition id of the new topic? Or
each partion id contains a broker id and a sequence id?
On Fri, Dec 28, 2012 at 12:08 PM, 永辉 赵 <[EMAIL PROTECTED]> wrote:

> Here is my understanding, but may be wrong.
>
> Every topic in one broker has num.partitions partitions if we only set
> num.partitions
>
> When a new broker is added, if there is topic data folder under kafka-logs
> folder,
> then the broker will register num.partitions of this topic in zk, else the
> broker doesn't know which topic should be registered.
> If no brokers register the topic(topic data folder is not existed in any
> broker), consumer will fall in bootstrap, assuming each broker has 1
> partition.
> The broker will register num.partitions of this topic in zk, after it
> receive this topic message .
>
> If only a few brokers haven't register the topic in zk, I think these
> brokers may not actually work for this topic.
>
> Please correct me if I am wrong.
>
> Thanks,
> Yonghui
>
>
>
>
>
> On 12-12-28 上午10:30, "王国栋" <[EMAIL PROTECTED]> wrote:
>
> >Hi ,
> >
> >I am reading the design doc for Kafka. But  I can not find the algorithm
> >which illustrates such scenario. When a new broker is registered in
> >zookeeper, how do all the brokers negotiates with all the others? How does
> >a new broker decide which partitions should be stored ?
> >
> >By the way, I find num.partitions in server configure, but I can not find
> >any parameter defining the topics of this broker.
> >
> >Could you tell me how to configure brokers in the following example. For
> >example, I want to set "broker1" to store "topic1" and "topic2", each
> >topic
> >has 3 partitions?
> >I think setting topic.partition.count.map as "topic1:3, topic2:3" should
> >works. Is there any simpler configuration?
> >
> >
> >Thanks.
> >
> >--
> >Guodong Wang(王国栋)
> >Email:[EMAIL PROTECTED]
>
>
>
--
Guodong Wang
王国栋