Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How does a new broker decide the partitions of the topic?


Copy link to this message
-
Re: How does a new broker decide the partitions of the topic?
Thanks yonghui.
That makes sense.
On Mon, Dec 31, 2012 at 1:45 AM, 永辉 赵 <[EMAIL PROTECTED]> wrote:

> Partition name registered in zk contains broker id and sequence(partition)
> id, such as "0-0", "0-1", ...
>
> Thanks,
> Yonghui
>
>
>
>
>
> On 12-12-30 下午11:44, "王国栋" <[EMAIL PROTECTED]> wrote:
>
> >Thanks Yonghui.
> >
> >But how does the new broker decide the partition id of the new topic? Or
> >each partion id contains a broker id and a sequence id?
> >
> >
> >
> >
> >On Fri, Dec 28, 2012 at 12:08 PM, 永辉 赵 <[EMAIL PROTECTED]> wrote:
> >
> >> Here is my understanding, but may be wrong.
> >>
> >> Every topic in one broker has num.partitions partitions if we only set
> >> num.partitions
> >>
> >> When a new broker is added, if there is topic data folder under
> >>kafka-logs
> >> folder,
> >> then the broker will register num.partitions of this topic in zk, else
> >>the
> >> broker doesn't know which topic should be registered.
> >> If no brokers register the topic(topic data folder is not existed in any
> >> broker), consumer will fall in bootstrap, assuming each broker has 1
> >> partition.
> >> The broker will register num.partitions of this topic in zk, after it
> >> receive this topic message .
> >>
> >> If only a few brokers haven't register the topic in zk, I think these
> >> brokers may not actually work for this topic.
> >>
> >> Please correct me if I am wrong.
> >>
> >> Thanks,
> >> Yonghui
> >>
> >>
> >>
> >>
> >>
> >> On 12-12-28 上午10:30, "王国栋" <[EMAIL PROTECTED]> wrote:
> >>
> >> >Hi ,
> >> >
> >> >I am reading the design doc for Kafka. But  I can not find the
> >>algorithm
> >> >which illustrates such scenario. When a new broker is registered in
> >> >zookeeper, how do all the brokers negotiates with all the others? How
> >>does
> >> >a new broker decide which partitions should be stored ?
> >> >
> >> >By the way, I find num.partitions in server configure, but I can not
> >>find
> >> >any parameter defining the topics of this broker.
> >> >
> >> >Could you tell me how to configure brokers in the following example.
> >>For
> >> >example, I want to set "broker1" to store "topic1" and "topic2", each
> >> >topic
> >> >has 3 partitions?
> >> >I think setting topic.partition.count.map as "topic1:3, topic2:3"
> >>should
> >> >works. Is there any simpler configuration?
> >> >
> >> >
> >> >Thanks.
> >> >
> >> >--
> >> >Guodong Wang(王国栋)
> >> >Email:[EMAIL PROTECTED]
> >>
> >>
> >>
> >
> >
> >--
> >Guodong Wang
> >王国栋
>
>
>
--
Guodong Wang
王国栋