Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: cluster expansion


Copy link to this message
-
Re: cluster expansion
>> They will occupy 4, 5, 6?

Yes, some of the partitions can occupy 4,5,6.

>> Can I keep the zk cluster running and shut down kafka processes
before I run the partition addition tool?

No. The tool won't run properly if you shut down kafka brokers. I think you
meant shut down kafka consumer processes.
On Mon, Dec 16, 2013 at 11:13 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> For the 1st question, I don't quite get the answer.
> Say I have broker 1,2,3,4,5,6. And the topic A has three partitions on
> broker 1,2,3.
> If I add another 3 partitions for A. They will occupy 4, 5, 6?
>
> One more  question. Does the partition addition tool work
> only with zookeeper? Can I keep the zk cluster running and shut down kafka
> processes
> before I run the partition addition tool? I am asking this question
> because we try to avoid
> any message loss when adding a partition in a production environment.
> Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Monday, December 16, 2013 12:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: cluster expansion
>
> If we have six brokers, and a topic has three partitions on broker 1, 2, 3.
> Now if I add another three partitions. Will they be evenly distributed to
> 1,2,3 or 4,5,6?
>
> Evenly distributed on all six brokers.
>
> If I use reassignment tool in 0.81 with 0.8 broker, will that work and get
> around the bugs?
>
> Your broker also needs to be on 0.8.1 for it to work correctly.
>
>
>
> On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
> > If we have six brokers, and a topic has three partitions on broker 1, 2,
> 3.
> > Now if I add another three partitions. Will they be evenly distributed
> > to
> > 1,2,3 or 4,5,6?
> >
> > One more question.
> > If I use reassignment tool in 0.81 with 0.8 broker, will that work and
> > get around the bugs?
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, December 16, 2013 10:26 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: cluster expansion
> >
> > They will be evenly distributed across the nodes in the cluster.
> >
> > Thanks,
> > Neha
> > On Dec 16, 2013 6:42 AM, "Yu, Libo" <[EMAIL PROTECTED]> wrote:
> >
> > > Assume we have three brokers and a topic already has three partitions.
> > > Now if I add another three partitions to this topic. Where will the
> > > three partitions be located after running the script?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > > Sent: Sunday, December 15, 2013 11:43 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: cluster expansion
> > >
> > > Yes, kafka-add-partitons.sh should be stable in 0.8.0.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Is the kafka-add-partitons.sh tool stable in 0.8.0?
> > > >
> > > >
> > > > On 13 December 2013 19:21, Neha Narkhede <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Partition movement is not an automatic operation in Kafka yet.
> > > > > You need
> > > > to
> > > > > use the partition reassignment tool -
> > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tool
> > > > s# Re plicationtools-6.ReassignPartitionsTool
> > > > > .
> > > > >
> > > > >
> > > > > Also, that feature is stable in 0.8.1.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner
> > > > > <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > No the 6 partitions for each topic will remain on the original
> > > brokers.
> > > > > You
> > > > > > could either reassign some partitions from all topics to the
> > > > > > new
> > > > brokers
> > > > > or
> > > > > > you could add partitions to the new brokers for each topic. In
> > > > > > 0.8.0