Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: cluster expansion


Copy link to this message
-
Re: cluster expansion
>> They will occupy 4, 5, 6?

Yes, some of the partitions can occupy 4,5,6.

>> Can I keep the zk cluster running and shut down kafka processes
before I run the partition addition tool?

No. The tool won't run properly if you shut down kafka brokers. I think you
meant shut down kafka consumer processes.
On Mon, Dec 16, 2013 at 11:13 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> For the 1st question, I don't quite get the answer.
> Say I have broker 1,2,3,4,5,6. And the topic A has three partitions on
> broker 1,2,3.
> If I add another 3 partitions for A. They will occupy 4, 5, 6?
>
> One more  question. Does the partition addition tool work
> only with zookeeper? Can I keep the zk cluster running and shut down kafka
> processes
> before I run the partition addition tool? I am asking this question
> because we try to avoid
> any message loss when adding a partition in a production environment.
> Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Monday, December 16, 2013 12:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: cluster expansion
>
> If we have six brokers, and a topic has three partitions on broker 1, 2, 3.
> Now if I add another three partitions. Will they be evenly distributed to
> 1,2,3 or 4,5,6?
>
> Evenly distributed on all six brokers.
>
> If I use reassignment tool in 0.81 with 0.8 broker, will that work and get
> around the bugs?
>
> Your broker also needs to be on 0.8.1 for it to work correctly.
>
>
>
> On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
> > If we have six brokers, and a topic has three partitions on broker 1, 2,
> 3.
> > Now if I add another three partitions. Will they be evenly distributed
> > to
> > 1,2,3 or 4,5,6?
> >
> > One more question.
> > If I use reassignment tool in 0.81 with 0.8 broker, will that work and
> > get around the bugs?
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, December 16, 2013 10:26 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: cluster expansion
> >
> > They will be evenly distributed across the nodes in the cluster.
> >
> > Thanks,
> > Neha
> > On Dec 16, 2013 6:42 AM, "Yu, Libo" <[EMAIL PROTECTED]> wrote:
> >
> > > Assume we have three brokers and a topic already has three partitions.
> > > Now if I add another three partitions to this topic. Where will the
> > > three partitions be located after running the script?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > > Sent: Sunday, December 15, 2013 11:43 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: cluster expansion
> > >
> > > Yes, kafka-add-partitons.sh should be stable in 0.8.0.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Is the kafka-add-partitons.sh tool stable in 0.8.0?
> > > >
> > > >
> > > > On 13 December 2013 19:21, Neha Narkhede <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Partition movement is not an automatic operation in Kafka yet.
> > > > > You need
> > > > to
> > > > > use the partition reassignment tool -
> > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tool
> > > > s# Re plicationtools-6.ReassignPartitionsTool
> > > > > .
> > > > >
> > > > >
> > > > > Also, that feature is stable in 0.8.1.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner
> > > > > <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > No the 6 partitions for each topic will remain on the original
> > > brokers.
> > > > > You
> > > > > > could either reassign some partitions from all topics to the
> > > > > > new
> > > > brokers
> > > > > or
> > > > > > you could add partitions to the new brokers for each topic. In
> > > > > > 0.8.0

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB