Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Best practices for changing partition numbers


+
David Ross 2013-01-08, 06:24
Copy link to this message
-
Re: Best practices for changing partition numbers
If you don't have a lot of topics, one thing you can do is to
over-partition a topic.

Also, in 0.7, # of partitions grows with brokers. This is going to change
in 0.8, in which # of partitions is specified at topic creation time and
won't change as brokers change. One needs to use an admin DDL to change #
of partitions.

Thanks,

Jun

On Mon, Jan 7, 2013 at 10:23 PM, David Ross <[EMAIL PROTECTED]> wrote:

> Hello,
>
> We have found that, for our application, having a number of total
> partitions as a multiple of the number of consumer hosts is beneficial.
> Because of this, whenever we add or remove consumer hosts, we have to
> change the number of partitions in the server config.
>
> What are best practices for changing the number of partitions? It seems
> like adding partitions is fine but removing partitions would result in data
> loss - am I right? Is that avoidable? Is it preferable to bring in new
> servers with new partitions? Anything else I should keep in mind on this
> issue?
>
>
> Thanks!
>
> David
>

 
+
David Ross 2013-01-08, 21:24
+
Corbin Hoenes 2013-01-08, 21:41
+
David Ross 2013-01-09, 00:26
+
Jun Rao 2013-01-09, 05:17