Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Best practices for changing partition numbers


+
David Ross 2013-01-08, 06:24
+
Jun Rao 2013-01-08, 20:43
+
David Ross 2013-01-08, 21:24
+
Corbin Hoenes 2013-01-08, 21:41
+
David Ross 2013-01-09, 00:26
Copy link to this message
-
Re: Best practices for changing partition numbers
Reducing # partitions is going to be tricky. The data for those dropped
partitions will just be lost.

Thanks,

Jun

On Tue, Jan 8, 2013 at 1:24 PM, David Ross <[EMAIL PROTECTED]> wrote:

> Yeah that makes sense, but what if we do need to change the number of
> partitions? What if we need to reduce it?
>
> On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > If you don't have a lot of topics, one thing you can do is to
> > over-partition a topic.
> >
> > Also, in 0.7, # of partitions grows with brokers. This is going to change
> > in 0.8, in which # of partitions is specified at topic creation time and
> > won't change as brokers change. One needs to use an admin DDL to change #
> > of partitions.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 7, 2013 at 10:23 PM, David Ross <[EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > We have found that, for our application, having a number of total
> > > partitions as a multiple of the number of consumer hosts is beneficial.
> > > Because of this, whenever we add or remove consumer hosts, we have to
> > > change the number of partitions in the server config.
> > >
> > > What are best practices for changing the number of partitions? It seems
> > > like adding partitions is fine but removing partitions would result in
> > data
> > > loss - am I right? Is that avoidable? Is it preferable to bring in new
> > > servers with new partitions? Anything else I should keep in mind on
> this
> > > issue?
> > >
> > >
> > > Thanks!
> > >
> > > David
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB