Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Best practices for changing partition numbers


Copy link to this message
-
Re: Best practices for changing partition numbers
Jun Rao 2013-01-09, 05:17
Reducing # partitions is going to be tricky. The data for those dropped
partitions will just be lost.

Thanks,

Jun

On Tue, Jan 8, 2013 at 1:24 PM, David Ross <[EMAIL PROTECTED]> wrote:

> Yeah that makes sense, but what if we do need to change the number of
> partitions? What if we need to reduce it?
>
> On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > If you don't have a lot of topics, one thing you can do is to
> > over-partition a topic.
> >
> > Also, in 0.7, # of partitions grows with brokers. This is going to change
> > in 0.8, in which # of partitions is specified at topic creation time and
> > won't change as brokers change. One needs to use an admin DDL to change #
> > of partitions.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 7, 2013 at 10:23 PM, David Ross <[EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > We have found that, for our application, having a number of total
> > > partitions as a multiple of the number of consumer hosts is beneficial.
> > > Because of this, whenever we add or remove consumer hosts, we have to
> > > change the number of partitions in the server config.
> > >
> > > What are best practices for changing the number of partitions? It seems
> > > like adding partitions is fine but removing partitions would result in
> > data
> > > loss - am I right? Is that avoidable? Is it preferable to bring in new
> > > servers with new partitions? Anything else I should keep in mind on
> this
> > > issue?
> > >
> > >
> > > Thanks!
> > >
> > > David
> > >
> >
>