Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Best practices for changing partition numbers


+
David Ross 2013-01-08, 06:24
+
Jun Rao 2013-01-08, 20:43
+
David Ross 2013-01-08, 21:24
+
Corbin Hoenes 2013-01-08, 21:41
Copy link to this message
-
Re: Best practices for changing partition numbers
I know how to change. I'm wondering what the ramifications are (losing
data, etc.).

On Tue, Jan 8, 2013 at 1:41 PM, Corbin Hoenes <[EMAIL PROTECTED]> wrote:

> David
>
> Sounds like there is an admin ddl you have to run on your existing topic
> to change the # after creation.
>
> Sent from my iPhone
>
> On Jan 8, 2013, at 2:24 PM, David Ross <[EMAIL PROTECTED]> wrote:
>
> > Yeah that makes sense, but what if we do need to change the number of
> > partitions? What if we need to reduce it?
> >
> > On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> If you don't have a lot of topics, one thing you can do is to
> >> over-partition a topic.
> >>
> >> Also, in 0.7, # of partitions grows with brokers. This is going to
> change
> >> in 0.8, in which # of partitions is specified at topic creation time and
> >> won't change as brokers change. One needs to use an admin DDL to change
> #
> >> of partitions.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Mon, Jan 7, 2013 at 10:23 PM, David Ross <[EMAIL PROTECTED]> wrote:
> >>
> >>> Hello,
> >>>
> >>> We have found that, for our application, having a number of total
> >>> partitions as a multiple of the number of consumer hosts is beneficial.
> >>> Because of this, whenever we add or remove consumer hosts, we have to
> >>> change the number of partitions in the server config.
> >>>
> >>> What are best practices for changing the number of partitions? It seems
> >>> like adding partitions is fine but removing partitions would result in
> >> data
> >>> loss - am I right? Is that avoidable? Is it preferable to bring in new
> >>> servers with new partitions? Anything else I should keep in mind on
> this
> >>> issue?
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> David
> >>
>

 
+
Jun Rao 2013-01-09, 05:17