Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - partition reassignment


+
Kane Kane 2013-10-15, 22:47
+
Joel Koshy 2013-10-16, 00:13
+
Kane Kane 2013-10-16, 02:03
+
Joel Koshy 2013-10-16, 07:05
+
Kane Kane 2013-10-16, 07:16
+
Joel Koshy 2013-10-16, 07:20
+
Kane Kane 2013-10-16, 07:30
Copy link to this message
-
Re: partition reassignment
"hsy541@...> 2013-10-16, 07:38
There is a ticket for auto-rebalancing, hopefully they'll do auto
redistribution soon
https://issues.apache.org/jira/browse/KAFKA-930
On Wed, Oct 16, 2013 at 12:29 AM, Kane Kane <[EMAIL PROTECTED]> wrote:

> Yes, thanks, looks like that's what i need, do you know why it tends to
> choose the leader for all partitions on the single broker, despite I have
> 3?
>
>
> On Wed, Oct 16, 2013 at 12:19 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Did the reassignment complete? If the assigned replicas are in ISR and
> > the preferred replicas for the partitions are evenly distributed
> > across the brokers (which seems to be a case on a cursory glance of
> > your assignment) you can use this tool:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
> >
> > On Wed, Oct 16, 2013 at 12:15 AM, Kane Kane <[EMAIL PROTECTED]>
> wrote:
> > > Oh i see, what is the better way to initiate the leader change? As I
> told
> > > somehow all my partitions have the same leader for some reason. I have
> 3
> > > brokers and all partitions have leader on single one.
> > >
> > >
> > > On Wed, Oct 16, 2013 at 12:04 AM, Joel Koshy <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> For a leader change yes, but this is partition reassignment which
> > >> completes when all the reassigned replicas are in sync with the
> > >> original replica(s). You can check the status of the command using the
> > >> option I mentioned earlier.
> > >>
> > >> On Tue, Oct 15, 2013 at 7:02 PM, Kane Kane <[EMAIL PROTECTED]>
> > wrote:
> > >> > I thought if i have all replicas in sync, leader change should be
> much
> > >> > faster?
> > >> >
> > >> >
> > >> > On Tue, Oct 15, 2013 at 5:12 PM, Joel Koshy <[EMAIL PROTECTED]>
> > wrote:
> > >> >
> > >> >> Depending on how much data there is in those partitions it can
> take a
> > >> >> while for reassignment to actually complete. You will need to use
> the
> > >> >> --status-check-json-file option of the reassign partitions command
> to
> > >> >> determine whether partition reassignment has completed or not.
> > >> >>
> > >> >> Joel
> > >> >>
> > >> >>
> > >> >> On Tue, Oct 15, 2013 at 3:46 PM, Kane Kane <[EMAIL PROTECTED]>
> > >> wrote:
> > >> >> > I have 3 brokers and a topic with replication factor of 3.
> > >> >> > Somehow all partitions ended up being on the same broker.
> > >> >> > I've created topic with 3 brokers alive, and they didn't die
> since
> > >> then.
> > >> >> >
> > >> >> > Even when i try to reassign it:
> > >> >> > bin/kafka-reassign-partitions.sh --zookeeper
> > >> >> > 10.80.42.147:2181--broker-list 0,1,2 --topics-to-move-json-file
> > >> >> > ~/reassign.txt --execute
> > >> >> >
> > >> >> > The leader of all partitions after that is still the single
> broker
> > >> (first
> > >> >> > one).
> > >> >> > This is how my json file looks like:
> > >> >> > {"topics":
> > >> >> >                                             [{"topic": "perf1"}],
> > >> >> >                                         "version":1
> > >> >> >                                         }
> > >> >> >
> > >> >> > The command reports success:
> > >> >> > Successfully started reassignment of partitions Map([perf1,33] ->
> > >> List(1,
> > >> >> > 0, 2), [perf1,13] -> List(2, 0, 1), [perf1,28] -> List(2, 1, 0),
> > >> >> [perf1,20]
> > >> >> > -> List(0, 1, 2), [perf1,5] -> List(0, 2, 1), [perf1,14] ->
> > List(0, 1,
> > >> >> 2),
> > >> >> > [perf1,39] -> List(1, 0, 2), [perf1,19] -> List(2, 0, 1),
> > [perf1,25]
> > >> ->
> > >> >> > List(2, 0, 1), [perf1,43] -> List(2, 0, 1), [perf1,11] -> List(0,
> > 2,
> > >> 1),
> > >> >> > [perf1,8] -> List(0, 1, 2), [perf1,22] -> List(2, 1, 0),
> > [perf1,16] ->
> > >> >> > List(2, 1, 0), [perf1,26] -> List(0, 1, 2), [perf1,41] -> List(0,
> > 2,
> > >> 1),
> > >> >> > [perf1,9] -> List(1, 0, 2), [perf1,27] -> List(1, 0, 2),
> > [perf1,44] ->
> > >> >> > List(0, 1, 2), [perf1,34] -> List(2, 1, 0), [perf1,6] -> List(1,

 
+
Jun Rao 2013-10-16, 14:57
+
Kane Kane 2013-10-16, 15:54