Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> partition reassignment


Copy link to this message
-
Re: partition reassignment
Yes, thanks, looks like that's what i need, do you know why it tends to
choose the leader for all partitions on the single broker, despite I have 3?
On Wed, Oct 16, 2013 at 12:19 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Did the reassignment complete? If the assigned replicas are in ISR and
> the preferred replicas for the partitions are evenly distributed
> across the brokers (which seems to be a case on a cursory glance of
> your assignment) you can use this tool:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
>
> On Wed, Oct 16, 2013 at 12:15 AM, Kane Kane <[EMAIL PROTECTED]> wrote:
> > Oh i see, what is the better way to initiate the leader change? As I told
> > somehow all my partitions have the same leader for some reason. I have 3
> > brokers and all partitions have leader on single one.
> >
> >
> > On Wed, Oct 16, 2013 at 12:04 AM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> >
> >> For a leader change yes, but this is partition reassignment which
> >> completes when all the reassigned replicas are in sync with the
> >> original replica(s). You can check the status of the command using the
> >> option I mentioned earlier.
> >>
> >> On Tue, Oct 15, 2013 at 7:02 PM, Kane Kane <[EMAIL PROTECTED]>
> wrote:
> >> > I thought if i have all replicas in sync, leader change should be much
> >> > faster?
> >> >
> >> >
> >> > On Tue, Oct 15, 2013 at 5:12 PM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> Depending on how much data there is in those partitions it can take a
> >> >> while for reassignment to actually complete. You will need to use the
> >> >> --status-check-json-file option of the reassign partitions command to
> >> >> determine whether partition reassignment has completed or not.
> >> >>
> >> >> Joel
> >> >>
> >> >>
> >> >> On Tue, Oct 15, 2013 at 3:46 PM, Kane Kane <[EMAIL PROTECTED]>
> >> wrote:
> >> >> > I have 3 brokers and a topic with replication factor of 3.
> >> >> > Somehow all partitions ended up being on the same broker.
> >> >> > I've created topic with 3 brokers alive, and they didn't die since
> >> then.
> >> >> >
> >> >> > Even when i try to reassign it:
> >> >> > bin/kafka-reassign-partitions.sh --zookeeper
> >> >> > 10.80.42.147:2181--broker-list 0,1,2 --topics-to-move-json-file
> >> >> > ~/reassign.txt --execute
> >> >> >
> >> >> > The leader of all partitions after that is still the single broker
> >> (first
> >> >> > one).
> >> >> > This is how my json file looks like:
> >> >> > {"topics":
> >> >> >                                             [{"topic": "perf1"}],
> >> >> >                                         "version":1
> >> >> >                                         }
> >> >> >
> >> >> > The command reports success:
> >> >> > Successfully started reassignment of partitions Map([perf1,33] ->
> >> List(1,
> >> >> > 0, 2), [perf1,13] -> List(2, 0, 1), [perf1,28] -> List(2, 1, 0),
> >> >> [perf1,20]
> >> >> > -> List(0, 1, 2), [perf1,5] -> List(0, 2, 1), [perf1,14] ->
> List(0, 1,
> >> >> 2),
> >> >> > [perf1,39] -> List(1, 0, 2), [perf1,19] -> List(2, 0, 1),
> [perf1,25]
> >> ->
> >> >> > List(2, 0, 1), [perf1,43] -> List(2, 0, 1), [perf1,11] -> List(0,
> 2,
> >> 1),
> >> >> > [perf1,8] -> List(0, 1, 2), [perf1,22] -> List(2, 1, 0),
> [perf1,16] ->
> >> >> > List(2, 1, 0), [perf1,26] -> List(0, 1, 2), [perf1,41] -> List(0,
> 2,
> >> 1),
> >> >> > [perf1,9] -> List(1, 0, 2), [perf1,27] -> List(1, 0, 2),
> [perf1,44] ->
> >> >> > List(0, 1, 2), [perf1,34] -> List(2, 1, 0), [perf1,6] -> List(1, 2,
> >> 0),
> >> >> > [perf1,1] -> List(2, 0, 1), [perf1,37] -> List(2, 0, 1),
> [perf1,24] ->
> >> >> > List(1, 2, 0), [perf1,29] -> List(0, 2, 1), [perf1,32] -> List(0,
> 1,
> >> 2),
> >> >> > [perf1,7] -> List(2, 0, 1), [perf1,17] -> List(0, 2, 1),
> [perf1,36] ->
> >> >> > List(1, 2, 0), [perf1,0] -> List(1, 2, 0), [perf1,42] -> List(1, 2,
> >> 0),
> >> >> > [perf1,4] -> List(2, 1, 0), [perf1,10] -> List(2, 1, 0),

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB