Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-783) Preferred replica assignment on leader failure may not be correct


Copy link to this message
-
[jira] [Commented] (KAFKA-783) Preferred replica assignment on leader failure may not be correct

    [ https://issues.apache.org/jira/browse/KAFKA-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592566#comment-13592566 ]

Chris Curtin commented on KAFKA-783:
------------------------------------

Confirmed same behavior on 0.8.0 HEAD as of 3/4/2013
                
> Preferred replica assignment on leader failure may not be correct
> -----------------------------------------------------------------
>
>                 Key: KAFKA-783
>                 URL: https://issues.apache.org/jira/browse/KAFKA-783
>             Project: Kafka
>          Issue Type: Bug
>          Components: replication
>    Affects Versions: 0.8
>         Environment: $ uname -a
> Linux vrd01.atlnp1 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> $ java -version
> java version "1.6.0_25"
> Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
> Kafka 0.8.0 loaded from HEAD on 1/29/2013
>            Reporter: Chris Curtin
>            Assignee: Neha Narkhede
>
> Based on an email thread in the user group, Neha asked me to submit this.
> Original question: "> I ran another test, again starting with a full cluster and all partitions
> > had a full set of copies. When I stop the broker which was leader for 9 of
> > the 10 partitions, the leaders were all elected on one machine instead of
> > the set of 3. Should the leaders have been better spread out? Also the
> > copies weren’t fully populated either."
> Neha: "For problem 2, we always try to make the preferred replica (1st replica
> in the list of all replicas for a partition) the leader, if it is
> available. We intended to spread the preferred replica for all partitions
> for a topic evenly across the brokers. If this is not happening, we need to
> look into it. Please can you file a bug and describe your test case there ?"
> Configuration:
> 4 node cluster
> 1 topic with 3 replicas
> 10 partitions: 0-9 below
> Current status:
> Partition: 0:vrd01.atlnp1 R:[  vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 1:vrd01.atlnp1 R:[  vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 2:vrd01.atlnp1 R:[  vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1]
> Partition: 3:vrd03.atlnp1 R:[  vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 4:vrd01.atlnp1 R:[  vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1]
> Partition: 5:vrd03.atlnp1 R:[  vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 6:vrd01.atlnp1 R:[  vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 7:vrd01.atlnp1 R:[  vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 8:vrd03.atlnp1 R:[  vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[ vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 9:vrd01.atlnp1 R:[  vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Shutdown vrd03:
> Partition: 0:vrd01.atlnp1 R:[ ] I:[]
> Partition: 1:vrd01.atlnp1 R:[  vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 2:vrd01.atlnp1 R:[ ] I:[]
> *Partition: 3:vrd04.atlnp1 R:[ ] I:[]
> Partition: 4:vrd01.atlnp1 R:[ ] I:[]
> *Partition: 5:vrd04.atlnp1 R:[ ] I:[]
> Partition: 6:vrd01.atlnp1 R:[ ] I:[]
> Partition: 7:vrd01.atlnp1 R:[  vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> *Partition: 8:vrd04.atlnp1 R:[ ] I:[]
> Partition: 9:vrd01.atlnp1 R:[ ] I:[]
> (* means leader changed)
> Note that partitions 3, 5 and 8 were assigned new leaders.
> Per an email group thread with Neha, the new leader should be assigned from the preferred replica. So 3 should have gotten vrd02, 5, vrd04 and 8 vrd02 (since 03 was shutdown). Instead 3 got vrd04, 5 got vrd04 and 8 got vrd04.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB