Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Partition data for deleted topic found in kafka-logs, also, found leader: -1


Copy link to this message
-
Partition data for deleted topic found in kafka-logs, also, found leader: -1
Hi Team,

I am exploring kafka 0.8 beta release to understand data flow, replication
features.
While testing i found that, the partition data for data for deleted topic
is preseved in kafka-logs, why this behavior? suppose below case,

 A topic (suppose test1) is created with partition 6 and replication 3 on a
system with 4 brokers, respective log and index files will be prepared per
partition in kafka-logs. If I delete the topic and recreate the same topic
‘test1’ after some time with partition 2 and replication 2. The kafka-logs
directory seems to be confusing to understand why the partitions for
previous topic are present. *Please help to understand this scenario*.

Also, while testing the replication and leader selection feature observed
leader -1 status,

Original status:
topic: test1    partition: 0    leader: 4       replicas: 4,2,3 isr: 4,2,3
topic: test1    partition: 1    leader: 0       replicas: 0,3,4 isr: 0,3,4
topic: test1    partition: 2    leader: 1       replicas: 1,4,0 isr: 1,4,0
if leader 4 goes down:
topic: test1    partition: 0    leader: 2       replicas: 4,2,3 isr: 2,3
topic: test1    partition: 1    leader: 0       replicas: 0,3,4 isr: 0,3,4
topic: test1    partition: 2    leader: 1       replicas: 1,4,0 isr: 1,0,4

if leader 2  goes down:
topic: test1    partition: 0    leader: 3       replicas: 4,2,3 isr: 3
topic: test1    partition: 1    leader: 0       replicas: 0,3,4 isr: 0,3,4
topic: test1    partition: 2    leader: 1       replicas: 1,4,0 isr: 1,0,4

if again leader 3 goes down:
topic: test1    partition: 0    leader: -1      replicas: 4,2,3 isr:
topic: test1    partition: 1    leader: 0       replicas: 0,3,4 isr: 0,4
topic: test1    partition: 2    leader: 1       replicas: 1,4,0 isr: 1,0,4

As per kafka protocol guide, *leader: -1 means If no leader exists because
we are in the middle of a leader election this id will be -1.*

*Does it mean that, the data from partition 0 will be unavailable due no
leader (leader selection in progress)?*
As per my understanding, can we have auto re-balancer facility to
re-balance the partition replications to available brokers if one of the
broker is down, as in above case of (if leader 4 goes down), we can
replicate the partition 0 to broker 0/1 to re-balance the replication.

Please correct me for any wrong understanding as those are my initial
observations.

Thanks in advance.

Thanks,
Yogesh Sangvikar