Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Copy availability when broker goes down?


Copy link to this message
-
Copy availability when broker goes down?
Hi,

(Hmm, take 2. Apache's spam filter doesn't like the word to describe the
copy of the data. 'R - E -P -L -I -C -A' so it blocked it from sending!
Using 'copy' below to mean that concept)

I’m running 0.8.0 with HEAD from end of January (not the merge you guys did
last night).

I’m testing how the producer responds to loss of brokers, what errors are
produced etc. and noticed some strange things as I shutdown servers in my
cluster.

Setup:
4 node cluster
1 topic, 3 copies in the set
10 partitions numbered 0-9

State of the cluster is determined using TopicMetadataRequest.

When I start with a full cluster (2nd column is the partition id, next is
leader, then the copy set and ISR):

Java: 0:vrd03.atlnp1 R:[  vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1]
Java: 1:vrd04.atlnp1 R:[  vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[
vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1]
Java: 2:vrd03.atlnp1 R:[  vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[
vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1]
Java: 3:vrd03.atlnp1 R:[  vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 4:vrd03.atlnp1 R:[  vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[
vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1]
Java: 5:vrd03.atlnp1 R:[  vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 6:vrd03.atlnp1 R:[  vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1]
Java: 7:vrd04.atlnp1 R:[  vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[
vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1]
Java: 8:vrd03.atlnp1 R:[  vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 9:vrd03.atlnp1 R:[  vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1]

When I stop vrd01, which isn’t leader on any:

Java: 0:vrd03.atlnp1 R:[ ] I:[]
Java: 1:vrd04.atlnp1 R:[ ] I:[]
Java: 2:vrd03.atlnp1 R:[ ] I:[]
Java: 3:vrd03.atlnp1 R:[  vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 4:vrd03.atlnp1 R:[ ] I:[]
Java: 5:vrd03.atlnp1 R:[  vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 6:vrd03.atlnp1 R:[ ] I:[]
Java: 7:vrd04.atlnp1 R:[ ] I:[]
Java: 8:vrd03.atlnp1 R:[  vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[
vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
Java: 9:vrd03.atlnp1 R:[ ] I:[]

Does this mean that none of the partitions that used to have a copy on
vrd01 are updating ANY of the copies?

I ran another test, again starting with a full cluster and all partitions
had a full set of copies. When I stop the broker which was leader for 9 of
the 10 partitions, the leaders were all elected on one machine instead of
the set of 3. Should the leaders have been better spread out? Also the
copies weren’t fully populated either.

Last test: started with a full cluster, showing all copies available.
Stopped a broker that was not a leader for any partition. Noticed that the
partitions where the stopped machine was in the copy set didn’t show any
copies like above. Let the cluster sit for 30 minutes and didn’t see any
new copies being brought on line. How should the cluster handle a machine
that is down for an extended period of time?

I don’t have a new machine I could add to the cluster, but what happens
when I do? Will it not be used until a new topic is added or how does it
become a valid option for a copy or eventually the leader?

Thanks,

Chris

 
+
Neha Narkhede 2013-03-04, 17:32
+
Jun Rao 2013-03-04, 18:15
+
Chris Curtin 2013-03-04, 18:22