Does this mean that none of the partitions that used to have a copy on vrd01 are updating ANY of the copies?
I ran another test, again starting with a full cluster and all partitions had a full set of copies. When I stop the broker which was leader for 9 of the 10 partitions, the leaders were all elected on one machine instead of the set of 3. Should the leaders have been better spread out? Also the copies weren’t fully populated either.
Last test: started with a full cluster, showing all copies available. Stopped a broker that was not a leader for any partition. Noticed that the partitions where the stopped machine was in the copy set didn’t show any copies like above. Let the cluster sit for 30 minutes and didn’t see any new copies being brought on line. How should the cluster handle a machine that is down for an extended period of time?
I don’t have a new machine I could add to the cluster, but what happens when I do? Will it not be used until a new topic is added or how does it become a valid option for a copy or eventually the leader?
Thanks for reporting the issues and running those tests.
1. For problem 1, if this is the output of topic metadata request after shutting down a broker that leads no partitions, then that is a bug. Please can you file a bug and describe a reproducible test case there ? 2. For problem 2, we always try to make the preferred replica (1st replica in the list of all replicas for a partition) the leader, if it is available. We intended to spread the preferred replica for all partitions for a topic evenly across the brokers. If this is not happening, we need to look into it. Please can you file a bug and describe your test case there ? 3. For a machine that is down, for some time or long time, it is taken out of ISR. When it starts back up again, it has to bootstrap from the current leader. 4. If you have a new machine that you want to add to the cluster, you might want to reassign some replicas for partitions to the new broker. We have a tool (that has not been thoroughly tested yet) that allows you to do that.
Thanks, Neha On Mon, Mar 4, 2013 at 8:32 AM, Chris Curtin <[EMAIL PROTECTED]> wrote:
As Neha said, the 1st copy of a partition is the preferred replica and we try to spread them evenly across the brokers. When a broker is restarted, we don't automatically move the leader back to the preferred replica though. You will have to run a command line tool PreferredReplicaLeaderElectionCommand to balance the leaders again.
Also, I recommend that you try the latest code in 0.8. A bunch of issues have been fixes since Jan. You will have to wipe out all your ZK and Kafka data first though.
On Mon, Mar 4, 2013 at 8:32 AM, Chris Curtin <[EMAIL PROTECTED]> wrote: