Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Dynamic reconfiguration

Jared Cantwell 2012-07-27, 17:06
Alexander Shraer 2012-07-28, 00:20
Alexander Shraer 2012-07-28, 00:25
Jared Cantwell 2012-07-28, 00:25
Alexander Shraer 2012-07-28, 00:29
Jared Cantwell 2012-07-28, 00:35
Alexander Shraer 2012-07-28, 00:55
Jared Cantwell 2012-07-28, 01:34
Alexander Shraer 2012-07-28, 01:42
Jared Cantwell 2012-07-28, 02:57
Jared Cantwell 2012-07-28, 15:43
Copy link to this message
Re: Dynamic reconfiguration
Hi Jared,

figuring out what happened and how to recover is part of the
reconfiguration protocol. I don't think that this is something you as a
user should do, unless I missunderstand what you're trying to do. This
should be handled by ZooKeeper just like it handles other failures without
admin intervention.

In your scenario, D-F come up and one of them is elected leader (since you
said they know about the commit), so they start running the new config
normally. When A-C come up, several things may happen:

1. During the preliminary FastLeaderElection, A-C will try to connect to D
and E, and in fact they'll also try to connect with the new config members
that they know was proposed. So most chances are that someone in the new
config will send them the new config file and they'll store it and act
accordingly (connect as non-voting followers in the new config). To make
this happen, I changed FastLeaderElection to talk with proposed configs (if
known) and to piggiback the last active config you know of on all messages.

2. Its possible that somehow A-C complete FastLeaderElection without
talking to D-F. But since a reconfiguration was committed, it was acked by
a quorum of the old config (and a quorum of the new one). Therefore,
whoever is "elected" in the old config, knows about the reconfig proposal
(this is guaranteed by normal ZooKeeper leader recovery). Before doing
anything else, the new leader among A-C will try to complete the
reconfiguration, which involves getting enough acks from a quorum of the
new config. But in your scenario the servers in the new config will not
connect to it because they moved on, so the candidate-leader will just give
up and go back to (1) above.

3. In the remote chance that someone who heard about the reconfig commit
connects to a candidate-leader who didn't hear about it, the first thing it
does  is to tell that candidate-leader that its not up to date, and the
leader just updates its config file, gives up on being a leader and returns
to (1). This was done by changing the first message that a
follower/observer sends to a leader it is connecting to, even before the
synchronization starts.


On Sat, Jul 28, 2012 at 8:43 AM, Jared Cantwell

> So I'm working through some failure scenarios and I want to make sure I
> fully understand the way that dynamic membership changes previous behavior,
> so are my expectations correct in this situation:
> As in my previous example, lets say that the current membership of voting
> participants is {A,B,C,D,E} and we're looking to change membership to
> {D,E,F,G,H}.
> 1. Reconfiguration to {D,E,F,G,H} completes internally
> 2. D-F update their local configuration files, but A-C do not yet.
> 3. Power loss to all nodes
> Now what happens if A,B, and C come up with configuration files that still
> say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C form a
> quorum and elect a leader since they all agree on the same state?  What
> then happens when the new membership of D-H starts up?
> We're trying to automatically handle node failures during reconfiguration
> situations, but it seems like without being able to query all nodes to make
> sure you know of the latest membership list there is no safe way to do
> this.  I'm wondering if only doing single node additions/removals would
> create less complicated failure scenarios.  What are your thoughts and best
> practices around this?
> Thanks!
> Jared
> On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <[EMAIL PROTECTED]>wrote:
>> We are trying to remove the need for all admin intervention so that is
>> one failure scenario that is interesting to us.
>> Jared
>> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <[EMAIL PROTECTED]> wrote:
>> Yes, this entry will be deleted. I don't like this either - if a new
>> follower reboots before added to the config it will not be able to boot up
>> without manual help from an admin. That's why I'm considering maybe to
Jared Cantwell 2012-07-28, 17:17
Alexander Shraer 2012-07-28, 17:33
Jared Cantwell 2012-07-28, 17:57
Alexander Shraer 2012-07-28, 18:15
Jared Cantwell 2012-07-28, 22:26
Alexander Shraer 2012-07-28, 23:36
Jared Cantwell 2012-07-28, 23:52
Alexander Shraer 2012-07-29, 01:00
Alexander Shraer 2012-07-28, 23:01
Jared Cantwell 2012-07-28, 00:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB