Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Dynamic reconfiguration

Jared Cantwell 2012-07-27, 17:06
Alexander Shraer 2012-07-28, 00:20
Alexander Shraer 2012-07-28, 00:25
Jared Cantwell 2012-07-28, 00:25
Alexander Shraer 2012-07-28, 00:29
Jared Cantwell 2012-07-28, 00:35
Alexander Shraer 2012-07-28, 00:55
Jared Cantwell 2012-07-28, 01:34
Alexander Shraer 2012-07-28, 01:42
Jared Cantwell 2012-07-28, 02:57
Copy link to this message
Re: Dynamic reconfiguration
So I'm working through some failure scenarios and I want to make sure I
fully understand the way that dynamic membership changes previous behavior,
so are my expectations correct in this situation:

As in my previous example, lets say that the current membership of voting
participants is {A,B,C,D,E} and we're looking to change membership to
1. Reconfiguration to {D,E,F,G,H} completes internally
2. D-F update their local configuration files, but A-C do not yet.
3. Power loss to all nodes

Now what happens if A,B, and C come up with configuration files that still
say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C form a
quorum and elect a leader since they all agree on the same state?  What
then happens when the new membership of D-H starts up?

We're trying to automatically handle node failures during reconfiguration
situations, but it seems like without being able to query all nodes to make
sure you know of the latest membership list there is no safe way to do
this.  I'm wondering if only doing single node additions/removals would
create less complicated failure scenarios.  What are your thoughts and best
practices around this?


On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <[EMAIL PROTECTED]>wrote:

> We are trying to remove the need for all admin intervention so that is one
> failure scenario that is interesting to us.
> Jared
> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <[EMAIL PROTECTED]> wrote:
> Yes, this entry will be deleted. I don't like this either - if a new
> follower reboots before added to the config it will not be able to boot up
> without manual help from an admin. That's why I'm considering maybe to
> remove the check that a participant must always initially be in its own
> config, but for now its there.
> Alex
> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <[EMAIL PROTECTED]>wrote:
>> Sorry for the confusion in terminology, I was unfamiliar with the exact
>> leader/follower semantics previously.
>> So if all connected servers update their config file, does that mean that
>> non-voting followers who aren't part of the new ensemble will lose the
>> entry specific to them in their config file?  I can test this myself, but
>> getting an inside perspective is very helpful.
>> Thanks again for the help!
>> Jared
>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <[EMAIL PROTECTED]> wrote:
>> Yes, any number of followers which are not in the configuration can just
>> connect and listen in. This has always been the case, also in 3.4, I just
>> made use of this for the purpose of adding members during reconfiguration.
>> Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
>> where the leader actually counts the votes of anyone connected,
>> regardless of config membership :) This is fixed in ZK-107, so they are
>> really non-voting followers.
>> >   I am assuming that's the case, and that it is a follower (and not
>> > participant) by virtue of not being in the official configuration
>> stored in
>> > zookeeper itself.
>> Follower and participant types of servers is not something that was
>> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
>> Its just that the votes of participants that are not in the configuration
>> are not counted that's why we call them non-voting followers. BTW,
>> obviously a non-voting follower can not become leader (like ZK-1113 this
>> was also not enforced before ZK-107).
>> > And a followup... does zookeeper only overwrite the dynamic
>> > configuration file for nodes that are voting participants?  Such that
>> if I
>> > started a follower and then left it running through some
>> > reconfigurations, its file would not get updated if it was never added
>> as
>> > part of those reconfigurations?
>> No, as soon as it connects to the current leader, its dynamic config file
>> is overwritten with the current configuration as part of the
Alexander Shraer 2012-07-28, 17:02
Jared Cantwell 2012-07-28, 17:17
Alexander Shraer 2012-07-28, 17:33
Jared Cantwell 2012-07-28, 17:57
Alexander Shraer 2012-07-28, 18:15
Jared Cantwell 2012-07-28, 22:26
Alexander Shraer 2012-07-28, 23:36
Jared Cantwell 2012-07-28, 23:52
Alexander Shraer 2012-07-29, 01:00
Alexander Shraer 2012-07-28, 23:01
Jared Cantwell 2012-07-28, 00:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB