Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Rolling config change considered harmful?


Copy link to this message
-
Re: Rolling config change considered harmful?
Hello,

Could you please clarify if this thread is about a rolling start in an
ensemble without the dynamic reconfiguration support?
And when you say "Create a 5 node ensemble", that means quorum is 5. But
then you give server lists with only 3 servers in each node?
If the server list has 3 servers, then quorum is actually 3 and what is
described may happen in that scenario.
In that case C follows B, E follows D and A follows either B or D and there
are two working ensembles.
It should be possible to create problems, even with more standard
configuration changes:
If we want to change a quorum of three to a quorum of five {A,B,C} to
{A,B,C,D,E}:
- First the configuration is changed in all the nodes, but they are not
restarted. Only A, B and C are running.
- One of them is stopped (e.g. A).
- At this point, if A, D and E are started with the new configuration, they
may elect a leader before any of them is aware of either B or C, form an
ensemble and start serving txns.
- However, if A is started, we wait until it connects to the leader of B
and C, and then D and E are started and then B and C are restarted,
everything should be ok. The fact that this depends on the human ability to
start D and E while A,B and C are connected to the ensemble seems a bit
risky though.
I have found a presentation on the topic:
http://www.slideshare.net/Hadoop_Summit/dynamic-reconfiguration-of-zookeeper

If anybody knows of a safer way to change a quorum of 3 to a quorum of 5
with e.g. zookeeper 3.4.5, please point it out.

Regards,

Germán.
On Fri, Jun 14, 2013 at 11:46 PM, Jordan Zimmerman <
[EMAIL PROTECTED]> wrote:

> I got the test cluster into the state described with 2 leaders. I then
> allocated 100 Curator clients to write nodes "/n" where n is the index
> (i.e. "/0", "/1", …). The idea that the nodes would be distributed around
> the cluster instances. I then allocated a single Curator instance dedicated
> to one of the servers instance, did a sync, and did an exists() to verify
> that each cluster instances had all the nodes. For the 2 leader cluster,
> this fails.
>
> -JZ
>
> On Jun 14, 2013, at 1:54 PM, "FPJ" <[EMAIL PROTECTED]> wrote:
>
> > I messed up the last sentence, here is what I was trying to say:
> >
> > It is ok to have two servers thinking they are leaders as long as only
> one
> > is
> > able to commit txns at a time by having a quorum of supporters. Each
> server
> > is going to follow a single leader, so I don't see a problem in your
> > scenario
> > with the information you provided. Now if you tell me that when you keep
> > sending new transactions to those leaders, both keep committing new
> > transactions (not the same txns), then we have a problem. I don't see how
> > this can happen, though.
> >
> > Also, one of the leaders should eventually time out and go back to leader
> > election.
> >
> >> -----Original Message-----
> >> From: FPJ [mailto:[EMAIL PROTECTED]]
> >> Sent: 14 June 2013 21:44
> >> To: [EMAIL PROTECTED]
> >> Subject: RE: Rolling config change considered harmful?
> >>
> >> It is ok to have two servers thinking they are leaders as long as only
> one
> > is
> >> able to commit txns at a time by having a quorum of supporters. Each
> > server
> >> is going to follow a single leader, so I don't see a problem in your
> > scenario
> >> with the information you provided. Now if you tell me that when you keep
> >> sending new transactions to those leaders and they keep committing them
> >> forever, both keep committing new transactions, then we have a problem.
> I
> >> don't see how this can happen, though.
> >>
> >> Also, one of the leaders should eventually time out and go back to
> leader
> >> election.
> >>
> >> -Flavio
> >>
> >>> -----Original Message-----
> >>> From: Jordan Zimmerman [mailto:[EMAIL PROTECTED]]
> >>> Sent: 14 June 2013 21:10
> >>> To: [EMAIL PROTECTED]
> >>> Subject: Re: Rolling config change considered harmful?
> >>>
> >>> More on this.
> >>>
> >