Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Rolling config change considered harmful?


+
Jordan Zimmerman 2013-06-14, 18:34
+
FPJ 2013-06-14, 19:07
+
Jordan Zimmerman 2013-06-14, 19:16
+
FPJ 2013-06-14, 19:27
+
Jordan Zimmerman 2013-06-14, 20:10
+
FPJ 2013-06-14, 20:44
+
FPJ 2013-06-14, 20:54
+
Jordan Zimmerman 2013-06-14, 21:46
Copy link to this message
-
Re: Rolling config change considered harmful?
Hello,

Could you please clarify if this thread is about a rolling start in an
ensemble without the dynamic reconfiguration support?
And when you say "Create a 5 node ensemble", that means quorum is 5. But
then you give server lists with only 3 servers in each node?
If the server list has 3 servers, then quorum is actually 3 and what is
described may happen in that scenario.
In that case C follows B, E follows D and A follows either B or D and there
are two working ensembles.
It should be possible to create problems, even with more standard
configuration changes:
If we want to change a quorum of three to a quorum of five {A,B,C} to
{A,B,C,D,E}:
- First the configuration is changed in all the nodes, but they are not
restarted. Only A, B and C are running.
- One of them is stopped (e.g. A).
- At this point, if A, D and E are started with the new configuration, they
may elect a leader before any of them is aware of either B or C, form an
ensemble and start serving txns.
- However, if A is started, we wait until it connects to the leader of B
and C, and then D and E are started and then B and C are restarted,
everything should be ok. The fact that this depends on the human ability to
start D and E while A,B and C are connected to the ensemble seems a bit
risky though.
I have found a presentation on the topic:
http://www.slideshare.net/Hadoop_Summit/dynamic-reconfiguration-of-zookeeper

If anybody knows of a safer way to change a quorum of 3 to a quorum of 5
with e.g. zookeeper 3.4.5, please point it out.

Regards,

Germán.
On Fri, Jun 14, 2013 at 11:46 PM, Jordan Zimmerman <
[EMAIL PROTECTED]> wrote:

> I got the test cluster into the state described with 2 leaders. I then
> allocated 100 Curator clients to write nodes "/n" where n is the index
> (i.e. "/0", "/1", …). The idea that the nodes would be distributed around
> the cluster instances. I then allocated a single Curator instance dedicated
> to one of the servers instance, did a sync, and did an exists() to verify
> that each cluster instances had all the nodes. For the 2 leader cluster,
> this fails.
>
> -JZ
>
> On Jun 14, 2013, at 1:54 PM, "FPJ" <[EMAIL PROTECTED]> wrote:
>
> > I messed up the last sentence, here is what I was trying to say:
> >
> > It is ok to have two servers thinking they are leaders as long as only
> one
> > is
> > able to commit txns at a time by having a quorum of supporters. Each
> server
> > is going to follow a single leader, so I don't see a problem in your
> > scenario
> > with the information you provided. Now if you tell me that when you keep
> > sending new transactions to those leaders, both keep committing new
> > transactions (not the same txns), then we have a problem. I don't see how
> > this can happen, though.
> >
> > Also, one of the leaders should eventually time out and go back to leader
> > election.
> >
> >> -----Original Message-----
> >> From: FPJ [mailto:[EMAIL PROTECTED]]
> >> Sent: 14 June 2013 21:44
> >> To: [EMAIL PROTECTED]
> >> Subject: RE: Rolling config change considered harmful?
> >>
> >> It is ok to have two servers thinking they are leaders as long as only
> one
> > is
> >> able to commit txns at a time by having a quorum of supporters. Each
> > server
> >> is going to follow a single leader, so I don't see a problem in your
> > scenario
> >> with the information you provided. Now if you tell me that when you keep
> >> sending new transactions to those leaders and they keep committing them
> >> forever, both keep committing new transactions, then we have a problem.
> I
> >> don't see how this can happen, though.
> >>
> >> Also, one of the leaders should eventually time out and go back to
> leader
> >> election.
> >>
> >> -Flavio
> >>
> >>> -----Original Message-----
> >>> From: Jordan Zimmerman [mailto:[EMAIL PROTECTED]]
> >>> Sent: 14 June 2013 21:10
> >>> To: [EMAIL PROTECTED]
> >>> Subject: Re: Rolling config change considered harmful?
> >>>
> >>> More on this.
> >>>
> >
+
Alexander Shraer 2013-06-15, 05:01
+
German Blanco 2013-06-15, 07:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB