Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # dev - Performing no downtime hardware changes to a live zookeeper cluster


+
Neha Narkhede 2011-12-20, 20:14
+
Camille Fournier 2011-12-20, 20:26
+
Ted Dunning 2011-12-20, 21:06
+
Neha Narkhede 2011-12-22, 01:43
+
Camille Fournier 2011-12-22, 03:21
+
Neha Narkhede 2012-01-09, 18:15
+
Camille Fournier 2012-01-09, 18:31
+
Neha Narkhede 2012-01-09, 18:51
+
Camille Fournier 2012-01-09, 19:04
+
Neha Narkhede 2012-01-09, 20:33
+
Camille Fournier 2012-01-09, 20:47
Copy link to this message
-
RE: Performing no downtime hardware changes to a live zookeeper cluster
Alexander Shraer 2012-01-09, 22:23
Hi Neha, Camille,

I wanted to share something we did as part of the dynamic membership change feature development (ZK-107) that seems very related to the discussion here and might solve the problem.

When the membership changes, similarly to what you wrote below, clients sometimes need to move. Obviously this is the case if the server they are connected to is no longer in the cluster. The idea is that they should move in a way that minimizes unnecessary client migration (such as the one that would happen if you have every client re-shuffle the new server list) and yet leaves the system in a balance state (the number of clients connected to each server is the same in expectation).

The idea was to come up with a set of probabilistic rules that each client applies locally to see whether and where it should migrate.

The resulting rules as well as the evaluation in Zookeeper are in the document attached to the jira (https://issues.apache.org/jira/browse/ZOOKEEPER-1355)
I'll provide a patch soon. (this is part of a larger paper about reconfiguration that is under submission).

In terms of implementation changes, I added a command "zk.updateServerList(hostlist);"
and an implantation of the probabilistic rules in StaticHostProvider.

I wanted to get your feedback and hopefully this may solve the problem you're discussing here.

Thanks a lot,
Alex
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Camille Fournier
> Sent: Monday, January 09, 2012 12:47 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Performing no downtime hardware changes to a live
> zookeeper cluster
>
> Sounds fine with me, probably should make it a flaggable option.
>
> C
>
>
> On Mon, Jan 9, 2012 at 3:33 PM, Neha Narkhede
> <[EMAIL PROTECTED]>wrote:
>
> > >> If you just have machine names in a list that you pass in, then
> yes, we
> > could re-resolve on every reconnect and you could just re-alias that
> name
> > to a new IP. But you'll have to put in logic that will do that but
> not
> > break people using DNS RR.
> >
> > Having a list of machine names that can be changed to point to new
> IPs
> > seems reasonable too. To be able to do the upgrade without having to
> > restart all clients, besides turning off DNS caching in the JVM, we
> > still have to solve the problem of zookeeper client caching the IPs
> in
> > code. Having 2 levels of DNS caching, one in the JVM and one in code
> > (which cannot be turned off) doesn't look like a good idea. Unless
> I'm
> > missing the purpose of such IP caching in zookeeper ?
> >
> > >> I realize that moving machines is difficult when you have lots of
> > clients.
> > I'm a bit surprised your admins can't maintain machine IP addresses
> on a
> > machine move given a cluster of that complexity, though
> >
> > Its not like it can't be done, it definitely has quite some
> > operational overhead. We are trying to brainstorm various approaches
> > and come up with one that will involve the least overhead on such
> > upgrades going forward.
> >
> > Having said that, seems like re-resolving host names in reconnect
> > doesn't look like a bad idea, provided it doesn't break the DNS RR
> use
> > case. If that sounds good, can I go ahead a file a JIRA for this ?
> >
> > Thanks,
> > Neha
> >
> > On Mon, Jan 9, 2012 at 11:04 AM, Camille Fournier
> <[EMAIL PROTECTED]>
> > wrote:
> > > We don't shuffle IPs after the initial resolution of IP addresses.
> > >
> > > In DNS RR, you resolve to a list of IPs, shuffle these, and then we
> round
> > > robin through them trying to connect. If you re-resolve on every
> > > round-robin, you have to put in logic to know which ones have
> changed and
> > > somehow maintain that shuffle order or you aren't doing a fair back
> end
> > > round robin, which people using the ZK client against DNS RR are
> relying
> > on
> > > today.
> > >
> > > If you just have machine names in a list that you pass in, then
> yes, we
> > > could re-resolve on every reconnect and you could just re-alias
+
Ted Dunning 2012-01-09, 23:17
+
Patrick Hunt 2012-01-10, 01:36
+
Neha Narkhede 2012-01-10, 02:49
+
Patrick Hunt 2012-01-10, 16:39