Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # dev - Performing no downtime hardware changes to a live zookeeper cluster


Copy link to this message
-
Re: Performing no downtime hardware changes to a live zookeeper cluster
Ted Dunning 2011-12-20, 21:06
In particular, aren't you using DNS names?  If you are, then you can

- expand the quorum with the new hardware on new IP addresses,
- switch DNS,
- wait for caches to die,
- restart applications without reconfig or otherwise force new connections,
- decrease quorum size again

On Tue, Dec 20, 2011 at 12:26 PM, Camille Fournier <[EMAIL PROTECTED]>wrote:

> How are your clients configured to find the zks now? How many clients do
> you have?
>
> From my phone
> On Dec 20, 2011 3:14 PM, "Neha Narkhede" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > As part of upgrading to Zookeeper 3.3.4, we also have to migrate our
> > zookeeper cluster to new hardware. I'm trying to figure out the best
> > strategy to achieve that with no downtime.
> > Here are some possible solutions I see at the moment, I could have
> > missed a few though -
> >
> > 1. Swap each machine out with a new machine, but with the same host/IP.
> >
> > Pros: No client side config needs to be changed.
> > Cons: Relatively tedious task for Operations
> >
> > 2. Add new machines, with different host/IPs to the existing cluster,
> > and remove the older machines, taking care to maintain the quorum at
> > all times
> >
> > Pros: Easier for Operations
> > Cons: Client side configs need to be changed and clients need to be
> > restarted/bounced. Another problem is having a large quorum for
> > sometime (potentially 9 nodes).
> >
> > 3. Hide the new cluster behind either a Hardware load balancer or a
> > DNS server resolving to all host ips.
> >
> > Pros: Makes it easier to move hardware around in the future
> > Cons: Possible timeout issues with load balancers messing with
> > zookeeper functionality or performance
> >
> > Read this and found it helpful -
> >
> >
> http://apache.markmail.org/message/44tbj53q2jufplru?q=load+balancer+list:org%2Eapache%2Ehadoop%2Ezookeeper-user&page=1
> > But would like to hear from the authors and the users who might have
> > tried this in a real production setup.
> >
> > I'm very interested in finding a long term solution for masking the
> > zookeeper host names. Any inputs here are appreciated !
> >
> > In addition to this, it will also be great to know what people think
> > about options 1 and 2, as a solution for hardware changes in
> > Zookeeper.
> >
> > Thanks,
> > Neha
> >
>