Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Performing no downtime hardware changes to a live zookeeper cluster


Copy link to this message
-
Performing no downtime hardware changes to a live zookeeper cluster
Hi,

As part of upgrading to Zookeeper 3.3.4, we also have to migrate our
zookeeper cluster to new hardware. I'm trying to figure out the best
strategy to achieve that with no downtime.
Here are some possible solutions I see at the moment, I could have
missed a few though -

1. Swap each machine out with a new machine, but with the same host/IP.

Pros: No client side config needs to be changed.
Cons: Relatively tedious task for Operations

2. Add new machines, with different host/IPs to the existing cluster,
and remove the older machines, taking care to maintain the quorum at
all times

Pros: Easier for Operations
Cons: Client side configs need to be changed and clients need to be
restarted/bounced. Another problem is having a large quorum for
sometime (potentially 9 nodes).

3. Hide the new cluster behind either a Hardware load balancer or a
DNS server resolving to all host ips.

Pros: Makes it easier to move hardware around in the future
Cons: Possible timeout issues with load balancers messing with
zookeeper functionality or performance

Read this and found it helpful -
http://apache.markmail.org/message/44tbj53q2jufplru?q=load+balancer+list:org%2Eapache%2Ehadoop%2Ezookeeper-user&page=1
But would like to hear from the authors and the users who might have
tried this in a real production setup.

I'm very interested in finding a long term solution for masking the
zookeeper host names. Any inputs here are appreciated !

In addition to this, it will also be great to know what people think
about options 1 and 2, as a solution for hardware changes in
Zookeeper.

Thanks,
Neha