Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Performing no downtime hardware changes to a live zookeeper cluster


Copy link to this message
-
Re: Performing no downtime hardware changes to a live zookeeper cluster
How are your clients configured to find the zks now? How many clients do
you have?

>From my phone
On Dec 20, 2011 3:14 PM, "Neha Narkhede" <[EMAIL PROTECTED]> wrote:

> Hi,
>
> As part of upgrading to Zookeeper 3.3.4, we also have to migrate our
> zookeeper cluster to new hardware. I'm trying to figure out the best
> strategy to achieve that with no downtime.
> Here are some possible solutions I see at the moment, I could have
> missed a few though -
>
> 1. Swap each machine out with a new machine, but with the same host/IP.
>
> Pros: No client side config needs to be changed.
> Cons: Relatively tedious task for Operations
>
> 2. Add new machines, with different host/IPs to the existing cluster,
> and remove the older machines, taking care to maintain the quorum at
> all times
>
> Pros: Easier for Operations
> Cons: Client side configs need to be changed and clients need to be
> restarted/bounced. Another problem is having a large quorum for
> sometime (potentially 9 nodes).
>
> 3. Hide the new cluster behind either a Hardware load balancer or a
> DNS server resolving to all host ips.
>
> Pros: Makes it easier to move hardware around in the future
> Cons: Possible timeout issues with load balancers messing with
> zookeeper functionality or performance
>
> Read this and found it helpful -
>
> http://apache.markmail.org/message/44tbj53q2jufplru?q=load+balancer+list:org%2Eapache%2Ehadoop%2Ezookeeper-user&page=1
> But would like to hear from the authors and the users who might have
> tried this in a real production setup.
>
> I'm very interested in finding a long term solution for masking the
> zookeeper host names. Any inputs here are appreciated !
>
> In addition to this, it will also be great to know what people think
> about options 1 and 2, as a solution for hardware changes in
> Zookeeper.
>
> Thanks,
> Neha
>