I have a weird (and a pretty serious) issue on my HBase cluster. Whenever one of my zookeeper server goes down, already running services work fine for a few hours but when I try to restart any service (be it region servers or clients), they fail with a NoRouteToHostException while trying to connect to zookeeper and I cannot restart any service successfully at all. I do realize that No Route to host is coming from my network infrastructure (ping gives the same error) but why would 1 zookeeper server going down bring down the entire HBase cluster. Why doesn't HBase ride over the exception and try some other zookeeper server?
Is this an issue other people face or its just me? We are running these on DHCP (but the IPs don't change because we have long leases). Do you guys think its a DHCP specific issue? Do you have pointers to avoid this issue with DHCP or do I have to move to static IPs?