Thanks Stack. Do you have any specific pointers as to what configs would help mitigate this issue with a DHCP setup (I am not a networking expert, other teams manage the network and if I have specific pointers that would help guide the discussion)
From: Stack <[EMAIL PROTECTED]>
To: Hbase-User <[EMAIL PROTECTED]>; Dhaval Shah <[EMAIL PROTECTED]>
Sent: Tuesday, 6 August 2013 1:29 PM
Subject: Re: NoRouteToHostException when zookeeper crashes
On Tue, Aug 6, 2013 at 7:48 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
> I have a weird (and a pretty serious) issue on my HBase cluster. Whenever
> one of my zookeeper server goes down, already running services work fine
> for a few hours but when I try to restart any service (be it region servers
> or clients), they fail with a NoRouteToHostException while trying to
> connect to zookeeper and I cannot restart any service successfully at all.
> I do realize that No Route to host is coming from my network infrastructure
> (ping gives the same error) but why would 1 zookeeper server going down
> bring down the entire HBase cluster. Why doesn't HBase ride over the
> exception and try some other zookeeper server?
> Is this an issue other people face or its just me? We are running these on
> DHCP (but the IPs don't change because we have long leases). Do you guys
> think its a DHCP specific issue? Do you have pointers to avoid this issue
> with DHCP or do I have to move to static IPs?
All bets are off in the face of NoRouteToHost. Please fixup your
networking (My guess is first lookup works and gets cached. On restart, we
run into your network issue).