After 2 hard restarts of all nodes, the cluster is back. But 7 of 18 nodes are dead. the NN is aware of all nodes, data (of non-dead) nodes is available.
We're pulling off data now, will give hardware back to the HW team next week.
*the emails in response to my original are very helpful in learning more about Hadoop concepts.many thanksJohn
Date: Sat, 23 Mar 2013 13:45:18 +0800
Subject: Re: Cluster lost IP addresses
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <[EMAIL PROTECTED]> wrote:
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>> On Friday, March 22, 2013, John Meza wrote:
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>> Is there a way to recover this cluster to a useable state?
John Meza 2013-03-22, 21:50
Balaji Narayanan 2013-03-23, 02:14
Azuryy Yu 2013-03-23, 03:31
Harsh J 2013-03-23, 05:12
Chris Embree 2013-03-23, 05:22
Harsh J 2013-03-23, 05:42