-RE: Namenode formatting problem
Marcin Mejran 2013-02-20, 01:46
The issue may be that the nodes are trying to use the ec2 public ip (which would be used for external access) to access each other which does not work (or doesn't work trivially). You need to use the private ips which are given by ifconfig.
ec2 gives you static ips as long as you don't restart or stop/start an instance.
That said, it gives you TWO ips and you need to be careful on which you use:
* Private ip: This is a local ip that cannot be access from outside ec2 but can be used to communicate between instance. This is what ifconfig returns.
* Public ip: This ip can be used for external access but it's not shown in ifconfig.
You can imagine each instance as having its own personal NAT.
Instances should use the private ip when communicating to each other. I'm not sure if the public ip cannot be used or is just a giant pain to setup correctly. I just did a test on my company ec2 instances and I can ping the public ip from an instance but I cannot use it for ssh. Not sure why off hand although I believe any data sent to a public ip (even from within ec2) gets charged transfer fees so it's not a good idea in any regard.
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, February 19, 2013 11:25 AM
To: <[EMAIL PROTECTED]>
Subject: Re: Namenode formatting problem
To simplify my previous post, your IPs for the master/slave/etc. in /etc/hosts file should match the ones reported by "ifconfig" always.
In proper deployments, IP is static. If IP is dynamic, we'll need to think of some different ways.
On Tue, Feb 19, 2013 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hey Keith,
> I'm guessing whatever "ip-13-0-177-110" is resolving to (ping to
> check), is not what is your local IP on that machine (or rather, it
> isn't the machine you intended to start it on)?
> Not sure if EC2 grants static IPs, but otherwise a change in the
> assigned IP (checkable via ifconfig) would probably explain the
> "Cannot assign" error received when we tried a bind() syscall.
> On Tue, Feb 19, 2013 at 4:30 AM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>> This is Hadoop 2.0. Formatting the namenode produces no errors in the shell, but the log shows this:
>> 2013-02-18 22:19:46,961 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> namenode join
>> java.net.BindException: Problem binding to [ip-13-0-177-110:9212] java.net.BindException: Cannot assign requested address; For more details see: http://wiki.apache.org/hadoop/BindException
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:710)
>> at org.apache.hadoop.ipc.Server.bind(Server.java:356)
>> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:454)
>> at org.apache.hadoop.ipc.Server.<init>(Server.java:1833)
>> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:866)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:375)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:350)
>> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695)
>> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:238)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:452)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> 2013-02-18 22:19:46,988 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>> 2013-02-18 22:19:46,990 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: