Can you post your namenode's log ? It seems that your data node can not
connect to the name node.

On Wed, Mar 17, 2010 at 2:43 PM, William Kang <[EMAIL PROTECTED]>wrote:

> Hi,
> I just moved from pseudo distributed hadoop to a four machine full
> distributed hadoop setup.
>
> But, after I start the dfs, there is no live node showing up. If I make
> master a slave too, then the datanode in master machine will show up.
>
> I looked up all logs and found no errors. The only thing
> looks suspicious  is the log in the datanode:
>
>
> ************************************
> 2010-03-17 02:39:04,003 INFO org.apache.hadoop.ipc.RPC: Server at
> /xx.xx.xx.xx:9000 not available yet, Zzzzz...
> 2010-03-17 02:39:06,064 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 0 time(s).
> 2010-03-17 02:39:07,076 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 1 time(s).
> 2010-03-17 02:39:08,081 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 2 time(s).
> 2010-03-17 02:39:09,098 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx6:9000. Already tried 3 time(s).
> 2010-03-17 02:39:10,159 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 4 time(s).
> 2010-03-17 02:39:11,179 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 5 time(s).
> 2010-03-17 02:39:12,221 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 6 time(s).
> 2010-03-17 02:39:13,372 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 7 time(s).
> 2010-03-17 02:39:14,545 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 8 time(s).
> 2010-03-17 02:39:15,558 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /xx.xx.xx.xx:9000. Already tried 9 time(s).
> *************************************
>
> Does anybody know what might cause this problem?
>
> ssh among these machines are fine without password. The owner of hadoop
> folder has been changed to the same hadoop user.
>
> Thanks!
>
>
> William
>

--
Best Regards

Jeff Zhang

Hi Jeff,
Here is the log from my namenode:
************************************************************/
2010-03-17 03:09:59,750 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubtserver01/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-03-17 03:09:59,903 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=9000
2010-03-17 03:09:59,909 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
ubtserver01/127.0.0.1:9000
2010-03-17 03:09:59,912 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2010-03-17 03:09:59,914 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing
NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-03-17 03:09:59,979 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=cakang,cakang,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
2010-03-17 03:09:59,980 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2010-03-17 03:09:59,980 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2010-03-17 03:09:59,995 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-03-17 03:09:59,998 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-03-17 03:10:00,042 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 1
2010-03-17 03:10:00,048 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 0
2010-03-17 03:10:00,048 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 96 loaded in 0 seconds.
2010-03-17 03:10:00,048 INFO org.apache.hadoop.hdfs.server.common.Storage:
Edits file /opt/hadoop/dfs/name/current/edits of size 4 edits # 0 loaded in
0 seconds.
2010-03-17 03:10:00,122 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 96 saved in 0 seconds.
2010-03-17 03:10:00,391 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
FSImage in 436 msecs
2010-03-17 03:10:00,393 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks
= 0
2010-03-17 03:10:00,393 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid
blocks = 0
2010-03-17 03:10:00,393 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
under-replicated blocks = 0
2010-03-17 03:10:00,393 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
 over-replicated blocks = 0
2010-03-17 03:10:00,393 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Leaving safe mode after 0 secs.
2010-03-17 03:10:00,393 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Network topology has 0 racks and 0 datanodes
2010-03-17 03:10:00,393 INFO org.apache.hadoop.hdfs.StateChange: STATE*
UnderReplicatedBlocks has 0 blocks
2010-03-17 03:10:05,572 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-03-17 03:10:05,661 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50070
2010-03-17 03:10:05,663 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50070
webServer.getConnectors()[0].getLocalPort() returned 50070
2010-03-17 03:10:05,663 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50070
2010-03-17 03:10:05,663 INFO org.mortbay.log: jetty-6.1.14
2010-03-17 03:11:10,095 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50070
2010-03-17 03:11:10,095 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
0.0.0.0:50070
2010-03-17 03:11:10,097 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-03-17 03:11:10,098 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 9000: starting
2010-03-17 03:11:10,110 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 9000: starting
2010-03-17 03:11:10,113 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 9000: starting
2010-03-17 03:11:10,114 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 9000: starting
2010-03-17 03:11:10,153 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 9000: starting
2010-03-17 03:11:10,153 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 9000: starting
2010-03-17 03:11:10,160 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 9000: starting
2010-03-17 03:11:10,160 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 9000: starting
2010-03-17 03:11:10,160 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 9000: starting
2010-03-17 03:11:10,161 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 9000: starting
2010-03-17 03:11:10,170 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 9000: starting
2010-03-17 03:15:51,270 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.registerDatanode: node registration from 127.0.0.1:50010 storage
DS-37429514-127.0.0.1-50010-1268798883208
2010-03-17 03:15:51,274 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/127.0.0.1:50010

Thanks for the replies.
I am looking forward to hearing from you.
William

On Wed, Mar 17, 2010 at 3:14 AM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

Hi Jeff,
I think I partly found out the reasons of this problem. The /etc/hosts
127.0.0.1 has the master's host name in it. And the namenode took 127.0.0.1
as the ip address of the namenode. I fixed it and I already found two nodes.
There is one still missing. I will let you guys know what happened.
Thanks.
William

On Wed, Mar 17, 2010 at 3:14 AM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> Can you post your namenode's log ? It seems that your data node can not
> connect to the name node.
>
> On Wed, Mar 17, 2010 at 2:43 PM, William Kang <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> > I just moved from pseudo distributed hadoop to a four machine full
> > distributed hadoop setup.
> >
> > But, after I start the dfs, there is no live node showing up. If I make
> > master a slave too, then the datanode in master machine will show up.
> >
> > I looked up all logs and found no errors. The only thing
> > looks suspicious  is the log in the datanode:
> >
> >
> > ************************************
> > 2010-03-17 02:39:04,003 INFO org.apache.hadoop.ipc.RPC: Server at
> > /xx.xx.xx.xx:9000 not available yet, Zzzzz...
> > 2010-03-17 02:39:06,064 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 0 time(s).
> > 2010-03-17 02:39:07,076 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 1 time(s).
> > 2010-03-17 02:39:08,081 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 2 time(s).
> > 2010-03-17 02:39:09,098 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx6:9000. Already tried 3 time(s).
> > 2010-03-17 02:39:10,159 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 4 time(s).
> > 2010-03-17 02:39:11,179 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 5 time(s).
> > 2010-03-17 02:39:12,221 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 6 time(s).
> > 2010-03-17 02:39:13,372 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 7 time(s).
> > 2010-03-17 02:39:14,545 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 8 time(s).
> > 2010-03-17 02:39:15,558 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: /xx.xx.xx.xx:9000. Already tried 9 time(s).
> > *************************************
> >
> > Does anybody know what might cause this problem?
> >
> > ssh among these machines are fine without password. The owner of hadoop
> > folder has been changed to the same hadoop user.
> >
> > Thanks!
> >
> >
> > William
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

William Kang wrote:
> Hi Jeff,
> I think I partly found out the reasons of this problem. The /etc/hosts
> 127.0.0.1 has the master's host name in it. And the namenode took 127.0.0.1
> as the ip address of the namenode. I fixed it and I already found two nodes.
> There is one still missing. I will let you guys know what happened.
> Thanks.
>

ongoing issue w/ Ubuntu. It's a feature designed to help laptops and the
like, useless for when you want to bring up servers visible from the
outside and addressed by hostname, which is what Hadoop and things like
Java RMI like.

see http://linux.derkeiler.com/Mailing-Lists/Ubuntu/2007-08/msg00681.html
http://ubuntuforums.org/showthread.php?t=432875
https://lists.ubuntu.com/archives/ubuntu-users/2008-December/168883.html
I've debated fixing this at a pre-hadoop level in my code by having
checks for the hostname and bailing out early if your hostname maps to
::1 or 127.*.*.1, because neither are things that are useful off the
specific host:
http://jira.smartfrog.org/jira/browse/SFOS-1184
Something could go into Hadoop directly, but either way you have a
problem that hostname/DNS problems can be fairly tricky to identify in a
timely manner, let alone handle.

-steve