Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> can't format namenode....


Copy link to this message
-
can't format namenode....
Hi guys : In order to fix some issues im having (recently posted), I've
decided to try to make sure my name node is formatted.... But the formatting
fails (see 1 below) .....

So... To trace the failure,  I figured I would grep through all log files
for exceptions.
I've curated the results here ... does this look familiar to anyone ?

Clearly, something is very wrong with my CDH hadoop installation.

1) To attempt to solve this, I figured I would format my namenode.  Oddly,
when I run "hadoop -namenode format" I get the following stack trace :

11/10/29 14:39:37 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2-cdh3u1
STARTUP_MSG:   build = file:///tmp/topdir/BUILD/hadoop-0.20.2-cdh3u1 -r
bdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638; compiled by 'root' on Mon Jul 18
09:40:22 PDT 2011
************************************************************/
Re-format filesystem in /var/lib/hadoop-0.20/cache/hadoop/dfs/name ? (Y or
N) Y
11/10/29 14:39:40 INFO util.GSet: VM type       = 64-bit
11/10/29 14:39:40 INFO util.GSet: 2% max memory = 19.33375 MB
11/10/29 14:39:40 INFO util.GSet: capacity      = 2^21 = 2097152 entries
11/10/29 14:39:40 INFO util.GSet: recommended=2097152, actual=2097152
11/10/29 14:39:40 INFO namenode.FSNamesystem: fsOwner=cloudera
11/10/29 14:39:40 INFO namenode.FSNamesystem: supergroup=supergroup
11/10/29 14:39:40 INFO namenode.FSNamesystem: isPermissionEnabled=false
11/10/29 14:39:40 INFO namenode.FSNamesystem:
dfs.block.invalidate.limit=1000
11/10/29 14:39:40 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/10/29 14:39:41 ERROR namenode.NameNode: java.io.IOException: Cannot
remove current directory: /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current
    at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:303)
    at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1244)
    at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1263)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1100)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1217)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)

11/10/29 14:39:41 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/
2) Here are the exceptions (abridged , i removed repetetive parts regarding
"replicated to 0 nodes instead of 1"

2011-10-28 22:36:52,669 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 8020, call
addBlock(/var/lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info,
DFSClient_-134960056, null) from 127.0.0.1:35163: error:
java.io.IOException: File /var
lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info could only be
replicated to 0 nodes, instead of 1
java.io.IOException: File /var/lib/hadoop-0.20/cache/mapred/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
STARTUP_MSG:   host = java.net.UnknownHostException: gcrc15.uchc.net:
gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
2011-10-28 22:30:03,413 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/var/lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info could only
be replicated to 0 nodes, in
tead of 1
.......................... REPEATED SEVERAL TIMES ......................
2011-10-28 22:36:52,716 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/var/lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info could only
be replicated to 0 nodes, in
tead of 1
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/var/lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info could only
be replicated to 0 nodes, instead of 1
STARTUP_MSG:   host = java.net.UnknownHostException: gcrc15.uchc.net:
gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
STARTUP_MSG:   host = java.net.UnknownHostException: gcrc15.uchc.net:
gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
2011-10-28 22:29:48,131 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
SHUTDOWN_MSG: Shutting down DataNode at java.net.UnknownHostException:
gcrc15.uchc.net: gcrc15.uchc.net
STARTUP_MSG:   host = java.net.UnknownHostException: gcrc15.uchc.net:
gcrc15.uchc.net
java.net.UnknownHostException: gcrc15.uchc.net: gcrc15.uchc.net
2011-10-28 22:30:03,406 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 8020, call
addBlock(/var/lib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info,
DFSClient_157537399, null) from 127.0.0.1:33519: error: java.io.IOException:
File /var/
ib/hadoop-0.20/cache/mapred/mapred/system/jobtracker.info could only be
replicated to 0 nodes, instead of 1
java.io.IOException: File /var/lib/hadoop-0.20/cache/mapred/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
...............................(repeated several times)
.................................
STARTUP_MSG:   host = java.net.UnknownHostException: gcrc15.uchc.net:
gcrc15.uchc.net
java.net.UnknownHostE
+
Uma Maheswara Rao G 72686... 2011-10-29, 16:53