Nick Klosterman
2010-02-10, 20:42
E. Sammer
2010-02-10, 20:49
Allen Wittenauer
2010-02-10, 20:49
Nick Klosterman
2010-02-10, 20:57
abhishek sharma
2010-02-10, 21:23
E. Sammer
2010-02-10, 22:09
Nick Klosterman
2010-02-10, 22:19
E. Sammer
2010-02-10, 22:26
Sonal Goyal
2010-02-12, 16:44
Aaron Kimball
2010-02-13, 01:10
Sonal Goyal
2010-02-13, 16:35
Todd Lipcon
2010-02-13, 17:37
Sonal Goyal
2010-02-13, 17:41
-
Ubuntu Single Node Tutorial failure. No live or dead nodes.Nick Klosterman 2010-02-10, 20:42
I've been following Michael Noll's Single node cluster tutorial but am
unable to run the wordcount example successfully.
It appears that I'm having some sort of problem involving the nodes. Using
copyFromLocal fails to replicate the data across 1 node.
When I try to look at the hadoop web interfaces I see that there aren't
any nodes (alive or dead) listed.
After I start things this is what I get from dfsadmin -report
/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
Here are the commands I'm entering and the output of them:
/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
/usr/local/hadoop/bin$ jps
24440 SecondaryNameNode
24626 TaskTracker
24527 JobTracker
24218 NameNode
24725 Jps
---> I had all ready created the txtinput directory with ./hadoop dfs
-mkdir txtinput
/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl
unable to run the wordcount example successfully.
It appears that I'm having some sort of problem involving the nodes. Using
copyFromLocal fails to replicate the data across 1 node.
When I try to look at the hadoop web interfaces I see that there aren't
any nodes (alive or dead) listed.
After I start things this is what I get from dfsadmin -report
/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
Here are the commands I'm entering and the output of them:
/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
/usr/local/hadoop/bin$ jps
24440 SecondaryNameNode
24626 TaskTracker
24527 JobTracker
24218 NameNode
24725 Jps
---> I had all ready created the txtinput directory with ./hadoop dfs
-mkdir txtinput
/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.E. Sammer 2010-02-10, 20:49
Nick:
It appears that the datanode daemon isn't running.
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
There's no process for DataNode. This is the process that is responsible
for storing blocks. In other words, no data node, no storage. The
namenode only manages the file system image and metadata.
Check the logs $HADOOP_HOME/logs/*datanode*.log for errors in start up.
Hope this helps.
On 2/10/10 3:42 PM, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes.
> Using copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't
> any nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs
> -mkdir txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
> /home/hadoop/Desktop/*.txt txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>
> at $Proxy0.addBlock(Unknown Source)
> at
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
It appears that the datanode daemon isn't running.
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
There's no process for DataNode. This is the process that is responsible
for storing blocks. In other words, no data node, no storage. The
namenode only manages the file system image and metadata.
Check the logs $HADOOP_HOME/logs/*datanode*.log for errors in start up.
Hope this helps.
On 2/10/10 3:42 PM, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes.
> Using copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't
> any nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs
> -mkdir txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
> /home/hadoop/Desktop/*.txt txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>
> at $Proxy0.addBlock(Unknown Source)
> at
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Allen Wittenauer 2010-02-10, 20:49
On 2/10/10 12:42 PM, "Nick Klosterman" <[EMAIL PROTECTED]> wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes. Using
> copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't
> any nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
You have no datanodes. Check the *out and *log files for the datanode
processes.
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Nick Klosterman 2010-02-10, 20:57
It appears I have incompatible namespaceIDs. Any thoughts on how to
resolve that?
This is what the full datanodes log is saying:
2010-02-10 15:25:09,125 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
2010-02-10 15:25:13,785 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID
= 2082816383; datanode namespaceID = 1109869136
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
2010-02-10 15:25:13,786 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
************************************************************/
On Wed, 10 Feb 2010, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes. Using
> copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't any
> nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs -mkdir
> txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt
> txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes, instead
> of 1
resolve that?
This is what the full datanodes log is saying:
2010-02-10 15:25:09,125 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
2010-02-10 15:25:13,785 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID
= 2082816383; datanode namespaceID = 1109869136
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
2010-02-10 15:25:13,786 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
************************************************************/
On Wed, 10 Feb 2010, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes. Using
> copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't any
> nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs -mkdir
> txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt
> txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes, instead
> of 1
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.abhishek sharma 2010-02-10, 21:23
So Michael Noll's tutorial page has the following tips for the error
you are facing.
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)#java.io.IOException:_Incompatible_namespaceIDs
Abhishek
On Wed, Feb 10, 2010 at 12:57 PM, Nick Klosterman
<[EMAIL PROTECTED]> wrote:
> It appears I have incompatible namespaceIDs. Any thoughts on how to resolve
> that?
> This is what the full datanodes log is saying:
>
> 2010-02-10 15:25:09,125 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = potr134pc26/127.0.0.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.1
> STARTUP_MSG: build > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
> ************************************************************/
> 2010-02-10 15:25:13,785 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in
> /home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID > 2082816383; datanode namespaceID = 1109869136
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
>
> 2010-02-10 15:25:13,786 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
> ************************************************************/
>
>
> On Wed, 10 Feb 2010, Nick Klosterman wrote:
>
>> I've been following Michael Noll's Single node cluster tutorial but am
>> unable to run the wordcount example successfully.
>>
>> It appears that I'm having some sort of problem involving the nodes. Using
>> copyFromLocal fails to replicate the data across 1 node.
>> When I try to look at the hadoop web interfaces I see that there aren't
>> any nodes (alive or dead) listed.
>>
>> After I start things this is what I get from dfsadmin -report
>> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
>> Configured Capacity: 0 (0 KB)
>> Present Capacity: 0 (0 KB)
>> DFS Remaining: 0 (0 KB)
>> DFS Used: 0 (0 KB)
>> DFS Used%: %
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 0 (0 total, 0 dead)
>>
>>
>> Here are the commands I'm entering and the output of them:
>>
>> /usr/local/hadoop/bin$ ./start-all.sh
>> starting namenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
>> localhost: starting datanode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
>> localhost: starting secondarynamenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>> starting jobtracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
>> localhost: starting tasktracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>>
>> /usr/local/hadoop/bin$ jps
>> 24440 SecondaryNameNode
>> 24626 TaskTracker
>> 24527 JobTracker
>> 24218 NameNode
>> 24725 Jps
>>
>> ---> I had all ready created the txtinput directory with ./hadoop dfs
you are facing.
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)#java.io.IOException:_Incompatible_namespaceIDs
Abhishek
On Wed, Feb 10, 2010 at 12:57 PM, Nick Klosterman
<[EMAIL PROTECTED]> wrote:
> It appears I have incompatible namespaceIDs. Any thoughts on how to resolve
> that?
> This is what the full datanodes log is saying:
>
> 2010-02-10 15:25:09,125 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = potr134pc26/127.0.0.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.1
> STARTUP_MSG: build > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
> ************************************************************/
> 2010-02-10 15:25:13,785 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in
> /home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID > 2082816383; datanode namespaceID = 1109869136
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
>
> 2010-02-10 15:25:13,786 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
> ************************************************************/
>
>
> On Wed, 10 Feb 2010, Nick Klosterman wrote:
>
>> I've been following Michael Noll's Single node cluster tutorial but am
>> unable to run the wordcount example successfully.
>>
>> It appears that I'm having some sort of problem involving the nodes. Using
>> copyFromLocal fails to replicate the data across 1 node.
>> When I try to look at the hadoop web interfaces I see that there aren't
>> any nodes (alive or dead) listed.
>>
>> After I start things this is what I get from dfsadmin -report
>> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
>> Configured Capacity: 0 (0 KB)
>> Present Capacity: 0 (0 KB)
>> DFS Remaining: 0 (0 KB)
>> DFS Used: 0 (0 KB)
>> DFS Used%: %
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 0 (0 total, 0 dead)
>>
>>
>> Here are the commands I'm entering and the output of them:
>>
>> /usr/local/hadoop/bin$ ./start-all.sh
>> starting namenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
>> localhost: starting datanode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
>> localhost: starting secondarynamenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>> starting jobtracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
>> localhost: starting tasktracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>>
>> /usr/local/hadoop/bin$ jps
>> 24440 SecondaryNameNode
>> 24626 TaskTracker
>> 24527 JobTracker
>> 24218 NameNode
>> 24725 Jps
>>
>> ---> I had all ready created the txtinput directory with ./hadoop dfs
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.E. Sammer 2010-02-10, 22:09
On 2/10/10 3:57 PM, Nick Klosterman wrote:
> It appears I have incompatible namespaceIDs. Any thoughts on how to
> resolve that?
> This is what the full datanodes log is saying:
Was this data node part of a another DFS cluster at some point? It looks
like you've reformatted the name node since the datanode connected to
it. The datanode will refuse to connect to a namenode with a different
namespaceId because the the data node would have blocks (possibly with
the same ids) from another cluster. It's a stop gap safety mechanism.
You'd have to destroy the data directory on the data node to
"reinitialize" it so it picks up the new namespaceId from the name node
at which point it will be allowed to connect.
Just to be clear, this will also kill all data that was stored on the
data node, so don't do this lightly.
HTH.
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
> It appears I have incompatible namespaceIDs. Any thoughts on how to
> resolve that?
> This is what the full datanodes log is saying:
Was this data node part of a another DFS cluster at some point? It looks
like you've reformatted the name node since the datanode connected to
it. The datanode will refuse to connect to a namenode with a different
namespaceId because the the data node would have blocks (possibly with
the same ids) from another cluster. It's a stop gap safety mechanism.
You'd have to destroy the data directory on the data node to
"reinitialize" it so it picks up the new namespaceId from the name node
at which point it will be allowed to connect.
Just to be clear, this will also kill all data that was stored on the
data node, so don't do this lightly.
HTH.
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Nick Klosterman 2010-02-10, 22:19
@E.Sammer, no I don't *think* that it is part of another cluster. The
tutorial is for a single node cluster just as a initial set up to see if
you can get things up and running. I have reformatted the namenode
several times in my effort to get hadoop to work.
@abishek
I tried the workaround you pointed me to no avail. I tried to modify
those directions since in the single node implementation I didn't have a
dfs.data.dir in hdfs-site.xml
My attempts at further debug
-----------------------------------------ATTEMPT AT FIXING THE DATANODES
PROBLEM
hadoop@potr134pc26:/usr/local/hadoop/bin$ rm -r
/usr/local/hadoop-datastore/
----NOW THERE IS NO HADOOP-DATASTORE FOLDER LOCALLY
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop namenode -format
10/02/10 16:33:50 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
Re-format filesystem in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name ? (Y or N) Y
10/02/10 16:33:54 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
10/02/10 16:33:54 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/10 16:33:54 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/10 16:33:54 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/10 16:33:54 INFO common.Storage: Storage directory
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name has been successfully
formatted.
10/02/10 16:33:54 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at potr134pc26/127.0.0.1
************************************************************/
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
27461 Jps
27354 TaskTracker
27158 SecondaryNameNode
27250 JobTracker
26923 NameNode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
---(((( AT THIS POINT WHEN I CHECKED THE LOG THE DATANODES STILL WASN'T UP
AND RUNNING)----------
mkdir /usr/local/hadoop-datastore
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: no datanode to stop
localhost: stopping secondarynamenode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
28038 NameNode
28536 Jps
28154 DataNode
28365 JobTracker
28470 TaskTracker
28272 SecondaryNameNode
./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt txtinput
copyFromLocal: `txtinput': specified destination directory doest not exist
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -mkdir txtinput
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 16:44:36 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.inv
tutorial is for a single node cluster just as a initial set up to see if
you can get things up and running. I have reformatted the namenode
several times in my effort to get hadoop to work.
@abishek
I tried the workaround you pointed me to no avail. I tried to modify
those directions since in the single node implementation I didn't have a
dfs.data.dir in hdfs-site.xml
My attempts at further debug
-----------------------------------------ATTEMPT AT FIXING THE DATANODES
PROBLEM
hadoop@potr134pc26:/usr/local/hadoop/bin$ rm -r
/usr/local/hadoop-datastore/
----NOW THERE IS NO HADOOP-DATASTORE FOLDER LOCALLY
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop namenode -format
10/02/10 16:33:50 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
Re-format filesystem in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name ? (Y or N) Y
10/02/10 16:33:54 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
10/02/10 16:33:54 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/10 16:33:54 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/10 16:33:54 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/10 16:33:54 INFO common.Storage: Storage directory
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name has been successfully
formatted.
10/02/10 16:33:54 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at potr134pc26/127.0.0.1
************************************************************/
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
27461 Jps
27354 TaskTracker
27158 SecondaryNameNode
27250 JobTracker
26923 NameNode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
---(((( AT THIS POINT WHEN I CHECKED THE LOG THE DATANODES STILL WASN'T UP
AND RUNNING)----------
mkdir /usr/local/hadoop-datastore
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: no datanode to stop
localhost: stopping secondarynamenode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
28038 NameNode
28536 Jps
28154 DataNode
28365 JobTracker
28470 TaskTracker
28272 SecondaryNameNode
./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt txtinput
copyFromLocal: `txtinput': specified destination directory doest not exist
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -mkdir txtinput
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 16:44:36 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.inv
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.E. Sammer 2010-02-10, 22:26
On 2/10/10 5:19 PM, Nick Klosterman wrote:
> @E.Sammer, no I don't *think* that it is part of another cluster. The
> tutorial is for a single node cluster just as a initial set up to see if
> you can get things up and running. I have reformatted the namenode
> several times in my effort to get hadoop to work.
What I mean is that the data node, at some point, connected to your name
node. If you reformat the name node, the data node must be wiped clean;
it's effectively trying to join a name node that no longer exists.
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
> @E.Sammer, no I don't *think* that it is part of another cluster. The
> tutorial is for a single node cluster just as a initial set up to see if
> you can get things up and running. I have reformatted the namenode
> several times in my effort to get hadoop to work.
What I mean is that the data node, at some point, connected to your name
node. If you reformat the name node, the data node must be wiped clean;
it's effectively trying to join a name node that no longer exists.
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Sonal Goyal 2010-02-12, 16:44
Hi
I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
start-all script to introduce a sleep time :
bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
. "$bin"/hadoop-config.sh
# start dfs daemons
"$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
*echo 'sleeping'
sleep 60
echo 'awake'*
# start mapred daemons
"$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
This seems to work. Please see if this works for you.
Thanks and Regards,
Sonal
On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 2/10/10 5:19 PM, Nick Klosterman wrote:
>
>> @E.Sammer, no I don't *think* that it is part of another cluster. The
>> tutorial is for a single node cluster just as a initial set up to see if
>> you can get things up and running. I have reformatted the namenode
>> several times in my effort to get hadoop to work.
>>
>
> What I mean is that the data node, at some point, connected to your name
> node. If you reformat the name node, the data node must be wiped clean; it's
> effectively trying to join a name node that no longer exists.
>
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>
I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
start-all script to introduce a sleep time :
bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
. "$bin"/hadoop-config.sh
# start dfs daemons
"$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
*echo 'sleeping'
sleep 60
echo 'awake'*
# start mapred daemons
"$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
This seems to work. Please see if this works for you.
Thanks and Regards,
Sonal
On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <[EMAIL PROTECTED]> wrote:
> On 2/10/10 5:19 PM, Nick Klosterman wrote:
>
>> @E.Sammer, no I don't *think* that it is part of another cluster. The
>> tutorial is for a single node cluster just as a initial set up to see if
>> you can get things up and running. I have reformatted the namenode
>> several times in my effort to get hadoop to work.
>>
>
> What I mean is that the data node, at some point, connected to your name
> node. If you reformat the name node, the data node must be wiped clean; it's
> effectively trying to join a name node that no longer exists.
>
>
> --
> Eric Sammer
> [EMAIL PROTECTED]
> http://esammer.blogspot.com
>
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Aaron Kimball 2010-02-13, 01:10
Sonal,
Can I ask why you're sleeping between starting hdfs and mapreduce? I've
never needed this in my own code. In general, Hadoop is pretty tolerant
about starting daemons "out of order."
If you need to wait for HDFS to be ready and come out of safe mode before
launching a job, that's another story, but you can accomplish that with:
$HADOOP_HOME/hadoop dfsadmin -safemode wait
... which will block until HDFS is ready for user commands in read/write
mode.
- Aaron
On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <[EMAIL PROTECTED]> wrote:
> Hi
>
> I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
> start-all script to introduce a sleep time :
>
> bin=`dirname "$0"`
> bin=`cd "$bin"; pwd`
>
> . "$bin"/hadoop-config.sh
>
> # start dfs daemons
> "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
> *echo 'sleeping'
> sleep 60
> echo 'awake'*
> # start mapred daemons
> "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
>
>
> This seems to work. Please see if this works for you.
> Thanks and Regards,
> Sonal
>
>
> On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <[EMAIL PROTECTED]> wrote:
>
> > On 2/10/10 5:19 PM, Nick Klosterman wrote:
> >
> >> @E.Sammer, no I don't *think* that it is part of another cluster. The
> >> tutorial is for a single node cluster just as a initial set up to see if
> >> you can get things up and running. I have reformatted the namenode
> >> several times in my effort to get hadoop to work.
> >>
> >
> > What I mean is that the data node, at some point, connected to your name
> > node. If you reformat the name node, the data node must be wiped clean;
> it's
> > effectively trying to join a name node that no longer exists.
> >
> >
> > --
> > Eric Sammer
> > [EMAIL PROTECTED]
> > http://esammer.blogspot.com
> >
>
Can I ask why you're sleeping between starting hdfs and mapreduce? I've
never needed this in my own code. In general, Hadoop is pretty tolerant
about starting daemons "out of order."
If you need to wait for HDFS to be ready and come out of safe mode before
launching a job, that's another story, but you can accomplish that with:
$HADOOP_HOME/hadoop dfsadmin -safemode wait
... which will block until HDFS is ready for user commands in read/write
mode.
- Aaron
On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <[EMAIL PROTECTED]> wrote:
> Hi
>
> I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
> start-all script to introduce a sleep time :
>
> bin=`dirname "$0"`
> bin=`cd "$bin"; pwd`
>
> . "$bin"/hadoop-config.sh
>
> # start dfs daemons
> "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
> *echo 'sleeping'
> sleep 60
> echo 'awake'*
> # start mapred daemons
> "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
>
>
> This seems to work. Please see if this works for you.
> Thanks and Regards,
> Sonal
>
>
> On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <[EMAIL PROTECTED]> wrote:
>
> > On 2/10/10 5:19 PM, Nick Klosterman wrote:
> >
> >> @E.Sammer, no I don't *think* that it is part of another cluster. The
> >> tutorial is for a single node cluster just as a initial set up to see if
> >> you can get things up and running. I have reformatted the namenode
> >> several times in my effort to get hadoop to work.
> >>
> >
> > What I mean is that the data node, at some point, connected to your name
> > node. If you reformat the name node, the data node must be wiped clean;
> it's
> > effectively trying to join a name node that no longer exists.
> >
> >
> > --
> > Eric Sammer
> > [EMAIL PROTECTED]
> > http://esammer.blogspot.com
> >
>
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Sonal Goyal 2010-02-13, 16:35
Hi Aaron,
I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
sleep time from my start-all.sh script, my jobtracker comes up momentarily
and then dies.
Here is a capture of my commands:
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem:
fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
current size 0
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
written 512 bytes at offset 1048576
10/02/13 21:54:22 INFO common.Storage: Storage directory
/tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
************************************************************/
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
starting namenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
localhost: starting datanode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
localhost: starting secondarynamenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
starting jobtracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
localhost: starting tasktracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26171 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
25863 DataNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
26203 Jps
25778 NameNode
26130 TaskTracker
25863 -- process information unavailable
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26239 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
26252 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26288 Jps
25966 SecondaryNameNode
25778 NameNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
25966 SecondaryNameNode
25778 NameNode
26298 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26308 Jps
25966 SecondaryNameNode
25778 NameNode
My jobtracker logs show:
2010-02-13 21:54:40,660 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2010-02-13 21:54:40,967 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=9001
2010-02-13 21:54:52,100 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-02-13 21:54:52,358 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2010-02-13 21:54:52,359 INFO org.mortbay.log: jetty-6.1.14
2010-02-13 21:55:13,222 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-02-13 21:55:13,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 9001
2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2010-02-13 21:55:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2010-02-13 21:55:14,049 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
sleep time from my start-all.sh script, my jobtracker comes up momentarily
and then dies.
Here is a capture of my commands:
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem:
fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
current size 0
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
written 512 bytes at offset 1048576
10/02/13 21:54:22 INFO common.Storage: Storage directory
/tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
************************************************************/
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
starting namenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
localhost: starting datanode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
localhost: starting secondarynamenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
starting jobtracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
localhost: starting tasktracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26171 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
25863 DataNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
26203 Jps
25778 NameNode
26130 TaskTracker
25863 -- process information unavailable
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26239 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
26252 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26288 Jps
25966 SecondaryNameNode
25778 NameNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
25966 SecondaryNameNode
25778 NameNode
26298 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26308 Jps
25966 SecondaryNameNode
25778 NameNode
My jobtracker logs show:
2010-02-13 21:54:40,660 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2010-02-13 21:54:40,967 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=9001
2010-02-13 21:54:52,100 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-02-13 21:54:52,358 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2010-02-13 21:54:52,359 INFO org.mortbay.log: jetty-6.1.14
2010-02-13 21:55:13,222 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-02-13 21:55:13,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 9001
2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2010-02-13 21:55:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2010-02-13 21:55:14,049 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Todd Lipcon 2010-02-13, 17:37
Hi Sonal,
Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
fixes in 0.20.1, and more in 0.20.2 which should be released any
minute now.
In particular, you're missing this change:
https://issues.apache.org/jira/browse/HADOOP-5921
which makes the JobTracker stubbornly wait for DFS to appear.
I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
or (c) Cloudera's 0.20.1 based build at
http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
-Todd
On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <[EMAIL PROTECTED]> wrote:
> Hi Aaron,
>
> I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> sleep time from my start-all.sh script, my jobtracker comes up momentarily
> and then dies.
>
> Here is a capture of my commands:
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = desktop/127.0.1.1
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
> compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
>
> Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> seconds.
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> current size 0
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
> written 512 bytes at offset 1048576
> 10/02/13 21:54:22 INFO common.Storage: Storage directory
> /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> ************************************************************/
>
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> starting namenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> localhost: starting datanode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> localhost: starting secondarynamenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> starting jobtracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> localhost: starting tasktracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26171 Jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 25778 NameNode
> 26130 TaskTracker
> 25863 DataNode
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 26203 Jps
> 25778 NameNode
Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
fixes in 0.20.1, and more in 0.20.2 which should be released any
minute now.
In particular, you're missing this change:
https://issues.apache.org/jira/browse/HADOOP-5921
which makes the JobTracker stubbornly wait for DFS to appear.
I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
or (c) Cloudera's 0.20.1 based build at
http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
-Todd
On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <[EMAIL PROTECTED]> wrote:
> Hi Aaron,
>
> I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> sleep time from my start-all.sh script, my jobtracker comes up momentarily
> and then dies.
>
> Here is a capture of my commands:
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = desktop/127.0.1.1
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
> compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
>
> Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> seconds.
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> current size 0
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
> written 512 bytes at offset 1048576
> 10/02/13 21:54:22 INFO common.Storage: Storage directory
> /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> ************************************************************/
>
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> starting namenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> localhost: starting datanode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> localhost: starting secondarynamenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> starting jobtracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> localhost: starting tasktracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26171 Jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 25778 NameNode
> 26130 TaskTracker
> 25863 DataNode
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 26203 Jps
> 25778 NameNode
-
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.Sonal Goyal 2010-02-13, 17:41
Yes, thanks Todd. I am looking to upgrade to 0.20.2.
Thanks and Regards,
Sonal
On Sat, Feb 13, 2010 at 11:07 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Sonal,
>
> Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
> fixes in 0.20.1, and more in 0.20.2 which should be released any
> minute now.
>
> In particular, you're missing this change:
> https://issues.apache.org/jira/browse/HADOOP-5921
>
> which makes the JobTracker stubbornly wait for DFS to appear.
>
> I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
> or (c) Cloudera's 0.20.1 based build at
> http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
> 0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
>
> -Todd
>
> On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <[EMAIL PROTECTED]>
> wrote:
> > Hi Aaron,
> >
> > I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> > sleep time from my start-all.sh script, my jobtracker comes up
> momentarily
> > and then dies.
> >
> > Here is a capture of my commands:
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> > 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting NameNode
> > STARTUP_MSG: host = desktop/127.0.1.1
> > STARTUP_MSG: args = [-format]
> > STARTUP_MSG: version = 0.20.0
> > STARTUP_MSG: build > > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504;
> > compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> > ************************************************************/
> > 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
> >
> > Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> > 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> >
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> >
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> > 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> > seconds.
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> > current size 0
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now
> 1049088
> > written 512 bytes at offset 1048576
> > 10/02/13 21:54:22 INFO common.Storage: Storage directory
> > /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> > 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> > ************************************************************/
> >
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> > starting namenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> > localhost: starting datanode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> > localhost: starting secondarynamenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> > starting jobtracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> > localhost: starting tasktracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
Thanks and Regards,
Sonal
On Sat, Feb 13, 2010 at 11:07 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Sonal,
>
> Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
> fixes in 0.20.1, and more in 0.20.2 which should be released any
> minute now.
>
> In particular, you're missing this change:
> https://issues.apache.org/jira/browse/HADOOP-5921
>
> which makes the JobTracker stubbornly wait for DFS to appear.
>
> I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
> or (c) Cloudera's 0.20.1 based build at
> http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
> 0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
>
> -Todd
>
> On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <[EMAIL PROTECTED]>
> wrote:
> > Hi Aaron,
> >
> > I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> > sleep time from my start-all.sh script, my jobtracker comes up
> momentarily
> > and then dies.
> >
> > Here is a capture of my commands:
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> > 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting NameNode
> > STARTUP_MSG: host = desktop/127.0.1.1
> > STARTUP_MSG: args = [-format]
> > STARTUP_MSG: version = 0.20.0
> > STARTUP_MSG: build > > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504;
> > compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> > ************************************************************/
> > 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
> >
> > Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> > 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> >
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> >
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> > 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> > seconds.
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> > current size 0
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now
> 1049088
> > written 512 bytes at offset 1048576
> > 10/02/13 21:54:22 INFO common.Storage: Storage directory
> > /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> > 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> > ************************************************************/
> >
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> > starting namenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> > localhost: starting datanode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> > localhost: starting secondarynamenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> > starting jobtracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> > localhost: starting tasktracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
