srinivas
2012-01-25, 06:54
Xu, Richard
2011-05-27, 00:01
DAN
2011-05-27, 02:22
Xu, Richard
2011-05-27, 11:34
Simon
2011-05-27, 12:30
DAN
2011-05-27, 14:26
Allen Wittenauer
2011-05-27, 15:42
Xu, Richard
2011-05-27, 20:18
Allen Wittenauer
2011-05-27, 22:52
Konstantin Boudnik
2011-05-27, 03:18
Harsh J
2011-05-27, 13:20
Xu, Richard
2011-05-27, 21:32
-
Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clustersrinivas 2012-01-25, 06:54
Harsh J <harsh@...> writes:
>
> Hello RX,
>
> Could you paste your DFS configuration and the DN end-to-end log into
> a mail/pastebin-link?
>
> On Fri, May 27, 2011 at 5:31 AM, Xu, Richard <richard.xu@...> wrote:
> > Hi Folks,
> >
> > We try to get hbase and hadoop running on clusters, take 2 Solaris servers
for now.
> >
> > Because of the incompatibility issue between hbase and hadoop, we have to
stick with hadoop
> 0.20.2-append release.
> >
> > It is very straight forward to make hadoop-0.20.203 running, but stuck for
several days with
> hadoop-0.20.2, even the official release, not the append version.
> >
> > 1. Once try to run start-mapred.sh(hadoop-daemon.sh --config
$HADOOP_CONF_DIR start jobtracker),
> following errors shown in namenode and jobtracker logs:
> >
> > 2011-05-26 12:30:29,169 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to
> place enough replicas, still in need of 1
> > 2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 9000, call
> addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
> > ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException:
File
> /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0
n
> > odes, instead of 1
> > java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info
could only be
> replicated to 0 nodes, instead of 1
> > at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesys
tem.java:1271)
> > at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> >
> > 2. Also, Configured Capacity is 0, cannot put any file to HDFS.
> >
> > 3. in datanode server, no error in logs, but tasktracker logs has the
following suspicious thing:
> > 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
> > 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 41904: starting
> > 2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 41904: starting
> > 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 41904: starting
> > 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 41904: starting
> > 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 41904: starting
> > .....
> > 2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 63 on 41904: starting
> > 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost/127.0.0.1:41904
> > 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_loanps3d:localhost/127.0.0.1:41904
> >
> >
> > I have tried all suggestions found so far, including
> > 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> > 2) clean up all temp files/folders under /tmp;
> >
> > But nothing works.
> >
> > Your help is greatly appreciated.
> >
> > Thanks,
> >
> > RX
> >
>
Hi,
I am able to start name node and data node,but while starting the
jobatracker,it's troughing an error like
FATAL mapred.JobTracker: java.net.BindException: Problem binding to
localhost/127.0.0.1:5102 : Address already in use
kindly help me ASAP......................
regards,
Srinivas
+
srinivas 2012-01-25, 06:54
-
Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterXu, Richard 2011-05-27, 00:01
Hi Folks,
We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
odes, instead of 1
java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2. Also, Configured Capacity is 0, cannot put any file to HDFS.
3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
.....
2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
I have tried all suggestions found so far, including
1) remove hadoop-name and hadoop-data folders and reformat namenode;
2) clean up all temp files/folders under /tmp;
But nothing works.
Your help is greatly appreciated.
Thanks,
RX
We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
odes, instead of 1
java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2. Also, Configured Capacity is 0, cannot put any file to HDFS.
3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
.....
2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
I have tried all suggestions found so far, including
1) remove hadoop-name and hadoop-data folders and reformat namenode;
2) clean up all temp files/folders under /tmp;
But nothing works.
Your help is greatly appreciated.
Thanks,
RX
+
Xu, Richard 2011-05-27, 00:01
-
Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterDAN 2011-05-27, 02:22
Hi, Richard
Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
setting of "dfs.replication" in hdfs-site.xml.
Good luck!
Dan
--
At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]> wrote:
>Hi Folks,
>
>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>odes, instead of 1
>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
>I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
>But nothing works.
>
>Your help is greatly appreciated.
>
>Thanks,
>
>RX
Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
setting of "dfs.replication" in hdfs-site.xml.
Good luck!
Dan
--
At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]> wrote:
>Hi Folks,
>
>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>odes, instead of 1
>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
>I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
>But nothing works.
>
>Your help is greatly appreciated.
>
>Thanks,
>
>RX
+
DAN 2011-05-27, 02:22
-
RE: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterXu, Richard 2011-05-27, 11:34
That setting is 3.
From: DAN [mailto:[EMAIL PROTECTED]]
Sent: Thursday, May 26, 2011 10:23 PM
To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
Hi, Richard
Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
setting of "dfs.replication" in hdfs-site.xml.
Good luck!
Dan
--
At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>Hi Folks,
>
>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>odes, instead of 1
>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
>I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
>But nothing works.
>
>Your help is greatly appreciated.
From: DAN [mailto:[EMAIL PROTECTED]]
Sent: Thursday, May 26, 2011 10:23 PM
To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
Hi, Richard
Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
setting of "dfs.replication" in hdfs-site.xml.
Good luck!
Dan
--
At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>Hi Folks,
>
>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>odes, instead of 1
>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
>I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
>But nothing works.
>
>Your help is greatly appreciated.
+
Xu, Richard 2011-05-27, 11:34
-
Re: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterSimon 2011-05-27, 12:30
First you need to make sure that your dfs daemons are running.
You can start you namenode and datanode separately on the master and slave
nodes, and see what happens with the following commands:
hadoop namenode
hadoop datanode
The chancess are that your data node can not be started correctly.
Let us know your error logs if there are errors.
HTH~
Thanks
Simon
2011/5/27 Xu, Richard <[EMAIL PROTECTED]>
> That setting is 3.
>
> From: DAN [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 26, 2011 10:23 PM
> To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
> Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203
> cluster
>
> Hi, Richard
>
> Pay attention to "Not able to place enough replicas, still in need of 1".
> Pls confirm right
> setting of "dfs.replication" in hdfs-site.xml.
>
> Good luck!
> Dan
> --
>
>
> At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>> wrote:
>
>
>
> >Hi Folks,
>
> >
>
> >We try to get hbase and hadoop running on clusters, take 2 Solaris servers
> for now.
>
> >
>
> >Because of the incompatibility issue between hbase and hadoop, we have to
> stick with hadoop 0.20.2-append release.
>
> >
>
> >It is very straight forward to make hadoop-0.20.203 running, but stuck for
> several days with hadoop-0.20.2, even the official release, not the append
> version.
>
> >
>
> >1. Once try to run start-mapred.sh(hadoop-daemon.sh --config
> $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and
> jobtracker logs:
>
> >
>
> >2011-05-26 12:30:29,169 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
> enough replicas, still in need of 1
>
> >2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/
> jobtracker.info, DFSCl
>
> >ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException:
> File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be
> replicated to 0 n
>
> >odes, instead of 1
>
> >java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.infocould only be replicated to 0 nodes, instead of 1
>
> > at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>
> > at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> > at java.lang.reflect.Method.invoke(Method.java:597)
>
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>
> > at java.security.AccessController.doPrivileged(Native Method)
>
> > at javax.security.auth.Subject.doAs(Subject.java:396)
>
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> >
>
> >
>
> >2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
> >
>
> >3. in datanode server, no error in logs, but tasktracker logs has the
> following suspicious thing:
>
> >2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> >2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 41904: starting
>
> >2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 41904: starting
>
> > .....
Regards,
Simon
You can start you namenode and datanode separately on the master and slave
nodes, and see what happens with the following commands:
hadoop namenode
hadoop datanode
The chancess are that your data node can not be started correctly.
Let us know your error logs if there are errors.
HTH~
Thanks
Simon
2011/5/27 Xu, Richard <[EMAIL PROTECTED]>
> That setting is 3.
>
> From: DAN [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 26, 2011 10:23 PM
> To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
> Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203
> cluster
>
> Hi, Richard
>
> Pay attention to "Not able to place enough replicas, still in need of 1".
> Pls confirm right
> setting of "dfs.replication" in hdfs-site.xml.
>
> Good luck!
> Dan
> --
>
>
> At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>> wrote:
>
>
>
> >Hi Folks,
>
> >
>
> >We try to get hbase and hadoop running on clusters, take 2 Solaris servers
> for now.
>
> >
>
> >Because of the incompatibility issue between hbase and hadoop, we have to
> stick with hadoop 0.20.2-append release.
>
> >
>
> >It is very straight forward to make hadoop-0.20.203 running, but stuck for
> several days with hadoop-0.20.2, even the official release, not the append
> version.
>
> >
>
> >1. Once try to run start-mapred.sh(hadoop-daemon.sh --config
> $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and
> jobtracker logs:
>
> >
>
> >2011-05-26 12:30:29,169 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
> enough replicas, still in need of 1
>
> >2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/
> jobtracker.info, DFSCl
>
> >ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException:
> File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be
> replicated to 0 n
>
> >odes, instead of 1
>
> >java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.infocould only be replicated to 0 nodes, instead of 1
>
> > at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>
> > at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> > at java.lang.reflect.Method.invoke(Method.java:597)
>
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>
> > at java.security.AccessController.doPrivileged(Native Method)
>
> > at javax.security.auth.Subject.doAs(Subject.java:396)
>
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> >
>
> >
>
> >2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
> >
>
> >3. in datanode server, no error in logs, but tasktracker logs has the
> following suspicious thing:
>
> >2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> >2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 41904: starting
>
> >2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 41904: starting
>
> >2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 41904: starting
>
> > .....
Regards,
Simon
+
Simon 2011-05-27, 12:30
-
Re:RE: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterDAN 2011-05-27, 14:26
Hi, Richard
You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
These don't match.
Good Luck
Dan
At 2011-05-27 19:34:10，"Xu, Richard " <[EMAIL PROTECTED]> wrote:
>That setting is 3.
>
>From: DAN [mailto:[EMAIL PROTECTED]]
>Sent: Thursday, May 26, 2011 10:23 PM
>To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
>Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
>
>Hi, Richard
>
>Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
>setting of "dfs.replication" in hdfs-site.xml.
>
>Good luck!
>Dan
>--
>
>
>At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>
>
>
>>Hi Folks,
>
>>
>
>>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>>
>
>>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>>
>
>>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>>
>
>>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>>
>
>>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>
>>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>
>>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>
>>odes, instead of 1
>
>>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
>
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>> at java.lang.reflect.Method.invoke(Method.java:597)
>
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>
>> at java.security.AccessController.doPrivileged(Native Method)
>
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>>
>
>>
>
>>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>>
>
>>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>
>>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>
>>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>
>>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
>
>> .....
>
>>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>
>>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>
>>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
These don't match.
Good Luck
Dan
At 2011-05-27 19:34:10，"Xu, Richard " <[EMAIL PROTECTED]> wrote:
>That setting is 3.
>
>From: DAN [mailto:[EMAIL PROTECTED]]
>Sent: Thursday, May 26, 2011 10:23 PM
>To: [EMAIL PROTECTED]; Xu, Richard [ICG-IT]
>Subject: Re:Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
>
>Hi, Richard
>
>Pay attention to "Not able to place enough replicas, still in need of 1". Pls confirm right
>setting of "dfs.replication" in hdfs-site.xml.
>
>Good luck!
>Dan
>--
>
>
>At 2011-05-27 08:01:37，"Xu, Richard " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>
>
>
>>Hi Folks,
>
>>
>
>>We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
>>
>
>>Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
>>
>
>>It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
>>
>
>>1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
>>
>
>>2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
>
>>2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
>
>>ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
>
>>odes, instead of 1
>
>>java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
>
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>> at java.lang.reflect.Method.invoke(Method.java:597)
>
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>
>> at java.security.AccessController.doPrivileged(Native Method)
>
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>>
>
>>
>
>>2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
>>
>
>>3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
>
>>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>
>>2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
>
>>2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
>
>>2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
>
>> .....
>
>>2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
>
>>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
>
>>2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
+
DAN 2011-05-27, 14:26
-
Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterAllen Wittenauer 2011-05-27, 15:42
On May 27, 2011, at 7:26 AM, DAN wrote:
> You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
> These don't match.
That doesn't matter. HDFS will basically flag any files written with a warning that they are under-replicated.
The problem is that the datanode processes aren't running and/or aren't communicating to the namenode. That's what the "java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1" means.
It should also be pointed out that writing to /tmp (the default) is a bad idea. This should get changed.
Also, since you are running Solaris, check the FAQ on some settings you'll need to do in order to make Hadoop's broken username detection to work properly, amongst other things.
+
Allen Wittenauer 2011-05-27, 15:42
-
RE: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterXu, Richard 2011-05-27, 20:18
Hi Allen,
Thanks a lot for your response.
I agree with you that it does not matter with replication settings.
What really bothered me is same environment, same configures, hadoop 0.20.203 takes us 3 mins, why 0.20.2 took 3 days.
Can you pls. shed more light on how "to make Hadoop's broken username detection to work properly"?
-----Original Message-----
From: Allen Wittenauer [mailto:[EMAIL PROTECTED]]
Sent: Friday, May 27, 2011 11:42 AM
To: [EMAIL PROTECTED]
Cc: Xu, Richard [ICG-IT]
Subject: Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
On May 27, 2011, at 7:26 AM, DAN wrote:
> You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
> These don't match.
That doesn't matter. HDFS will basically flag any files written with a warning that they are under-replicated.
The problem is that the datanode processes aren't running and/or aren't communicating to the namenode. That's what the "java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1" means.
It should also be pointed out that writing to /tmp (the default) is a bad idea. This should get changed.
Also, since you are running Solaris, check the FAQ on some settings you'll need to do in order to make Hadoop's broken username detection to work properly, amongst other things.
Thanks a lot for your response.
I agree with you that it does not matter with replication settings.
What really bothered me is same environment, same configures, hadoop 0.20.203 takes us 3 mins, why 0.20.2 took 3 days.
Can you pls. shed more light on how "to make Hadoop's broken username detection to work properly"?
-----Original Message-----
From: Allen Wittenauer [mailto:[EMAIL PROTECTED]]
Sent: Friday, May 27, 2011 11:42 AM
To: [EMAIL PROTECTED]
Cc: Xu, Richard [ICG-IT]
Subject: Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
On May 27, 2011, at 7:26 AM, DAN wrote:
> You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
> These don't match.
That doesn't matter. HDFS will basically flag any files written with a warning that they are under-replicated.
The problem is that the datanode processes aren't running and/or aren't communicating to the namenode. That's what the "java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1" means.
It should also be pointed out that writing to /tmp (the default) is a bad idea. This should get changed.
Also, since you are running Solaris, check the FAQ on some settings you'll need to do in order to make Hadoop's broken username detection to work properly, amongst other things.
+
Xu, Richard 2011-05-27, 20:18
-
Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterAllen Wittenauer 2011-05-27, 22:52
On May 27, 2011, at 1:18 PM, Xu, Richard wrote:
> Hi Allen,
>
> Thanks a lot for your response.
>
> I agree with you that it does not matter with replication settings.
>
> What really bothered me is same environment, same configures, hadoop 0.20.203 takes us 3 mins, why 0.20.2 took 3 days.
>
> Can you pls. shed more light on how "to make Hadoop's broken username detection to work properly"?
It's in the FAQ so that I don't have to do that.
http://wiki.apache.org/hadoop/FAQ
Also, check your logs. All your logs. Not just the namenode log.
+
Allen Wittenauer 2011-05-27, 22:52
-
Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterKonstantin Boudnik 2011-05-27, 03:18
On Thu, May 26, 2011 at 07:01PM, Xu, Richard wrote:
> 2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
> ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
> odes, instead of 1
> java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
Is your DFS up running, by any chance?
Cos
> 2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
> ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
> odes, instead of 1
> java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
Is your DFS up running, by any chance?
Cos
+
Konstantin Boudnik 2011-05-27, 03:18
-
Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterHarsh J 2011-05-27, 13:20
Hello RX,
Could you paste your DFS configuration and the DN end-to-end log into
a mail/pastebin-link?
On Fri, May 27, 2011 at 5:31 AM, Xu, Richard <[EMAIL PROTECTED]> wrote:
> Hi Folks,
>
> We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
> Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
> It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
> 1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
> 2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
> 2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
> ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
> odes, instead of 1
> java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
> 2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
> 3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
> 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
> 2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
> 2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
> 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
> 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
> I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
> But nothing works.
>
> Your help is greatly appreciated.
>
> Thanks,
>
> RX
>
--
Harsh J
Could you paste your DFS configuration and the DN end-to-end log into
a mail/pastebin-link?
On Fri, May 27, 2011 at 5:31 AM, Xu, Richard <[EMAIL PROTECTED]> wrote:
> Hi Folks,
>
> We try to get hbase and hadoop running on clusters, take 2 Solaris servers for now.
>
> Because of the incompatibility issue between hbase and hadoop, we have to stick with hadoop 0.20.2-append release.
>
> It is very straight forward to make hadoop-0.20.203 running, but stuck for several days with hadoop-0.20.2, even the official release, not the append version.
>
> 1. Once try to run start-mapred.sh(hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker), following errors shown in namenode and jobtracker logs:
>
> 2011-05-26 12:30:29,169 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1
> 2011-05-26 12:30:29,175 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/tmp/hadoop-cfadm/mapred/system/jobtracker.info, DFSCl
> ient_2146408809) from 169.193.181.212:55334: error: java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 n
> odes, instead of 1
> java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
> 2. Also, Configured Capacity is 0, cannot put any file to HDFS.
>
> 3. in datanode server, no error in logs, but tasktracker logs has the following suspicious thing:
> 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2011-05-25 23:36:10,839 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 41904: starting
> 2011-05-25 23:36:10,852 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 41904: starting
> 2011-05-25 23:36:10,853 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 41904: starting
> .....
> 2011-05-25 23:36:10,855 INFO org.apache.hadoop.ipc.Server: IPC Server handler 63 on 41904: starting
> 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:41904
> 2011-05-25 23:36:10,950 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_loanps3d:localhost/127.0.0.1:41904
>
>
> I have tried all suggestions found so far, including
> 1) remove hadoop-name and hadoop-data folders and reformat namenode;
> 2) clean up all temp files/folders under /tmp;
>
> But nothing works.
>
> Your help is greatly appreciated.
>
> Thanks,
>
> RX
>
--
Harsh J
+
Harsh J 2011-05-27, 13:20
-
RE: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 clusterXu, Richard 2011-05-27, 21:32
Add more to that:
I also tried start 0.20.2 on a linux machine in distributed mode, same error.
I had successfully started 0.20.203 on this linux machine with same config.
Seems that it is not related to Solaris.
Could it caused by port? I checked a few, did not find anyone blocked.
-----Original Message-----
From: Xu, Richard [ICG-IT]
Sent: Friday, May 27, 2011 4:18 PM
To: 'Allen Wittenauer'; '[EMAIL PROTECTED]'
Subject: RE: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
Hi Allen,
Thanks a lot for your response.
I agree with you that it does not matter with replication settings.
What really bothered me is same environment, same configures, hadoop 0.20.203 takes us 3 mins, why 0.20.2 took 3 days.
Can you pls. shed more light on how "to make Hadoop's broken username detection to work properly"?
-----Original Message-----
From: Allen Wittenauer [mailto:[EMAIL PROTECTED]]
Sent: Friday, May 27, 2011 11:42 AM
To: [EMAIL PROTECTED]
Cc: Xu, Richard [ICG-IT]
Subject: Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
On May 27, 2011, at 7:26 AM, DAN wrote:
> You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
> These don't match.
That doesn't matter. HDFS will basically flag any files written with a warning that they are under-replicated.
The problem is that the datanode processes aren't running and/or aren't communicating to the namenode. That's what the "java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1" means.
It should also be pointed out that writing to /tmp (the default) is a bad idea. This should get changed.
Also, since you are running Solaris, check the FAQ on some settings you'll need to do in order to make Hadoop's broken username detection to work properly, amongst other things.
I also tried start 0.20.2 on a linux machine in distributed mode, same error.
I had successfully started 0.20.203 on this linux machine with same config.
Seems that it is not related to Solaris.
Could it caused by port? I checked a few, did not find anyone blocked.
-----Original Message-----
From: Xu, Richard [ICG-IT]
Sent: Friday, May 27, 2011 4:18 PM
To: 'Allen Wittenauer'; '[EMAIL PROTECTED]'
Subject: RE: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
Hi Allen,
Thanks a lot for your response.
I agree with you that it does not matter with replication settings.
What really bothered me is same environment, same configures, hadoop 0.20.203 takes us 3 mins, why 0.20.2 took 3 days.
Can you pls. shed more light on how "to make Hadoop's broken username detection to work properly"?
-----Original Message-----
From: Allen Wittenauer [mailto:[EMAIL PROTECTED]]
Sent: Friday, May 27, 2011 11:42 AM
To: [EMAIL PROTECTED]
Cc: Xu, Richard [ICG-IT]
Subject: Re: Unable to start hadoop-0.20.2 but able to start hadoop-0.20.203 cluster
On May 27, 2011, at 7:26 AM, DAN wrote:
> You see you have "2 Solaris servers for now", and dfs.replication is setted as 3.
> These don't match.
That doesn't matter. HDFS will basically flag any files written with a warning that they are under-replicated.
The problem is that the datanode processes aren't running and/or aren't communicating to the namenode. That's what the "java.io.IOException: File /tmp/hadoop-cfadm/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1" means.
It should also be pointed out that writing to /tmp (the default) is a bad idea. This should get changed.
Also, since you are running Solaris, check the FAQ on some settings you'll need to do in order to make Hadoop's broken username detection to work properly, amongst other things.
+
Xu, Richard 2011-05-27, 21:32
