HDFS, mail # user - Re: Namenode and Jobtracker dont start

Namenode and Jobtracker dont start

Björn-Elmar Macek 2012-07-18, 14:29

Hi,

i have lately been running into problems since i started running hadoop
on a cluster:

The setup is the following:
1 Computer is NameNode and Jobtracker
1 Computer is SecondaryNameNode
2 Computers are TaskTracker and DataNode

I ran into problems with running the wordcount example: NameNode and
Jobtracker do not start properly both having connection problems of some
kind.
And this is although ssh is configured that way, that no prompt happens
when i connect from any node in the cluster to any other.

Is there any reason why this happens?

The logs look like the following:
\________ JOBTRACKER__________________________________________________
2012-07-18 16:08:05,808 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = its-cs100.its.uni-kassel.de/141.51.205.10
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r
1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-18 16:08:06,479 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2012-07-18 16:08:06,534 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-07-18 16:08:06,554 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-07-18 16:08:06,554 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
system started
2012-07-18 16:08:07,157 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
QueueMetrics,q=default registered.
2012-07-18 16:08:10,395 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ugi registered.
2012-07-18 16:08:10,417 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2012-07-18 16:08:10,436 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-07-18 16:08:10,438 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-07-18 16:08:10,440 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Starting expired delegation token remover thread,
tokenRemoverScanInterval=60 min(s)
2012-07-18 16:08:10,465 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2012-07-18 16:08:10,510 INFO org.apache.hadoop.mapred.JobTracker:
Starting jobtracker with owner as bmacek
2012-07-18 16:08:10,620 WARN org.apache.hadoop.mapred.JobTracker: Error
starting tracker: java.net.SocketException: Permission denied
   at sun.nio.ch.Net.bind(Native Method)
   at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
   at org.apache.hadoop.ipc.Server.bind(Server.java:225)
   at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
   at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
   at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
   at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2306)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
   at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
   at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
   at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2012-07-18 16:08:13,861 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name
QueueMetrics,q=default already exists!
2012-07-18 16:08:13,885 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2012-07-18 16:08:13,885 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2012-07-18 16:08:13,910 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-07-18 16:08:13,911 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-07-18 16:08:13,911 INFO org.apache.hadoop.mapred.JobTracker:
Starting jobtracker with owner as bmacek
2012-07-18 16:08:13,912 WARN org.apache.hadoop.mapred.JobTracker: Error
starting tracker: java.net.SocketException: Permission denied
   at sun.nio.ch.Net.bind(Native Method)
   at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
   at org.apache.hadoop.ipc.Server.bind(Server.java:225)
   at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
   at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
   at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
   at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2306)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
   at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
   at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
   at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
   at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2012-07-18 16:08:13,912 INFO
o

Re: Namenode and Jobtracker dont start

Suresh Srinivas 2012-07-18, 17:47

Can you share information on the java version that you are using.
- Is it as obvious as some previous processes still running and new
processes cannot bind to the port?
- Another pointer -
http://stackoverflow.com/questions/8360913/weird-java-net-socketexception-permission-denied-connect-error-when-running-groo

On Wed, Jul 18, 2012 at 7:29 AM, Björn-Elmar Macek
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> i have lately been running into problems since i started running hadoop on
> a cluster:
>
> The setup is the following:
> 1 Computer is NameNode and Jobtracker
> 1 Computer is SecondaryNameNode
> 2 Computers are TaskTracker and DataNode
>
> I ran into problems with running the wordcount example: NameNode and
> Jobtracker do not start properly both having connection problems of some
> kind.
> And this is although ssh is configured that way, that no prompt happens
> when i connect from any node in the cluster to any other.
>
> Is there any reason why this happens?
>
> The logs look like the following:
> \________ JOBTRACKER____________________**______________________________
> 2012-07-18 16:08:05,808 INFO org.apache.hadoop.mapred.**JobTracker:
> STARTUP_MSG:
> /****************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = its-cs100.its.uni-kassel.de/**141.51.205.10<http://its-cs100.its.uni-kassel.de/141.51.205.10>
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.2
> STARTUP_MSG: build = https://svn.apache.org/repos/**
> asf/hadoop/common/branches/**branch-1.0.2<https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2>-r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
> ****************************************************************/
> 2012-07-18 16:08:06,479 INFO org.apache.hadoop.metrics2.**impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2012-07-18 16:08:06,534 INFO org.apache.hadoop.metrics2.**impl.MetricsSourceAdapter:
> MBean for source MetricsSystem,sub=Stats registered.
> 2012-07-18 16:08:06,554 INFO org.apache.hadoop.metrics2.**impl.MetricsSystemImpl:
> Scheduled snapshot period at 10 second(s).
> 2012-07-18 16:08:06,554 INFO org.apache.hadoop.metrics2.**impl.MetricsSystemImpl:
> JobTracker metrics system started
> 2012-07-18 16:08:07,157 INFO org.apache.hadoop.metrics2.**impl.MetricsSourceAdapter:
> MBean for source QueueMetrics,q=default registered.
> 2012-07-18 16:08:10,395 INFO org.apache.hadoop.metrics2.**impl.MetricsSourceAdapter:
> MBean for source ugi registered.
> 2012-07-18 16:08:10,417 INFO org.apache.hadoop.security.**
> token.delegation.**AbstractDelegationTokenSecretM**anager: Updating the
> current master key for generating delegation tokens
> 2012-07-18 16:08:10,436 INFO org.apache.hadoop.mapred.**JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2012-07-18 16:08:10,438 INFO org.apache.hadoop.util.**HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2012-07-18 16:08:10,440 INFO org.apache.hadoop.security.**
> token.delegation.**AbstractDelegationTokenSecretM**anager: Starting
> expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> 2012-07-18 16:08:10,465 INFO org.apache.hadoop.security.**
> token.delegation.**AbstractDelegationTokenSecretM**anager: Updating the
> current master key for generating delegation tokens
> 2012-07-18 16:08:10,510 INFO org.apache.hadoop.mapred.**JobTracker:
> Starting jobtracker with owner as bmacek
> 2012-07-18 16:08:10,620 WARN org.apache.hadoop.mapred.**JobTracker: Error
> starting tracker: java.net.SocketException: Permission denied
> at sun.nio.ch.Net.bind(Native Method)
> at sun.nio.ch.**ServerSocketChannelImpl.bind(**
> ServerSocketChannelImpl.java:**119)
> at sun.nio.ch.**ServerSocketAdaptor.bind(**
> ServerSocketAdaptor.java:59)
> at org.apache.hadoop.ipc.Server.**bind(Server.java:225)
> at org.apache.hadoop.ipc.Server$**Listener.<init>(Server.java:**301)

http://hortonworks.com/download/

Re: Namenode and Jobtracker dont start

Björn-Elmar Macek 2012-07-20, 14:15

Hi Srinivas,

thanks for your reply! I have been following your link and idea and been
playing around alot, but still got problems with the connection (though
they are different now):

\_______ JAVA VERSION_________
"which java" tells me it is 1.6.0_01. If i got it right version 1.7 got
problems with ssh.

\_______MY TESTS_____________
According to your suggestion to look for processes running on that port
i changed ports alot:
When i was posting the first post of this thread. i was using ports 999
for namenode and 1000 for jobtracker.
Since due to some reasons commands like "lsof -i" etc dont give me any
output when usedin the cluster enviroment. So i started looking for
ports that are in general unused by programs.
When i changed the ports to 9004 and 9005 i got different errors which
look very much like the ones you posted in the beginning of this year in
the lucene section (
http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html
).

It seems as if a DataNode can not communicate with the NameNode.

The logs look like the following:

\_______TEST RESULTS__________
########## A DataNode #############
2012-07-20 14:47:59,536 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = its-cs102.its.uni-kassel.de/141.51.205.12
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r
1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-20 14:47:59,824 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2012-07-20 14:47:59,841 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-07-20 14:47:59,843 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-07-20 14:47:59,844 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics
system started
2012-07-20 14:47:59,969 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ugi registered.
2012-07-20 14:48:26,792 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: its-cs100/141.51.205.10:9005. Already tried 0 time(s).
2012-07-20 14:48:26,889 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
FSDatasetStatusMBean
2012-07-20 14:48:26,934 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
2012-07-20 14:48:26,949 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
1048576 bytes/s
2012-07-20 14:48:27,014 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-07-20 14:48:27,147 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-07-20 14:48:27,160 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50075
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50075
webServer.getConnectors()[0].getLocalPort() returned 50075
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50075
2012-07-20 14:48:27,160 INFO org.mortbay.log: jetty-6.1.26
2012-07-20 14:48:27,805 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50075
2012-07-20 14:48:27,811 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
jvm registered.
2012-07-20 14:48:27,813 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
DataNode registered.
2012-07-20 14:48:28,484 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort50020 registered.
2012-07-20 14:48:28,484 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort50020 registered.
2012-07-20 14:48:28,487 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(its-cs102.its.uni-kassel.de:50010, storageID=,
infoPort=50075, ipcPort=50020)
2012-07-20 14:48:28,489 INFO org.apache.hadoop.ipc.Server: Starting
SocketReader
2012-07-20 14:48:38,706 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException:
Datanode denied communication with namenode:
its-cs102.its.uni-kassel.de:50010
   at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391)
   at
org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

   at org.apache.hadoop.ipc.Client.call(Client.java:1066)
   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
   at $Pro

Re: Namenode and Jobtracker dont start

Björn-Elmar Macek 2012-07-20, 14:54

Hi together,

well just stumbled upon this post:
http://ankitasblogger.blogspot.de/2012/01/error-that-occured-in-hadoop-and-its.html

And it says:
"Problem: Hadoop-datanode job failed or datanode not running:
java.io.IOException: File ../mapred/system/jobtracker.info could only be
replicated to 0 nodes, instead of 1.
...
Cause: You may also get this message due to permissions. May be
JobTracker can not create jobtracker.info on startup."

Since the file does not exist i think, this might be a probable reason
for my errors. But why should the JobTracker not be able to create that
file. It created several other directories on this node with easy via
the slave.sh script that i started with the very same user that calls
start-all.sh.

Any help would be really appreciated.
Am 20.07.2012 16:15, schrieb Bjï¿½rn-Elmar Macek:
> Hi Srinivas,
>
> thanks for your reply! I have been following your link and idea and
> been playing around alot, but still got problems with the connection
> (though they are different now):
>
> \_______ JAVA VERSION_________
> "which java" tells me it is 1.6.0_01. If i got it right version 1.7
> got problems with ssh.
>
> \_______MY TESTS_____________
> According to your suggestion to look for processes running on that
> port i changed ports alot:
> When i was posting the first post of this thread. i was using ports
> 999 for namenode and 1000 for jobtracker.
> Since due to some reasons commands like "lsof -i" etc dont give me any
> output when usedin the cluster enviroment. So i started looking for
> ports that are in general unused by programs.
> When i changed the ports to 9004 and 9005 i got different errors which
> look very much like the ones you posted in the beginning of this year
> in the lucene section (
> http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html
> ).
>
> It seems as if a DataNode can not communicate with the NameNode.
>
> The logs look like the following:
>
> \_______TEST RESULTS__________
> ########## A DataNode #############
> 2012-07-20 14:47:59,536 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = its-cs102.its.uni-kassel.de/141.51.205.12
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2
> -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
> ************************************************************/
> 2012-07-20 14:47:59,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-07-20 14:47:59,841 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-20 14:47:59,843 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-07-20 14:47:59,844 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics
> system started
> 2012-07-20 14:47:59,969 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ugi registered.
> 2012-07-20 14:48:26,792 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: its-cs100/141.51.205.10:9005. Already tried 0 time(s).
> 2012-07-20 14:48:26,889 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
> FSDatasetStatusMBean
> 2012-07-20 14:48:26,934 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at
> 50010
> 2012-07-20 14:48:26,949 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
> 1048576 bytes/s
> 2012-07-20 14:48:27,014 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-07-20 14:48:27,147 INFO org.apache.hadoop.http.HttpServer: Added

Re: Namenode and Jobtracker dont start

Mohammad Tariq 2012-07-20, 14:58

Hello sir,

If possible, could you please paste your config files??

Regards,
Mohammad Tariq
On Fri, Jul 20, 2012 at 8:24 PM, Björn-Elmar Macek
<[EMAIL PROTECTED]> wrote:
> Hi together,
>
> well just stumbled upon this post:
> http://ankitasblogger.blogspot.de/2012/01/error-that-occured-in-hadoop-and-its.html
>
> And it says:
> "Problem: Hadoop-datanode job failed or datanode not running:
> java.io.IOException: File ../mapred/system/jobtracker.info could only be
> replicated to 0 nodes, instead of 1.
> ...
> Cause: You may also get this message due to permissions. May be JobTracker
> can not create jobtracker.info on startup."
>
> Since the file does not exist i think, this might be a probable reason for
> my errors. But why should the JobTracker not be able to create that file. It
> created several other directories on this node with easy via the slave.sh
> script that i started with the very same user that calls start-all.sh.
>
> Any help would be really appreciated.
>
>
> Am 20.07.2012 16:15, schrieb Björn-Elmar Macek:
>
> Hi Srinivas,
>
> thanks for your reply! I have been following your link and idea and been
> playing around alot, but still got problems with the connection (though they
> are different now):
>
> \_______ JAVA VERSION_________
> "which java" tells me it is 1.6.0_01. If i got it right version 1.7 got
> problems with ssh.
>
> \_______MY TESTS_____________
> According to your suggestion to look for processes running on that port i
> changed ports alot:
> When i was posting the first post of this thread. i was using ports 999 for
> namenode and 1000 for jobtracker.
> Since due to some reasons commands like "lsof -i" etc dont give me any
> output when usedin the cluster enviroment. So i started looking for ports
> that are in general unused by programs.
> When i changed the ports to 9004 and 9005 i got different errors which look
> very much like the ones you posted in the beginning of this year in the
> lucene section (
> http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html
> ).
>
> It seems as if a DataNode can not communicate with the NameNode.
>
> The logs look like the following:
>
> \_______TEST RESULTS__________
> ########## A DataNode #############
> 2012-07-20 14:47:59,536 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = its-cs102.its.uni-kassel.de/141.51.205.12
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.2
> STARTUP_MSG: build > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r
> 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
> ************************************************************/
> 2012-07-20 14:47:59,824 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2012-07-20 14:47:59,841 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-20 14:47:59,843 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2012-07-20 14:47:59,844 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2012-07-20 14:47:59,969 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-07-20 14:48:26,792 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: its-cs100/141.51.205.10:9005. Already tried 0 time(s).
> 2012-07-20 14:48:26,889 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
> FSDatasetStatusMBean
> 2012-07-20 14:48:26,934 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
> 2012-07-20 14:48:26,949 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
> 1048576 bytes/s
> 2012-07-20 14:48:27,014 INFO org.mortbay.log: Logging to

Re: Namenode and Jobtracker dont start

Björn-Elmar Macek 2012-07-20, 15:38

Hi Mohammad,

Thanks for your fast reply. Here they are:

\_____________hadoop-env.sh___
I added those 2 lines:

# The java implementation to use. Required.
export JAVA_HOME=/opt/jdk1.6.0_01/
export JAVA_OPTS="-Djava.net.preferIPv4Stack=true $JAVA_OPTS"
\_____________core-site.xml_____
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
   <property>
   <name>fs.default.name</name>
   <value>hdfs://its-cs100:9005</value>
   </property>
</configuration>
\_____________hdfs-site.xml____
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
   <property>
   <name>dfs.name.dir</name>
   <value>/home/work/bmacek/hadoop/master</value>
   </property>
   
   <property>
   <name>dfs.data.dir</name>
<value>/home/work/bmacek/hadoop/hdfs/slave</value>
   </property>
   <property>
   <name>hadoop.tmp.dir</name>
<value>/home/work/bmacek/hadoop/hdfs/tmp</value>
   </property>

   <property>
   <name>dfs.replication</name>
   <value>1</value>
   </property>
</configuration>
\_______mapred-site.xml____
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
   
   <property>
   <name>mapred.job.tracker</name>
   <value>its-cs100:9004</value>
   </property>
   
   <property>
   <name>dfs.hosts</name>
<value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
   </property>

   <property>
   <name>mapred.hosts</name>
<value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
   </property>
</configuration>

\_______masters____
its-cs101

\_______slaves______
its-cs102
its-cs103
Thats about it, i think. I hope i didnt forget anything.

Regards,
Bjï¿½rn-Elmar

Am 20.07.2012 16:58, schrieb Mohammad Tariq:
> Hello sir,
>
> If possible, could you please paste your config files??
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jul 20, 2012 at 8:24 PM, Bjï¿½rn-Elmar Macek
> <[EMAIL PROTECTED]> wrote:
>> Hi together,
>>
>> well just stumbled upon this post:
>> http://ankitasblogger.blogspot.de/2012/01/error-that-occured-in-hadoop-and-its.html
>>
>> And it says:
>> "Problem: Hadoop-datanode job failed or datanode not running:
>> java.io.IOException: File ../mapred/system/jobtracker.info could only be
>> replicated to 0 nodes, instead of 1.
>> ...
>> Cause: You may also get this message due to permissions. May be JobTracker
>> can not create jobtracker.info on startup."
>>
>> Since the file does not exist i think, this might be a probable reason for
>> my errors. But why should the JobTracker not be able to create that file. It
>> created several other directories on this node with easy via the slave.sh
>> script that i started with the very same user that calls start-all.sh.
>>
>> Any help would be really appreciated.
>>
>>
>> Am 20.07.2012 16:15, schrieb Bjï¿½rn-Elmar Macek:
>>
>> Hi Srinivas,
>>
>> thanks for your reply! I have been following your link and idea and been
>> playing around alot, but still got problems with the connection (though they
>> are different now):
>>
>> \_______ JAVA VERSION_________
>> "which java" tells me it is 1.6.0_01. If i got it right version 1.7 got
>> problems with ssh.
>>
>> \_______MY TESTS_____________
>> According to your suggestion to look for processes running on that port i
>> changed ports alot:
>> When i was posting the first post of this thread. i was using ports 999 for
>> namenode and 1000 for jobtracker.
>> Since due to some reasons commands like "lsof -i" etc dont give me any
>> output when usedin the cluster enviroment. So i started looking for ports
>> that are in general unused by programs.

Re: Namenode and Jobtracker dont start

Harsh J 2012-07-20, 16:01

Hi,

<property>
<name>dfs.hosts</name>
<value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
</property>

This one is probably the cause of all your trouble. It makes the
"hosts" file a white-list of allowed nodes. Ensure, hence, that
"its-cs103.its.uni-kassel.de" is in this file for sure.

Also, dfs.hosts must be in hdfs-site.xml, and mapred.hosts in
mapred-site.xml, but you've got both of them in the latter. You should
fix this up as well.

Or if you do not need such a white-lister feature, just remove both
properties away and restart.

On Fri, Jul 20, 2012 at 9:08 PM, Björn-Elmar Macek
<[EMAIL PROTECTED]> wrote:
> Hi Mohammad,
>
> Thanks for your fast reply. Here they are:
>
> \_____________hadoop-env.sh___
> I added those 2 lines:
>
> # The java implementation to use. Required.
> export JAVA_HOME=/opt/jdk1.6.0_01/
> export JAVA_OPTS="-Djava.net.preferIPv4Stack=true $JAVA_OPTS"
>
>
> \_____________core-site.xml_____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://its-cs100:9005</value>
> </property>
> </configuration>
>
>
> \_____________hdfs-site.xml____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> <property>
> <name>dfs.name.dir</name>
> <value>/home/work/bmacek/hadoop/master</value>
> </property>
> 
> <property>
> <name>dfs.data.dir</name>
> <value>/home/work/bmacek/hadoop/hdfs/slave</value>
> </property>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/home/work/bmacek/hadoop/hdfs/tmp</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> \_______mapred-site.xml____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> 
> <property>
> <name>mapred.job.tracker</name>
> <value>its-cs100:9004</value>
> </property>
> 
> <property>
> <name>dfs.hosts</name>
> <value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
> </property>
>
> <property>
> <name>mapred.hosts</name>
> <value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
> </property>
> </configuration>
>
> \_______masters____
> its-cs101
>
> \_______slaves______
> its-cs102
> its-cs103
>
>
> Thats about it, i think. I hope i didnt forget anything.
>
> Regards,
> Björn-Elmar
>
> Am 20.07.2012 16:58, schrieb Mohammad Tariq:
>
>> Hello sir,
>>
>> If possible, could you please paste your config files??
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Fri, Jul 20, 2012 at 8:24 PM, Björn-Elmar Macek
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi together,
>>>
>>> well just stumbled upon this post:
>>>
>>> http://ankitasblogger.blogspot.de/2012/01/error-that-occured-in-hadoop-and-its.html
>>>
>>> And it says:
>>> "Problem: Hadoop-datanode job failed or datanode not running:
>>> java.io.IOException: File ../mapred/system/jobtracker.info could only be
>>> replicated to 0 nodes, instead of 1.
>>> ...
>>> Cause: You may also get this message due to permissions. May be
>>> JobTracker
>>> can not create jobtracker.info on startup."
>>>
>>> Since the file does not exist i think, this might be a probable reason
>>> for
>>> my errors. But why should the JobTracker not be able to create that file.
>>> It
>>> created several other directories on this node with easy via the slave.sh
>>> script that i started with the very same user that calls start-all.sh.
>>>
>>> Any help would be really appreciated.

Harsh J

Re: Namenode and Jobtracker dont start

Mohammad Tariq 2012-07-20, 15:44

Hi Macek,

hadoop.tmp.dir actually belongs to core-site.xml. So,it would be better
to move it there.

On Friday, July 20, 2012, Björn-Elmar Macek <[EMAIL PROTECTED]> wrote:
> Hi Mohammad,
>
> Thanks for your fast reply. Here they are:
>
> \_____________hadoop-env.sh___
> I added those 2 lines:
>
> # The java implementation to use. Required.
> export JAVA_HOME=/opt/jdk1.6.0_01/
> export JAVA_OPTS="-Djava.net.preferIPv4Stack=true $JAVA_OPTS"
>
>
> \_____________core-site.xml_____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://its-cs100:9005</value>
> </property>
> </configuration>
>
>
> \_____________hdfs-site.xml____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> <property>
> <name>dfs.name.dir</name>
> <value>/home/work/bmacek/hadoop/master</value>
> </property>
> 
> <property>
> <name>dfs.data.dir</name>
> <value>/home/work/bmacek/hadoop/hdfs/slave</value>
> </property>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/home/work/bmacek/hadoop/hdfs/tmp</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> \_______mapred-site.xml____
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> 
>
> <configuration>
> 
> <property>
> <name>mapred.job.tracker</name>
> <value>its-cs100:9004</value>
> </property>
> 
> <property>
> <name>dfs.hosts</name>
> <value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
> </property>
>
> <property>
> <name>mapred.hosts</name>
> <value>/home/fb16/bmacek/hadoop-1.0.2/conf/hosts</value>
> </property>
> </configuration>
>
> \_______masters____
> its-cs101
>
> \_______slaves______
> its-cs102
> its-cs103
>
>
> Thats about it, i think. I hope i didnt forget anything.
>
> Regards,
> Björn-Elmar
>
> Am 20.07.2012 16:58, schrieb Mohammad Tariq:
>
> Hello sir,
>
> If possible, could you please paste your config files??
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jul 20, 2012 at 8:24 PM, Björn-Elmar Macek
> <[EMAIL PROTECTED]> wrote:
>
> Hi together,
>
> well just stumbled upon this post:
>
http://ankitasblogger.blogspot.de/2012/01/error-that-occured-in-hadoop-and-its.html
>
> And it says:
> "Problem: Hadoop-datanode job failed or datanode not running:
> java.io.IOException: File ../mapred/system/jobtracker.info could only be
> replicated to 0 nodes, instead of 1.
> ...
> Cause: You may also get this message due to permissions. May be JobTracker
> can not create jobtracker.info on startup."
>
> Since the file does not exist i think, this might be a probable reason for
> my errors. But why should the JobTracker not be able to create that file.
It
> created several other directories on this node with easy via the slave.sh
> script that i started with the very same user that calls start-all.sh.
>
> Any help would be really appreciated.
>
>
> Am 20.07.2012 16:15, schrieb Björn-Elmar Macek:
>
> Hi Srinivas,
>
> thanks for your reply! I have been following your link and idea and been
> playing around alot, but still got problems with the connection (though
they
> are different now):
>
> \_______ JAVA VERSION_________
> "which java" tells me it is 1.6.0_01. If i got it right version 1.7 got
> problems with ssh.
>
> \_______MY TESTS_____________
> According to your suggestion to look for processes running on that port i
> changed ports alot:
> When i was posting the first post of this thread. i was using ports 999
for
look
http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html
org.apache.hadoop.metrics2.impl.MetricsConfig:
period
connect

Regards,
Mohammad Tariq