Dear all,

Yesterday I was working on a cluster of 6 Hadoop nodes ( Load data,
perform some jobs ). But today when I start my cluster I came across a
problem on one of my datanodes.

Datanodes fails to start due to following error :-
2011-01-11 12:54:10,367 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = hadoop3/172.16.1.4
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-01-11 12:55:57,031 INFO
org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: No
locks available
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)

2011-01-11 12:55:57,043 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: No
locks available
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
"hadoop-hadoop-datanode-hadoop3.log" 1775L,
210569C                                                        
1,1           Top
Can Please is familiar with this issue. Please help.
Thanks & Regards

Adarsh Sharma

On Jan 11, 2011, at 2:39 AM, Adarsh Sharma wrote:

> Dear all,
>
> Yesterday I was working on a cluster of 6 Hadoop nodes ( Load data, perform some jobs ). But today when I start my cluster I came across a problem on one of my datanodes.

Are you running this on NFS?

>
> 2011-01-11 12:55:57,031 INFO org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: No locks available

Allen Wittenauer wrote:
> On Jan 11, 2011, at 2:39 AM, Adarsh Sharma wrote:
>
>  
>> Dear all,
>>
>> Yesterday I was working on a cluster of 6 Hadoop nodes ( Load data, perform some jobs ). But today when I start my cluster I came across a problem on one of my datanodes.
>>    
>
> Are you running this on NFS?
>  
No Sir,

I am running this on 3 Servers with local filesystem. Each Server
contains 2 Hard Disks ( /hdd2-1, /hdd1-1 ) and on each servers there run
2 VM's and one occupy /hdd2-1 and the other /hdd1-1.

My Namenode contains all the predefined Ip of VM's.
Thanks
>  
>> 2011-01-11 12:55:57,031 INFO org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: No locks available
>>    

Dear all,
I know this a silly mistake but not able to find the reason of the
exception that causes one datanode to fail to start.

I mount  /hdd2-1 of a phsical machine into this VM and start
datanode,tasktracker.

Datanode fails after few seconds.

Can someone tell  me  the root  cause.

Below is the exception :

2011-01-17 18:01:08,199 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = hadoop7/172.16.1.8
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-01-17 18:03:36,391 INFO
org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: No
locks available
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)

2011-01-17 18:03:36,393 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: No
locks available
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
"~/project/hadoop-0.20.2/logs/hadoop-hadoop-datanode-hadoop7.log" 42L,
3141C                               1,1           Top
Thanks

Adarsh

did you format the namenode before you start? try to format it and start:
1) go to HADOOP_HOME/bin

2) ./hadoop namenode -format

On Mon, Jan 17, 2011 at 1:43 PM, Adarsh Sharma <[EMAIL PROTECTED]>wrote:

> Dear all,
>
>
> I know this a silly mistake but not able to find the reason of the
> exception that causes one datanode to fail to start.
>
> I mount  /hdd2-1 of a phsical machine into this VM and start
> datanode,tasktracker.
>
> Datanode fails after few seconds.
>
> Can someone tell  me  the root  cause.
>
> Below is the exception :
>
> 2011-01-17 18:01:08,199 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = hadoop7/172.16.1.8
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2011-01-17 18:03:36,391 INFO org.apache.hadoop.hdfs.server.common.Storage:
> java.io.IOException: No locks available
>       at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>       at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>       at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
>
> 2011-01-17 18:03:36,393 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: No
> locks available
>       at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>       at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>       at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>       at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> "~/project/hadoop-0.20.2/logs/hadoop-hadoop-datanode-hadoop7.log" 42L,
> 3141C                               1,1           Top
>
>
>
>
> Thanks
>
> Adarsh
>

xiufeng liu wrote:
> did you format the namenode before you start? try to format it and start:
> 1) go to HADOOP_HOME/bin
>
> 2) ./hadoop namenode -format
>  

I format the namenode and then issue the command :

bin/start-all.sh

this results 2 of my datanodes to run properly but causes the below
exception for one datanode.
Can i know why it occurs.
Thanx

> On Mon, Jan 17, 2011 at 1:43 PM, Adarsh Sharma <[EMAIL PROTECTED]>wrote:
>
>  
>> Dear all,
>>
>>
>> I know this a silly mistake but not able to find the reason of the
>> exception that causes one datanode to fail to start.
>>
>> I mount  /hdd2-1 of a phsical machine into this VM and start
>> datanode,tasktracker.
>>
>> Datanode fails after few seconds.
>>
>> Can someone tell  me  the root  cause.
>>
>> Below is the exception :
>>
>> 2011-01-17 18:01:08,199 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG:   host = hadoop7/172.16.1.8
>> STARTUP_MSG:   args = []
>> STARTUP_MSG:   version = 0.20.2
>> STARTUP_MSG:   build >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
>> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> ************************************************************/
>> 2011-01-17 18:03:36,391 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> java.io.IOException: No locks available
>>       at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>>       at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>>       at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
>>
>> 2011-01-17 18:03:36,393 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: No
>> locks available
>>       at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>>       at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>>       at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>       at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
>>       at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
>> "~/project/hadoop-0.20.2/logs/hadoop-hadoop-datanode-hadoop7.log" 42L,
>> 3141C                               1,1           Top
>>
>>
>>
>>
>> Thanks
>>
>> Adarsh
>>
>>    
>
>  

Could you re-check your permissions on the $(dfs.data.dir)s for your
failing DataNode versus the user that runs it?

On Mon, Jan 17, 2011 at 6:33 PM, Adarsh Sharma <[EMAIL PROTECTED]> wrote:
> Can i know why it occurs.

--
Harsh J
www.harshj.com

Harsh J wrote:
> Could you re-check your permissions on the $(dfs.data.dir)s for your
> failing DataNode versus the user that runs it?
>
> On Mon, Jan 17, 2011 at 6:33 PM, Adarsh Sharma <[EMAIL PROTECTED]> wrote:
>  
>> Can i know why it occurs.
>>    
>
>  
Thanx Harsh , I know this issue and I cross-check several times
permissions of of all dirs ( dfs.name.dir, dfs.data.dir, mapred.local.dir ).

It is 755 and is owned by hadoop user and group.

I found that in failed datanode dir , it is unable to create 5 files in
dfs.data.dir whereas on the other hand, it creates following files in
successsful datanode :

curent
tmp
storage
in_use.lock

Does it helps.

Thanx

On Mon, Jan 17, 2011 at 8:13 AM, Adarsh Sharma <[EMAIL PROTECTED]> wrote:
> Harsh J wrote:
>>
>> Could you re-check your permissions on the $(dfs.data.dir)s for your
>> failing DataNode versus the user that runs it?
>>
>> On Mon, Jan 17, 2011 at 6:33 PM, Adarsh Sharma <[EMAIL PROTECTED]>
>> wrote:
>>
>>>
>>> Can i know why it occurs.
>>>
>>
>>
>
> Thanx Harsh , I know this issue and I cross-check several times permissions
> of of all dirs ( dfs.name.dir, dfs.data.dir, mapred.local.dir ).
>
> It is 755 and is owned by hadoop user and group.
>
> I found that in failed datanode dir , it is unable to create 5 files in
> dfs.data.dir whereas on the other hand, it creates following files in
> successsful datanode :
>
> curent
> tmp
> storage
> in_use.lock
>
> Does it helps.
>
> Thanx
>

No locks available can mean that you are trying to use hadoop on a
filesystem that does not support file level locking. Are you trying to
run your name node storage in NFS space?

Edward Capriolo wrote:
> On Mon, Jan 17, 2011 at 8:13 AM, Adarsh Sharma <[EMAIL PROTECTED]> wrote:
>  
>> Harsh J wrote:
>>    
>>> Could you re-check your permissions on the $(dfs.data.dir)s for your
>>> failing DataNode versus the user that runs it?
>>>
>>> On Mon, Jan 17, 2011 at 6:33 PM, Adarsh Sharma <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>      
>>>> Can i know why it occurs.
>>>>
>>>>        
>>>      
>> Thanx Harsh , I know this issue and I cross-check several times permissions
>> of of all dirs ( dfs.name.dir, dfs.data.dir, mapred.local.dir ).
>>
>> It is 755 and is owned by hadoop user and group.
>>
>> I found that in failed datanode dir , it is unable to create 5 files in
>> dfs.data.dir whereas on the other hand, it creates following files in
>> successsful datanode :
>>
>> curent
>> tmp
>> storage
>> in_use.lock
>>
>> Does it helps.
>>
>> Thanx
>>
>>    
>
> No locks available can mean that you are trying to use hadoop on a
> filesystem that does not support file level locking. Are you trying to
> run your name node storage in NFS space?
>  

Yes Edward U'r absolutely right.

I mount hard disk path to the Datanode ( VM ) dfs.data.path.

But it causes no problem in other nodes.

Thanx

Edward Capriolo wrote:
> On Mon, Jan 17, 2011 at 8:13 AM, Adarsh Sharma <[EMAIL PROTECTED]> wrote:
>  
>> Harsh J wrote:
>>    
>>> Could you re-check your permissions on the $(dfs.data.dir)s for your
>>> failing DataNode versus the user that runs it?
>>>
>>> On Mon, Jan 17, 2011 at 6:33 PM, Adarsh Sharma <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>      
>>>> Can i know why it occurs.
>>>>
>>>>        
>>>      
>> Thanx Harsh , I know this issue and I cross-check several times permissions
>> of of all dirs ( dfs.name.dir, dfs.data.dir, mapred.local.dir ).
>>
>> It is 755 and is owned by hadoop user and group.
>>
>> I found that in failed datanode dir , it is unable to create 5 files in
>> dfs.data.dir whereas on the other hand, it creates following files in
>> successsful datanode :
>>
>> curent
>> tmp
>> storage
>> in_use.lock
>>
>> Does it helps.
>>
>> Thanx
>>
>>    
>
> No locks available can mean that you are trying to use hadoop on a
> filesystem that does not support file level locking. Are you trying to
> run your name node storage in NFS space?
>  
I am sorry but my Namenode is in separate Machine outside CLoud.

The path is in /home/hadoop/project/hadoop-0.20.2/name

It's is running properly.

I find it difficult because I followed the same steps in the other 2
VM's and they are running.

How could I debug this for 1 exceptional case where it is failing.
Thanks & Regards

Adarsh Sharma

Hi,

On 2011-01-17 14:28, Adarsh Sharma wrote:
> Edward Capriolo wrote:
>> No locks available can mean that you are trying to use hadoop on a
>> filesystem that does not support file level locking. Are you trying to
>> run your name node storage in NFS space?
>
> Yes Edward U'r absolutely right.
> I mount hard disk path to the Datanode ( VM ) dfs.data.path.
> But it causes no problem in other nodes.

are the NFS locking services up and running?

Benjamin

Benjamin Gufler wrote:
> Hi,
>
> On 2011-01-17 14:28, Adarsh Sharma wrote:
>> Edward Capriolo wrote:
>>> No locks available can mean that you are trying to use hadoop on a
>>> filesystem that does not support file level locking. Are you trying to
>>> run your name node storage in NFS space?
>>
>> Yes Edward U'r absolutely right.
>> I mount hard disk path to the Datanode ( VM ) dfs.data.path.
>> But it causes no problem in other nodes.
>
> are the NFS locking services up and running?
>

Thanx Benjamin ,

I am sorry but I don't know how to check it.

Can you Please tell me how do I find this feature and answer your question.

I think this might be the problem, but the other datanodes doesn't have
this issue.

Please note that my name node storage (dfs.name.dir) is not in NFS.
Best Regards

Adarsh Sharma
> Benjamin

Hi,

On 2011-01-18 12:02, Adarsh Sharma wrote:
> Benjamin Gufler wrote:
>> are the NFS locking services up and running?
> Thanx Benjamin ,
> I am sorry but I don't know how to check it.
> Can you Please tell me how do I find this feature and answer your question.

there should be a "rpc.statd" or "rpc.lockd" process running.
On RedHat basedsystems, you can also check by issuing "service nfslock
status", on Debian based systems it should be something like
"/etc/init.d/nfs-common status".

> I think this might be the problem, but the other datanodes doesn't have
> this issue.
>
> Please note that my name node storage (dfs.name.dir) is not in NFS.

Your first posting looks like the exception occurs in the DataNode, not
in the NameNode.

Best regards, Benjamin
--
Dipl.-Inf. Benjamin Gufler       TU M�nchen, Lehrstuhl Informatik III
Tel. +49 89 289-17250            Fax +49 89 289-17263
Boltzmannstr. 3, 85748 Garching  Raum FMI 02.11.060