|
|
Jeffrey Wang 2011-04-12, 23:55
Hey all,
I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
at sun.nio.ch.FileChannelImpl.lock0(Native Method)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
11/04/12 16:47:32 ERROR namenode.NameNode: java.io.IOException: Input/output error
at sun.nio.ch.FileChannelImpl.lock0(Native Method)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
I'm running Hadoop 0.20.2 and never ran into this problem in the several times I've formatted previously, but now it shows up every time and I can't format or start the NameNode. Any ideas? Thanks.
-Jeffrey
-
Re: namenode format error
Harsh J 2011-04-13, 10:48
Hello,
On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <[EMAIL PROTECTED]> wrote:
> Hey all,
>
> I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
>
> 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> at
What's your underlying FS of the ${dfs.name.dir} directories? Are they
NFS mounts?
--
Harsh J
-
RE: namenode format error
Jeffrey Wang 2011-04-13, 19:38
Hi,
It's just in my home directory, which is an NFS mount. I moved it off NFS and it seems to work fine. Is there some reason it doesn't work with NFS?
-Jeffrey
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 13, 2011 3:48 AM
To: [EMAIL PROTECTED]
Subject: Re: namenode format error
Hello,
On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <[EMAIL PROTECTED]> wrote:
> Hey all,
>
> I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
>
> 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> at
What's your underlying FS of the ${dfs.name.dir} directories? Are they
NFS mounts?
--
Harsh J
-
Re: namenode format error
Raghu Angadi 2011-04-13, 20:30
Your NFS mount is not letting NameNode lock a file.
On Wed, Apr 13, 2011 at 12:38 PM, Jeffrey Wang <[EMAIL PROTECTED]> wrote:
> Hi,
>
> It's just in my home directory, which is an NFS mount. I moved it off NFS
> and it seems to work fine. Is there some reason it doesn't work with NFS?
>
> -Jeffrey
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 13, 2011 3:48 AM
> To: [EMAIL PROTECTED]
> Subject: Re: namenode format error
>
> Hello,
>
> On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <[EMAIL PROTECTED]> wrote:
> > Hey all,
> >
> > I'm trying to format my NameNode (I've done it successfully in the past),
> but I'm getting a strange error:
> >
> > 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output
> error
> > at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > at
>
> What's your underlying FS of the ${dfs.name.dir} directories? Are they
> NFS mounts?
>
> --
> Harsh J
>
-
Re: namenode format error
Allen Wittenauer 2011-04-13, 21:07
On Apr 13, 2011, at 12:38 PM, Jeffrey Wang wrote:
> It's just in my home directory, which is an NFS mount. I moved it off NFS and it seems to work fine. Is there some reason it doesn't work with NFS?
Locking on NFS--regardless of application--is a dice roll, especially when client/server are different OSes and platforms. Heck even when they are the same, it might not work. For example, on some older Linux kernels, flock() on NFS only created a lock locally and didn't tell the NFS server, allowing another client to claim a lock! (IIRC, fcntl did work...)
Best bet is to make sure rpc.lockd (or whatever daemon provides NFS locking) is running/working and to use the highest compatible version of NFS. NFSv4, in particular, has file locking built into the protocol rather than as an 'external' service.
|
|
