Thanks Harsh,  /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?

Older Path

# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs

# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path

# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,

On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> The exact error is displayed on your log and should be somewhat self
> explanatory:
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> state: storage directory does not exist or is not accessible.
>
> Please check this one's availability, permissions (the NN user should
> be able to access it).
>
> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> <[EMAIL PROTECTED]> wrote:
> > Please find below logs for shutting down the namenode service. Can
> anybody
> > check this
> >
> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack
> > of redundant storage directories!
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> > dataloss due to lack of redundant storage directories!
> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2013-02-28 02:07:51,890 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> > dfs.block.invalidate.limit=1000
> > 2013-02-28 02:07:51,909 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > dfs.block.access.token.enable=false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > defaultReplication         = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> > = 512
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> > = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > maxReplicationStreams      = 2
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > shouldCheckForEnoughRacks  = false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > replicationRecheckInterval = 3000
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > encryptDataTransfer        = false
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> > > hdfs (auth:SIMPLE)
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>  > > hadmin
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
> > > true
> > 2013-02-28 02:07:51,920 INFO

Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.

Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?

2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
[EMAIL PROTECTED]l
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.

Command to format the NN.

sudo -u hdfs hdfs namenode -format

Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <[EMAIL PROTECTED]
> wrote:

> After creating the directory and setting permission I tried to restart the
> services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> acquired by nodename [EMAIL PROTECTED]l" and services are not
> being started.
>
> Need to check few logs from below logs.
> ==================================> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
>
>
> ************************************************************/
> 2013-02-28 05:06:23,385 WARN
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 05:06:25,623 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 05:06:26,015 INFO

On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <[EMAIL PROTECTED]
> wrote:

> mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.

∞
Shashwat Shriparv

And alos check for this parameter

dfs.namenode.safemode.threshold-pct

check the file system if health status is correct.

and if underreplication is there??

∞
Shashwat Shriparv

On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
[EMAIL PROTECTED]> wrote:

>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> mpl: Stopping NameNo
>
>
> Please check how disk space uses is fluctuation while running MR Jobs.
>
>
>
> ∞
> Shashwat Shriparv
>
>

Hi,

May i know the answer plz ?

Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <[EMAIL PROTECTED]
> wrote:

>  Even i created the
> file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
> when i restart hadoop services. It removes and I find below logs.
>
> Do I need to format the NN?
> Below is the command to format the NN ?
> Any kind of loss while formatting ?
> Is there any way to avoid formatting and change the cache path ?
>
> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> [EMAIL PROTECTED]l
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:57:50,905 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
>
> Command to format the NN.
>
> sudo -u hdfs hdfs namenode -format
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> After creating the directory and setting permission I tried to restart
>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> [EMAIL PROTECTED]l" and services are not being started.
>>
>> Need to check few logs from below logs.
>> ==================================>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>>
>>
>> ************************************************************/
>> 2013-02-28 05:06:23,385 WARN
>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:

Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.
On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
<[EMAIL PROTECTED]>wrote:

> thanks Harsh but you didn't answer on this before, I will try to move old
> directory "name" to new location and restart services  . Hope it will not
> loose any data .
>
> old Location
>
> $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
>  drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> New location
>
> $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
> total 8
> drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
> drwxr-xr-x  2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> Thanks,
>
>
> On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> I believe I already responded to another one of your multiple threads
>> on this problem, that you have to move contents of the old dir to the
>> new dir and not format. Thats how one changes the dfs.name.dir
>> properly.
>>
>> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > May i know the answer plz ?
>> >
>> > Thanks,
>> >
>> >
>> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >>  Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >> and set permission . when i restart hadoop services. It removes and I
>> find
>> >> below logs.
>> >>
>> >> Do I need to format the NN?
>> >> Below is the command to format the NN ?
>> >> Any kind of loss while formatting ?
>> >> Is there any way to avoid formatting and change the cache path ?
>> >>
>> >> 2013-02-28 05:57:50,902 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
>> nodename
>> >> [EMAIL PROTECTED]l
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >> system...
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> stopped.
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> shutdown complete.
>> >> 2013-02-28 05:57:50,905 FATAL
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> >> java.io.IOException: NameNode is not formatted.
>> >>
>> >> Command to format the NN.
>> >>
>> >> sudo -u hdfs hdfs namenode -format
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> >> <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> After creating the directory and setting permission I tried to restart
>> >>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >>> acquired by nodename [EMAIL PROTECTED]l" and services are
>> not
>> >>> being started.
>> >>>
>> >>> Need to check few logs from below logs.
>> >>> ==================================>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
Harsh J