Moving from general@ to user@.  The general list is not for technical
questions, it's to discuss project-wide issues.

On Tue, Jan 22, 2013 at 1:03 PM, Mohit Vadhera
<[EMAIL PROTECTED]> wrote:
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it. Below are log messages for
> low space on root / fs. Is there any parameter so that i can reduce
> reserved amount.
>
> 2013-01-21 01:22:52,217 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
> the configured reserved amount 104857600
> 2013-01-21 01:22:52,218 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.

If your Namenode filesystem really only has 10MB available, you are
running well into a red zone for data loss. You cannot safely operate
a NN when the filesystem is that close to being out of space and going
to safemode is appropriate.

The default reserve of 100MB is a good margin of safety for a small
cluster. Instead of decreasing your safety margin, fix your machine so
that there is more than 100MB available on the NN filesystem.

-andy

Edit your hdfs-site.xml (or whatever place of config your NN uses) to lower
the value of property "dfs.namenode.resource.du.reserved". Create a new
property if one does not exist, and set the value of space to a suitable
level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <[EMAIL PROTECTED]
> wrote:

> Ok Steve. I am forwarding my issue again to the list that you said. The
> version is
>
> Hi,
>
> Namenode switches into safemode when it has low disk space on the root fs
> / i have to manually run a command to leave it. Below are log messages for
> low space on root / fs. Is there any parameter so that i can reduce
> reserved amount.Hadoop 2.0.0-cdh4.1.2
>
> 2013-01-21 01:22:52,217 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
> the configured reserved amount 104857600
> 2013-01-21 01:22:52,218 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
>
>
>
> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:
>
>> [EMAIL PROTECTED]ist
>
>
>
>
--
Harsh J

Thanks Guys, As you said the level is already pretty low i.e 100 MB but in
my case the root fs / has 14 G available.  What can be the root cause then ?

/dev/mapper/vg_operamast1-lv_root
                       50G   33G   14G  71% /

As per logs.
    2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
On Wed, Jan 23, 2013 at 11:13 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi again,
>
> Yes, you need to add it to hdfs-site.xml and restart the NN.
>
> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
> service namenode.
> > +  public static final String  DFS_NAMENODE_DU_RESERVED_KEY > "dfs.namenode.resource.du.
> reserved";
> > +  public static final long    DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024 *
> 1024 * 100; // 100 MB
>
>
> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Edit your hdfs-site.xml (or whatever place of config your NN uses) to
>> lower the value of property "dfs.namenode.resource.du.reserved". Create a
>> new property if one does not exist, and set the value of space to a
>> suitable level. The default itself is pretty low - 100 MB in bytes.
>>
>>
>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Ok Steve. I am forwarding my issue again to the list that you said. The
>>> version is
>>>
>>> Hi,
>>>
>>> Namenode switches into safemode when it has low disk space on the root
>>> fs / i have to manually run a command to leave it. Below are log messages
>>> for low space on root / fs. Is there any parameter so that i can reduce
>>> reserved amount.Hadoop 2.0.0-cdh4.1.2
>>>
>>> 2013-01-21 01:22:52,217 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
>>> the configured reserved amount 104857600
>>> 2013-01-21 01:22:52,218 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>>
>>>
>>>
>>> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> [EMAIL PROTECTED]ist
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Harsh J
>

Mohit,

When do you specifically get the error at the NN? Does your NN consistently
not start with that error?

Your local disk space availability can certainly fluctuate if you use the
same disk for MR and other activity which creates temporary files.
On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <[EMAIL PROTECTED]
> wrote:

> Can somebody answer me on this plz ?
>
>
> On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> Thanks Guys, As you said the level is already pretty low i.e 100 MB but
>> in my case the root fs / has 14 G available.  What can be the root cause
>> then ?
>>
>> /dev/mapper/vg_operamast1-lv_root
>>                        50G   33G   14G  71% /
>>
>> As per logs.
>>     2013-01-21 01:22:52,217 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
>> is below the configured reserved amount 104857600
>>
>>
>> On Wed, Jan 23, 2013 at 11:13 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Hi again,
>>>
>>> Yes, you need to add it to hdfs-site.xml and restart the NN.
>>>
>>> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
>>> service namenode.
>>> > +  public static final String  DFS_NAMENODE_DU_RESERVED_KEY >>> "dfs.namenode.resource.du.
>>> reserved";
>>> > +  public static final long    DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024
>>> * 1024 * 100; // 100 MB
>>>
>>>
>>> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>
>>>> Edit your hdfs-site.xml (or whatever place of config your NN uses) to
>>>> lower the value of property "dfs.namenode.resource.du.reserved". Create a
>>>> new property if one does not exist, and set the value of space to a
>>>> suitable level. The default itself is pretty low - 100 MB in bytes.
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Ok Steve. I am forwarding my issue again to the list that you said.
>>>>> The version is
>>>>>
>>>>> Hi,
>>>>>
>>>>> Namenode switches into safemode when it has low disk space on the root
>>>>> fs / i have to manually run a command to leave it. Below are log messages
>>>>> for low space on root / fs. Is there any parameter so that i can reduce
>>>>> reserved amount.Hadoop 2.0.0-cdh4.1.2
>>>>>
>>>>> 2013-01-21 01:22:52,217 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
>>>>> the configured reserved amount 104857600
>>>>> 2013-01-21 01:22:52,218 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Entering safe mode.
>>>>> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ON.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> [EMAIL PROTECTED]ist
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
--
Harsh J

NN switches randomly into the safemode then I run command to leave safemode
manually. I never got alerts for low disk space on machine level and i
didn't see the space fluctuates GBs into MBs .

On Wed, Jan 23, 2013 at 9:10 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Mohit,
>
> When do you specifically get the error at the NN? Does your NN
> consistently not start with that error?
>
> Your local disk space availability can certainly fluctuate if you use the
> same disk for MR and other activity which creates temporary files.
>
>
> On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> Can somebody answer me on this plz ?
>>
>>
>> On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Thanks Guys, As you said the level is already pretty low i.e 100 MB but
>>> in my case the root fs / has 14 G available.  What can be the root cause
>>> then ?
>>>
>>> /dev/mapper/vg_operamast1-lv_root
>>>                        50G   33G   14G  71% /
>>>
>>> As per logs.
>>>     2013-01-21 01:22:52,217 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
>>> is below the configured reserved amount 104857600
>>>
>>>
>>> On Wed, Jan 23, 2013 at 11:13 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi again,
>>>>
>>>> Yes, you need to add it to hdfs-site.xml and restart the NN.
>>>>
>>>> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
>>>> restart service namenode.
>>>> > +  public static final String  DFS_NAMENODE_DU_RESERVED_KEY >>>> "dfs.namenode.resource.du.
>>>> reserved";
>>>> > +  public static final long    DFS_NAMENODE_DU_RESERVED_DEFAULT >>>> 1024 * 1024 * 100; // 100 MB
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Edit your hdfs-site.xml (or whatever place of config your NN uses) to
>>>>> lower the value of property "dfs.namenode.resource.du.reserved". Create a
>>>>> new property if one does not exist, and set the value of space to a
>>>>> suitable level. The default itself is pretty low - 100 MB in bytes.
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Ok Steve. I am forwarding my issue again to the list that you said.
>>>>>> The version is
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Namenode switches into safemode when it has low disk space on the
>>>>>> root fs / i have to manually run a command to leave it. Below are log
>>>>>> messages for low space on root / fs. Is there any parameter so that i can
>>>>>> reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
>>>>>>
>>>>>> 2013-01-21 01:22:52,217 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
>>>>>> the configured reserved amount 104857600
>>>>>> 2013-01-21 01:22:52,218 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Entering safe mode.
>>>>>> 2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ON.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> [EMAIL PROTECTED]ist
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
>
> --
> Harsh J
>

MR operation are running on the same machine. i checked the parameter "
mapred.local.dir" in my installed directory /etc/hadoop/ but didn't find .
One question the disk space reserved size displayed in logs in KB or MB ?
I am layman on hadoop. The link I followed to install is given below

https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

Thanks,
On Wed, Jan 23, 2013 at 10:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> A random switching behavior can only be explained by a fluctuating disk
> space I'd think. Are you running MR operations on the same disk (i.e. is it
> part of mapred.local.dir as well)?
>
>
> On Wed, Jan 23, 2013 at 9:24 PM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> NN switches randomly into the safemode then I run command to leave
>> safemode manually. I never got alerts for low disk space on machine level
>> and i didn't see the space fluctuates GBs into MBs .
>>
>>
>>
>>
>>
>> On Wed, Jan 23, 2013 at 9:10 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Mohit,
>>>
>>> When do you specifically get the error at the NN? Does your NN
>>> consistently not start with that error?
>>>
>>> Your local disk space availability can certainly fluctuate if you use
>>> the same disk for MR and other activity which creates temporary files.
>>>
>>>
>>> On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Can somebody answer me on this plz ?
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Thanks Guys, As you said the level is already pretty low i.e 100 MB
>>>>> but in my case the root fs / has 14 G available.  What can be the root
>>>>> cause then ?
>>>>>
>>>>> /dev/mapper/vg_operamast1-lv_root
>>>>>                        50G   33G   14G  71% /
>>>>>
>>>>> As per logs.
>>>>>     2013-01-21 01:22:52,217 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
>>>>> is below the configured reserved amount 104857600
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 11:13 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi again,
>>>>>>
>>>>>> Yes, you need to add it to hdfs-site.xml and restart the NN.
>>>>>>
>>>>>> > Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
>>>>>> restart service namenode.
>>>>>> > +  public static final String  DFS_NAMENODE_DU_RESERVED_KEY >>>>>> "dfs.namenode.resource.du.
>>>>>> reserved";
>>>>>> > +  public static final long    DFS_NAMENODE_DU_RESERVED_DEFAULT >>>>>> 1024 * 1024 * 100; // 100 MB
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 10:12 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Edit your hdfs-site.xml (or whatever place of config your NN uses)
>>>>>>> to lower the value of property "dfs.namenode.resource.du.reserved". Create
>>>>>>> a new property if one does not exist, and set the value of space to a
>>>>>>> suitable level. The default itself is pretty low - 100 MB in bytes.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>>> Ok Steve. I am forwarding my issue again to the list that you said.
>>>>>>>> The version is
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Namenode switches into safemode when it has low disk space on the
>>>>>>>> root fs / i have to manually run a command to leave it. Below are log
>>>>>>>> messages for low space on root / fs. Is there any parameter so that i can
>>>>>>>> reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
>>>>>>>>
>>>>>>>> 2013-01-21 01:22:52,217 WARN
>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>>>> available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
>>>>>>>> the configured reserved amount 104857600
>>>>>>>> 2013-01-21 01:22:52,218 WARN
>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on

Hi,

The exact error is displayed on your log and should be somewhat self
explanatory:

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.

Please check this one's availability, permissions (the NN user should
be able to access it).

On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
<[EMAIL PROTECTED]> wrote:
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
> of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication         = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams      = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks  = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer        = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             > hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          > hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled > true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
> more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics

Harsh J