Keith,

Assuming that you were seeing the problem when you captured the namenode
webUI info, it is not related to what I suspect. This might be a good
question for CDH forums given this is not an Apache release.

Regards,
Suresh

On Tue, Sep 4, 2012 at 10:20 AM, Keith Wiley <[EMAIL PROTECTED]> wrote:

> On Sep 4, 2012, at 10:05 , Suresh Srinivas wrote:
>
> > When these errors are thrown, please send the namenode web UI
> information. It has storage related information in the cluster summary.
> That will help debug.
>
> Sure thing.  Thanks.  Here's what I currently see.  It looks like the
> problem isn't the datanode, but rather the namenode.  Would you agree with
> that assessment?
>
> NameNode 'localhost:9000'
>
> Started:         Tue Sep 04 10:06:52 PDT 2012
> Version:         0.20.2-cdh3u3, 03b655719d13929bd68bb2c2f9cee615b389cea9
> Compiled:        Thu Jan 26 11:55:16 PST 2012 by root from Unknown
> Upgrades:        There are no upgrades in progress.
>
> Browse the filesystem
> Namenode Logs
> Cluster Summary
>
> Safe mode is ON. Resources are low on NN. Safe mode must be turned off
> manually.
> 1639 files and directories, 585 blocks = 2224 total. Heap Size is 39.55 MB
> / 888.94 MB (4%)
> Configured Capacity      :       49.21 GB
> DFS Used         :       9.9 MB
> Non DFS Used     :       2.68 GB
> DFS Remaining    :       46.53 GB
> DFS Used%        :       0.02 %
> DFS Remaining%   :       94.54 %
> Live Nodes       :       1
> Dead Nodes       :       0
> Decommissioning Nodes    :       0
> Number of Under-Replicated Blocks        :       5
>
> NameNode Storage:
>
> Storage Directory       Type    State
> /var/lib/hadoop-0.20/cache/hadoop/dfs/name      IMAGE_AND_EDITS Active
>
> Cloudera's Distribution including Apache Hadoop, 2012.
>
>
> ________________________________________________________________________________
> Keith Wiley     [EMAIL PROTECTED]     keithwiley.com
> music.keithwiley.com
>
> "And what if we picked the wrong religion?  Every week, we're just making
> God
> madder and madder!"
>                                            --  Homer Simpson
>
> ________________________________________________________________________________
>
>
--
http://hortonworks.com/download/

I had moved the data directory to the larger disk but left the namenode directory on the smaller disk figuring it didn't need much room.  Moving that to the larger disk seems to have improved the situation...although I'm still surprised the NN needed so much room.

Problem is solved for now.
Thanks.
________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"I used to be with it, but then they changed what it was.  Now, what I'm with
isn't it, and what's it seems weird and scary to me."
                                           --  Abe (Grandpa) Simpson
________________________________________________________________________________

Hi Keith,

See http://search-hadoop.com/m/z9oYUIhhUg and the method isGoodTarget
under http://search-hadoop.com/c/Hadoop:/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java||isGoodTarget

On Tue, Sep 4, 2012 at 10:24 PM, Keith Wiley <[EMAIL PROTECTED]> wrote:
> If the datanode is definitely not running out of space, and the overall system has basically been working leading up to the "replicated to 0 nodes" error (which proves the configuration and permissions are all basically correct), then what other explanations are there for why hdfs would suddenly start exhibiting this error out of the blue?
>
> Thanks.
>
> ________________________________________________________________________________
> Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com
>
> "Luminous beings are we, not this crude matter."
>                                            --  Yoda
> ________________________________________________________________________________
>

--
Harsh J

Keith,

The NameNode has a resource-checker thread in it by design to help
prevent cases of on-disk metadata corruption in event of filled up
dfs.namenode.name.dir disks, etc.. By default, an NN will lock itself
up if the free disk space (among its configured metadata mounts)
reaches a value < 100 MB, controlled by
dfs.namenode.resource.du.reserved. You can probably set that to 0 if
you do not want such an automatic preventive measure. Its not exactly
a need, just a check to help avoid accidental data loss due to
non-monitoring of disk space.

On Tue, Sep 4, 2012 at 11:33 PM, Keith Wiley <[EMAIL PROTECTED]> wrote:
> I had moved the data directory to the larger disk but left the namenode directory on the smaller disk figuring it didn't need much room.  Moving that to the larger disk seems to have improved the situation...although I'm still surprised the NN needed so much room.
>
> Problem is solved for now.
>
>
> Thanks.
> ________________________________________________________________________________
> Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com
>
> "I used to be with it, but then they changed what it was.  Now, what I'm with
> isn't it, and what's it seems weird and scary to me."
>                                            --  Abe (Grandpa) Simpson
> ________________________________________________________________________________
>

--
Harsh J