Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - namenode directory disappear after machines restart
Brendan cheng
2012-05-22, 03:21
Marcos Ortiz
2012-05-22, 05:24
Mohammad Tariq
2012-05-22, 07:33
Brendan cheng
2012-05-22, 09:34
Mohammad Tariq
2012-05-22, 09:48
Copy link to this message
-
Re: namenode directory disappear after machines restart
Harsh J 2012-05-22, 13:54
Brendan,

The hdfs-default.xml does have dfs.name.dir listed:
http://hadoop.apache.org/common/docs/current/hdfs-default.html. The
configuration is also mentioned on the official tutorial:
http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuration+Files

On Tue, May 22, 2012 at 3:04 PM, Brendan cheng <[EMAIL PROTECTED]> wrote:
>
> Thanks and it works!
> I wonder where can we find all the settings.  I check the code for hdfs-default.xml but it doesn't have the settings you mentioned.
> Brendan
>
> ----------------------------------------
>> From: [EMAIL PROTECTED]
>> Date: Tue, 22 May 2012 13:03:17 +0530
>> Subject: Re: namenode directory disappear after machines restart
>> To: [EMAIL PROTECTED]
>>
>> Hello Brendan,
>>
>> Do as suggested by Marcos..If you do not set these properties,
>> Hadoop uses "tmp" directory by default..Apart from setting these
>> properties in your hdfs-site.xml file add the following property in
>> your core-site.xml file -
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>some_permanent_path</value>
>> </property>
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Tue, May 22, 2012 at 10:54 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:
>> > This is an usual behavior on Unix/Linux systems. When you restart the
>> > system, the content of the /tmp directory is cleaned, because precisely, the
>> > purpose of this directory is to keep files temporally.
>> > For that reason, the data directory for the HDFS filesystem should be
>> > another directory, /var/hadoop/data for example, of course, a directory
>> > durable in time.
>> > So, you should change your dfs.name.dir and your dfs.data.dir variable in
>> > your hdfs-site.xml.
>> >
>> > Regards
>> >
>> >
>> > On 05/21/2012 11:21 PM, Brendan cheng wrote:
>> >>
>> >> Hi,
>> >> I'm not sure if there is a setting to avoid the Namenode removed after
>> >> hosting machine of Namenode restart.I found that after successfully
>> >> installed single node pseudo distributed hadoop following from your website,
>> >> the name node dir /tmp/hadoop-brendan/dfs/name are removed if machine
>> >> reboot.
>> >> What do I miss?
>> >> Brendan
>> >> 2012-05-22 11:14:05,678 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> >> Storage directory /tmp/hadoop-brendan/dfs/name does not exist.2012-05-22
>> >> 11:14:05,680 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >> FSNamesystem initialization
>> >> failed.org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >> Directory /tmp/hadoop-brendan/dfs/name is in an inconsistent state: storage
>> >> directory does not exist or is not accessible.    at
>> >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
>> >>       at
>> >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>> >> at
>> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>> >>        at
>> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>> >>    at
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>> >>        at
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>> >>  at
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> >>   at
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)2012-05-22
>> >> 11:14:05,685 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
>> >> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
>> >> /tmp/hadoop-brendan/dfs/name is in an inconsistent state: storage directory
>> >> does not exist or is not accessible. at
>> >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
>> >>       at
>> >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>> >> at
>> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)

Harsh J