Hello,

I had a running hadoop cluster.
I restarted it and after that namenode is unable to start. I am getting
error saying that it's not formatted. :(
Is it possible to recover the data on HDFS?

2012-08-24 03:17:55,378 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.io.IOException: NameNode is not formatted.
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
2012-08-24 03:17:55,380 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
NameNode is not formatted.
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)

Regards,
Abhay

did you run the command bin/hadoop namenode -format before starting
the namenode ?

On Fri, Aug 24, 2012 at 12:58 PM, Abhay Ratnaparkhi
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> I had a running hadoop cluster.
> I restarted it and after that namenode is unable to start. I am getting
> error saying that it's not formatted. :(
> Is it possible to recover the data on HDFS?
>
> 2012-08-24 03:17:55,378 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.io.IOException: NameNode is not formatted.
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> 2012-08-24 03:17:55,380 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> NameNode is not formatted.
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>
> Regards,
> Abhay
>
>

--
Nitin Pawar

Hi Abhay

What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to /tmp the contents would be deleted on a OS restart. You need to change this location before you start your NN.
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Abhay Ratnaparkhi <[EMAIL PROTECTED]>
Date: Fri, 24 Aug 2012 12:58:41
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: namenode not starting

Hello,

I had a running hadoop cluster.
I restarted it and after that namenode is unable to start. I am getting
error saying that it's not formatted. :(
Is it possible to recover the data on HDFS?

2012-08-24 03:17:55,378 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.io.IOException: NameNode is not formatted.
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
2012-08-24 03:17:55,380 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
NameNode is not formatted.
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)

Regards,
Abhay

hi,
Have u rubn the command namenode -format???
Thanks & regards ,
Vivek

On Fri, Aug 24, 2012 at 12:58 PM, Abhay Ratnaparkhi <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> I had a running hadoop cluster.
> I restarted it and after that namenode is unable to start. I am getting
> error saying that it's not formatted. :(
> Is it possible to recover the data on HDFS?
>
> 2012-08-24 03:17:55,378 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.io.IOException: NameNode is not formatted.
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> 2012-08-24 03:17:55,380 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> NameNode is not formatted.
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>
> Regards,
> Abhay
>
>
>
--

Thanks and Regards,

VIVEK KOUL

You should start with a reboot of the system.

A lesson to everyone, this is exactly why you should have a secondary
name node (http://wiki.apache.org/hadoop/FAQ#What_is_the_purpose_of_the_secondary_name-node.3F)
and run the namenode a mirrored RAID-5/10 disk.
-Håvard

On Fri, Aug 24, 2012 at 9:40 AM, Abhay Ratnaparkhi
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> I was using cluster for long time and not formatted the namenode.
> I ran bin/stop-all.sh and bin/start-all.sh scripts only.
>
> I am using NFS for dfs.name.dir.
> hadoop.tmp.dir is a /tmp directory. I've not restarted the OS.  Any way to
> recover the data?
>
> Thanks,
> Abhay
>
>
> On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
>>
>> Hi Abhay
>>
>> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to
>> /tmp the contents would be deleted on a OS restart. You need to change this
>> location before you start your NN.
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ________________________________
>> From: Abhay Ratnaparkhi <[EMAIL PROTECTED]>
>> Date: Fri, 24 Aug 2012 12:58:41 +0530
>> To: <[EMAIL PROTECTED]>
>> ReplyTo: [EMAIL PROTECTED]
>> Subject: namenode not starting
>>
>> Hello,
>>
>> I had a running hadoop cluster.
>> I restarted it and after that namenode is unable to start. I am getting
>> error saying that it's not formatted. :(
>> Is it possible to recover the data on HDFS?
>>
>> 2012-08-24 03:17:55,378 ERROR
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>> initialization failed.
>> java.io.IOException: NameNode is not formatted.
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>> 2012-08-24 03:17:55,380 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>> NameNode is not formatted.
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>>
>> Regards,
>> Abhay
>>
>>
>

--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/

Abhay,

I suspect that if you haven't set your dfs.name.dir explicitly, then
you haven't set fs.checkpoint.dir either, and since both use
hadoop.tmp.dir paths, you may have lost your data completely and there
is no recovery possible now.

On Fri, Aug 24, 2012 at 1:10 PM, Abhay Ratnaparkhi
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> I was using cluster for long time and not formatted the namenode.
> I ran bin/stop-all.sh and bin/start-all.sh scripts only.
>
> I am using NFS for dfs.name.dir.
> hadoop.tmp.dir is a /tmp directory. I've not restarted the OS.  Any way to
> recover the data?
>
> Thanks,
> Abhay
>
>
> On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
>>
>> Hi Abhay
>>
>> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to
>> /tmp the contents would be deleted on a OS restart. You need to change this
>> location before you start your NN.
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ________________________________
>> From: Abhay Ratnaparkhi <[EMAIL PROTECTED]>
>> Date: Fri, 24 Aug 2012 12:58:41 +0530
>> To: <[EMAIL PROTECTED]>
>> ReplyTo: [EMAIL PROTECTED]
>> Subject: namenode not starting
>>
>> Hello,
>>
>> I had a running hadoop cluster.
>> I restarted it and after that namenode is unable to start. I am getting
>> error saying that it's not formatted. :(
>> Is it possible to recover the data on HDFS?
>>
>> 2012-08-24 03:17:55,378 ERROR
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>> initialization failed.
>> java.io.IOException: NameNode is not formatted.
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>> 2012-08-24 03:17:55,380 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>> NameNode is not formatted.
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>>
>> Regards,
>> Abhay
>>
>>
>

--
Harsh J

Thank you Harsh,

I have set "dfs.name.dir" explicitly. Still don't know why the data loss
has happened.

<property>
  <name>dfs.name.dir</name>
  <value>/wsadfs/${host.name}/name</value>
  <description>Determines where on the local filesystem the DFS name node
      should store the name table.  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>

The secondary namenode was same as namenode. Does this affect  anyway since
path of "dfs.name.dir" were same?
I have now configured another machine as secondary namenode.
I have now  formatted the filesystem since not seen any way of recovering.

I have some questions.

1. Apart from setting secondary namenode what are the other techniques used
for namenode directory backups?
2. Is there any way or tools to recover some of the data even if namenode
crashes.

Regards,
Abhay
On Sat, Aug 25, 2012 at 7:45 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Abhay,
>
> I suspect that if you haven't set your dfs.name.dir explicitly, then
> you haven't set fs.checkpoint.dir either, and since both use
> hadoop.tmp.dir paths, you may have lost your data completely and there
> is no recovery possible now.
>
> On Fri, Aug 24, 2012 at 1:10 PM, Abhay Ratnaparkhi
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > I was using cluster for long time and not formatted the namenode.
> > I ran bin/stop-all.sh and bin/start-all.sh scripts only.
> >
> > I am using NFS for dfs.name.dir.
> > hadoop.tmp.dir is a /tmp directory. I've not restarted the OS.  Any way
> to
> > recover the data?
> >
> > Thanks,
> > Abhay
> >
> >
> > On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hi Abhay
> >>
> >> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to
> >> /tmp the contents would be deleted on a OS restart. You need to change
> this
> >> location before you start your NN.
> >> Regards
> >> Bejoy KS
> >>
> >> Sent from handheld, please excuse typos.
> >> ________________________________
> >> From: Abhay Ratnaparkhi <[EMAIL PROTECTED]>
> >> Date: Fri, 24 Aug 2012 12:58:41 +0530
> >> To: <[EMAIL PROTECTED]>
> >> ReplyTo: [EMAIL PROTECTED]
> >> Subject: namenode not starting
> >>
> >> Hello,
> >>
> >> I had a running hadoop cluster.
> >> I restarted it and after that namenode is unable to start. I am getting
> >> error saying that it's not formatted. :(
> >> Is it possible to recover the data on HDFS?
> >>
> >> 2012-08-24 03:17:55,378 ERROR
> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> >> initialization failed.
> >> java.io.IOException: NameNode is not formatted.
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
> >>         at
> >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> >> 2012-08-24 03:17:55,380 ERROR
> >> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> >> NameNode is not formatted.
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)

Hello Abhay,

    Along with dfs.name.dir, also include dfs.data.dir in hdfs-site.xml.

On Monday, August 27, 2012, Abhay Ratnaparkhi <[EMAIL PROTECTED]>
wrote:
> Thank you Harsh,
> I have set "dfs.name.dir" explicitly. Still don't know why the data loss
has happened.
> <property>
>   <name>dfs.name.dir</name>
>   <value>/wsadfs/${host.name}/name</value>
>   <description>Determines where on the local filesystem the DFS name node
>       should store the name table.  If this is a comma-delimited list
>       of directories then the name table is replicated in all of the
>       directories, for redundancy. </description>
> </property>
> The secondary namenode was same as namenode. Does this affect  anyway
since path of "dfs.name.dir" were same?
> I have now configured another machine as secondary namenode.
> I have now  formatted the filesystem since not seen any way of
recovering.
> I have some questions.
> 1. Apart from setting secondary namenode what are the other techniques
used for namenode directory backups?
> 2. Is there any way or tools to recover some of the data even if namenode
crashes.
> Regards,
> Abhay
>
>
>
>
> On Sat, Aug 25, 2012 at 7:45 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> Abhay,
>
> I suspect that if you haven't set your dfs.name.dir explicitly, then
> you haven't set fs.checkpoint.dir either, and since both use
> hadoop.tmp.dir paths, you may have lost your data completely and there
> is no recovery possible now.
>
> On Fri, Aug 24, 2012 at 1:10 PM, Abhay Ratnaparkhi
> <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>> I was using cluster for long time and not formatted the namenode.
>> I ran bin/stop-all.sh and bin/start-all.sh scripts only.
>>
>> I am using NFS for dfs.name.dir.
>> hadoop.tmp.dir is a /tmp directory. I've not restarted the OS.  Any way
to
>> recover the data?
>>
>> Thanks,
>> Abhay
>>
>>
>> On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi Abhay
>>>
>>> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to
>>> /tmp the contents would be deleted on a OS restart. You need to change
this
>>> location before you start your NN.
>>> Regards
>>> Bejoy KS
>>>
>>> Sent from handheld, please excuse typos.
>>> ________________________________
>>> From: Abhay Ratnaparkhi <[EMAIL PROTECTED]>
>>> Date: Fri, 24 Aug 2012 12:58:41 +0530
>>> To: <[EMAIL PROTECTED]>
>>> ReplyTo: [EMAIL PROTECTED]
>>> Subject: namenode not starting
>>>
>>> Hello,
>>>
>>> I had a running hadoop cluster.
>>> I restarted it and after that namenode is unable to start. I am getting
>>> error saying that it's not formatted. :(
>>> Is it possible to recover the data on HDFS?
>>>
>>> 2012-08-24 03:17:55,378 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>>> initialization failed.
>>> java.io.IOException: NameNode is not formatted.
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:433)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
>>>         at
>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>>> 2012-08-24 03:17:55,380 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>> NameNode is not formatted.
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434)
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:270)

Regards,
    Mohammad Tariq

Abhay,

On Mon, Aug 27, 2012 at 11:19 AM, Abhay Ratnaparkhi
<[EMAIL PROTECTED]> wrote:
> Thank you Harsh,
>
> I have set "dfs.name.dir" explicitly. Still don't know why the data loss has
> happened.
>
> <property>
>   <name>dfs.name.dir</name>
>   <value>/wsadfs/${host.name}/name</value>
>   <description>Determines where on the local filesystem the DFS name node
>       should store the name table.  If this is a comma-delimited list
>       of directories then the name table is replicated in all of the
>       directories, for redundancy. </description>
> </property>

Sorry, I missed you had said NFS above. Is the data not present at all
in that directory there?

> The secondary namenode was same as namenode. Does this affect  anyway since
> path of "dfs.name.dir" were same?
> I have now configured another machine as secondary namenode.
> I have now  formatted the filesystem since not seen any way of recovering.
>
> I have some questions.
>
> 1. Apart from setting secondary namenode what are the other techniques used
> for namenode directory backups?

Duplicate dfs.name.dir directories are what we use in production. That
is, at least two paths, one local FS and another NFS mounted:

dfs.name.dir = /path/to/local/dfs/name,/path/to/nfs/dfs/name

This will give you two copies of good metadata, and loss of one can
still be handled.

> 2. Is there any way or tools to recover some of the data even if namenode
> crashes.

If there's any form of fsimage/edits left, a manual/automated recovery
can be made via tools such as oiv/oev and the NN's "-recover" flag, if
your version has it, or even with a hexdump and some time.

If there's no trace of fsimage files, its backups from any date, any
SNN checkpoints from past, then the metadata is all gone and there's
no recovery.

--
Harsh J

I suggested a while back to Abhay, he should check/track-down  the data on his NFS server / dir.

  Now the ${host.name} looks odd.

  I hope this is something you (Abhay)  edited out and not something that "IS" in the xml file.

  If it is, please fix that.  It needs to be a static name not a variable.

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 27, 2012 12:30 AM
To: [EMAIL PROTECTED]
Subject: Re: namenode not starting

Abhay,

On Mon, Aug 27, 2012 at 11:19 AM, Abhay Ratnaparkhi <[EMAIL PROTECTED]> wrote:
> Thank you Harsh,
>
> I have set "dfs.name.dir" explicitly. Still don't know why the data
> loss has happened.
>
> <property>
>   <name>dfs.name.dir</name>
>   <value>/wsadfs/${host.name}/name</value>
>   <description>Determines where on the local filesystem the DFS name node
>       should store the name table.  If this is a comma-delimited list
>       of directories then the name table is replicated in all of the
>       directories, for redundancy. </description> </property>

Sorry, I missed you had said NFS above. Is the data not present at all in that directory there?

> The secondary namenode was same as namenode. Does this affect  anyway
> since path of "dfs.name.dir" were same?
> I have now configured another machine as secondary namenode.
> I have now  formatted the filesystem since not seen any way of recovering.
>
> I have some questions.
>
> 1. Apart from setting secondary namenode what are the other techniques
> used for namenode directory backups?

Duplicate dfs.name.dir directories are what we use in production. That is, at least two paths, one local FS and another NFS mounted:

dfs.name.dir = /path/to/local/dfs/name,/path/to/nfs/dfs/name

This will give you two copies of good metadata, and loss of one can still be handled.

> 2. Is there any way or tools to recover some of the data even if
> namenode crashes.

If there's any form of fsimage/edits left, a manual/automated recovery can be made via tools such as oiv/oev and the NN's "-recover" flag, if your version has it, or even with a hexdump and some time.

If there's no trace of fsimage files, its backups from any date, any SNN checkpoints from past, then the metadata is all gone and there's no recovery.

--
Harsh J