Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to do HADOOP RECOVERY ???


Copy link to this message
-
Re: How to do HADOOP RECOVERY ???
Hi Yogesh

You have the dfs.name.dir of the previous install and hence just the metadata alone is available. Hdfs stores the actual blocks in dfs.data.dir which may no longer be there with you. If you have the previous data dir, pointing dfs.data.dir to the old one would resolve your issue.

Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: <[EMAIL PROTECTED]>
Date: Mon, 29 Oct 2012 10:43:44
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: RE: How to do HADOOP RECOVERY ???

Thanks Uma,

I am using hadoop-0.20.2 version.

UI shows.
Cluster Summary
379 files and directories, 270 blocks = 649 total. Heap Size is 81.06 MB / 991.69 MB (8%)

WARNING : There are about 270 missing blocks. Please check the log or run fsck.

Configured Capacity     :       465.44 GB
DFS Used        :       20 KB
Non DFS Used    :       439.37 GB
DFS Remaining   :       26.07 GB
DFS Used%       :       0 %
DFS Remaining%  :       5.6 %
Live Nodes<http://localhost:50070/dfsnodelist.jsp?whatNodes=LIVE>       :       1
Dead Nodes<http://localhost:50070/dfsnodelist.jsp?whatNodes=DEAD>       :       0
Firstly I have configured single node cluster and worked over it, after that I have added another machine and made another one as a master + worker and the fist machine as a worker only.

I have saved the dfs.name.dir seprately, and started with fresh cluster...

Now I have switched back to previous stage with single node with same old machine having single node cluster.
I have given the path for dfs.name.dir where I have kept that.

Now I am running and getting this.

I did -ls / operation and got this exception
mediaadmins-iMac-2:haadoop-0.20.2 mediaadmin$ HADOOP dfs -ls /user/hive/warehouse/vw_cc/
Found 1 items

-rw-r--r--   1 mediaadmin supergroup       1774 2012-10-17 16:15 /user/hive/warehouse/vw_cc/000000_0
mediaadmins-iMac-2:haadoop-0.20.2 mediaadmin$ HADOOP dfs -cat /user/hive/warehouse/vw_cc/000000_0
12/10/29 16:01:15 INFO hdfs.DFSClient: No node available for block: blk_-1280621588594166706_3595 file=/user/hive/warehouse/vw_cc/000000_0
12/10/29 16:01:15 INFO hdfs.DFSClient: Could not obtain block blk_-1280621588594166706_3595 from any node:  java.io.IOException: No live nodes contain current block
12/10/29 16:01:18 INFO hdfs.DFSClient: No node available for block: blk_-1280621588594166706_3595 file=/user/hive/warehouse/vw_cc/000000_0
12/10/29 16:01:18 INFO hdfs.DFSClient: Could not obtain block blk_-1280621588594166706_3595 from any node:  java.io.IOException: No live nodes contain current block
12/10/29 16:01:21 INFO hdfs.DFSClient: No node available for block: blk_-1280621588594166706_3595 file=/user/hive/warehouse/vw_cc/000000_0
12/10/29 16:01:21 INFO hdfs.DFSClient: Could not obtain block blk_-1280621588594166706_3595 from any node:  java.io.IOException: No live nodes contain current block
12/10/29 16:01:24 WARN hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_-1280621588594166706_3595 file=/user/hive/warehouse/vw_cc/000000_0
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1812)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1638)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
    at java.io.DataInputStream.read(DataInputStream.java:83)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
    at org.apache.hadoop.fs.FsShell.printToStdout(FsShell.java:114)
    at org.apache.hadoop.fs.FsShell.access$100(FsShell.java:49)
    at org.apache.hadoop.fs.FsShell$1.process(FsShell.java:352)
    at org.apache.hadoop.fs.FsShell$DelayedExceptionThrowing.globAndProcess(FsShell.java:1898)
    at org.apache.hadoop.fs.FsShell.cat(FsShell.java:346)
I looked at NN Logs for one of the file..

it showing

2012-10-29 15:26:02,560 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=null    ip=null    cmd=open    src=/user/hive/warehouse/vw_cc/000000_0    dst=null    perm=null
.
.
.
.

Please suggest

Regards
Yogesh Kumar

________________________________
From: Uma Maheswara Rao G [[EMAIL PROTECTED]]
Sent: Monday, October 29, 2012 3:52 PM
To: [EMAIL PROTECTED]
Subject: RE: How to do HADOOP RECOVERY ???
Which version of Hadoop are you using?

Do you have all DNs running? can you check UI report, wehther all DN are a live?

Can you check the DN disks are good or not?

Can you grep the NN and DN logs with one of the corrupt blockID from below?

Regards,

Uma

________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]]
Sent: Monday, October 29, 2012 2:03 PM
To: [EMAIL PROTECTED]
Subject: How to do HADOOP RECOVERY ???

Hi All,

I run this command

hadoop fsck -Ddfs.http.address=localhost:50070 /

and found that some blocks are missing and corrupted

results comes like..

/user/hive/warehouse/tt_report_htcount/000000_0: MISSING 2 blocks of total size 71826120 B..
/user/hive/warehouse/tt_report_perhour_hit/000000_0: CORRUPT block blk_75438572351073797

/user/hive/warehouse/tt_report_perhour_hit/000000_0: MISSING 1 blocks of total size 1531 B..
/user/hive/warehouse/vw_cc/000000_0: CORRUPT block blk_-1280621588594166706

/user/hive/warehouse/vw_cc/000000_0: MISSING 1 blocks of total size 1774 B..
/user/hive/warehouse/vw_report2/000000_0: CORRUPT block blk_8637186139854977656

/user/hive/warehouse/vw_report2/000000_0: CORRUPT block blk_4019541597438638886

/user/hive/warehouse/vw_report2/000000_0: MISSING 2 blocks of total size 71826120 B..
/user/zoo/foo.har/_index: CORRUPT block blk_3404803591387558276
.
.
.
.
.

Total size:    7600625746 B
 Total dirs:    205
 Total files:    173
 Total blocks (validated):    270 (avg. block size 28150465 B)
  ********************************
  CORRUPT FILES:    171
  MISSING BLOCKS:    269
  MISSING SIZE:        7600625742 B
  CORRUPT BLOCKS:     269
  ***