Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - Timeouts in Datanodes while  block scanning


+
Uma Maheswara Rao G 2012-01-05, 16:59
+
Aaron T. Myers 2012-01-05, 18:06
Copy link to this message
-
RE: Timeouts in Datanodes while block scanning
Uma Maheswara Rao G 2012-01-06, 01:42
Hi Aaron,
 Presently i am in 0.20.2 version.
I debugged the problem for some time. Could not find any clue. Wanted to know any of the dev/users faced this situation in their clusters.
 
Regards,
Uma
________________________________________
From: Aaron T. Myers [[EMAIL PROTECTED]]
Sent: Thursday, January 05, 2012 11:36 PM
To: [EMAIL PROTECTED]
Subject: Re: Timeouts in Datanodes while block scanning

What version of HDFS? This question might be more appropriate for hdfs-user@
.

--
Aaron T. Myers
Software Engineer, Cloudera

On Thu, Jan 5, 2012 at 8:59 AM, Uma Maheswara Rao G <[EMAIL PROTECTED]>wrote:

> Hi,
>
>  I have 10 Node cluster running from last 25days( running with Hbase
> cluster). Recently observed that for every continuos blocks scans, there
> are many timeouts coming in DataNode.
>  After this block scan verifications, again reads succeeded. This
> situation keep occurring many times now, for every continuous block scans.
>  Here Hbase continuously  performing many random reads.
>
> Whether any one faced this situation in your clusters?
>
> Below is the logs with timeouts.
> 2011-12-28 11:30:42,618 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52764, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251633953_187190
> 2011-12-28 11:30:42,621 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52772, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635735_188342
> 2011-12-28 11:30:42,641 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52796, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251634096_187277
> 2011-12-28 11:30:42,889 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52732, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635763_188363
> 2011-12-28 11:30:42,889 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52637, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251634921_187798
> 2011-12-28 11:30:42,976 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52755, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635359_188075
> 2011-12-28 11:30:57,757 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251602823_167208
> 2011-12-28 11:32:15,757 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251599175_166755
> 2011-12-28 11:32:54,561 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251673745_194676
> 2011-12-28 11:33:33,561 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251640709_189383
> 2011-12-28 11:34:12,557 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for