Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Timeouts in Datanodes while  block scanning


Copy link to this message
-
RE: Timeouts in Datanodes while block scanning
Hi Aaron,
 Presently i am in 0.20.2 version.
I debugged the problem for some time. Could not find any clue. Wanted to know any of the dev/users faced this situation in their clusters.
 
Regards,
Uma
________________________________________
From: Aaron T. Myers [[EMAIL PROTECTED]]
Sent: Thursday, January 05, 2012 11:36 PM
To: [EMAIL PROTECTED]
Subject: Re: Timeouts in Datanodes while block scanning

What version of HDFS? This question might be more appropriate for hdfs-user@
.

--
Aaron T. Myers
Software Engineer, Cloudera

On Thu, Jan 5, 2012 at 8:59 AM, Uma Maheswara Rao G <[EMAIL PROTECTED]>wrote:

> Hi,
>
>  I have 10 Node cluster running from last 25days( running with Hbase
> cluster). Recently observed that for every continuos blocks scans, there
> are many timeouts coming in DataNode.
>  After this block scan verifications, again reads succeeded. This
> situation keep occurring many times now, for every continuous block scans.
>  Here Hbase continuously  performing many random reads.
>
> Whether any one faced this situation in your clusters?
>
> Below is the logs with timeouts.
> 2011-12-28 11:30:42,618 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52764, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251633953_187190
> 2011-12-28 11:30:42,621 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52772, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635735_188342
> 2011-12-28 11:30:42,641 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52796, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251634096_187277
> 2011-12-28 11:30:42,889 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52732, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635763_188363
> 2011-12-28 11:30:42,889 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52637, bytes: 264192, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251634921_187798
> 2011-12-28 11:30:42,976 INFO  DataNode.clienttrace
> (BlockSender.java:sendBlock(529)) - src: /107.252.175.3:10010, dest: /
> 107.252.175.3:52755, bytes: 396288, op: HDFS_READ, cliID:
> DFSClient_hb_rs_107-252-175-3,20020,1324837769603_1324837770095_1770885334_27,
> srvID: DS-306564179-107.252.175.3-10010-1322019943818, blockid:
> blk_1323251635359_188075
> 2011-12-28 11:30:57,757 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251602823_167208
> 2011-12-28 11:32:15,757 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251599175_166755
> 2011-12-28 11:32:54,561 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251673745_194676
> 2011-12-28 11:33:33,561 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
> blk_1323251640709_189383
> 2011-12-28 11:34:12,557 INFO  datanode.DataBlockScanner
> (DataBlockScanner.java:verifyBlock(481)) - Verification succeeded for
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB