-RE: Blocks are getting corrupted under very high load
Uma Maheswara Rao G 2011-11-23, 05:17
Thanks a lot for taking a look.
Yes, I also suspect the same initially. But after analysing the logs, we found below client trace logs in DN, which means block finalization completed.
./hadoop-root-datanode-xx-xx-132-22.log.1:2011-11-20 18:18:45,498 INFO DataNode.clienttrace (BlockReceiver.java:run(1130)) - src: /xx.xx.132.26:55882, dest: /xx.xx.132.22:10010, bytes: 255954944, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_827638122_13, srvID: DS-1518903564-220.127.116.11-10010-1321492867433, blockid: blk_1321803251510_85379
./hadoop-root-datanode-xx-xx-132-22.log.1:2011-11-20 18:18:45,498 INFO datanode.DataNode (BlockReceiver.java:run(1185)) - PacketResponder 0 for block blk_1321803251510_85379 terminating
blk_1321803251510_85379 is recent generationTimeStamp. Also NN logs clearly saying that addStoredBlock called for this block id.
Below are the logs..
./hadoop-root-HANameNode-xx-xx-132-27.log.9:2011-11-20 18:18:15,836 INFO namenode.FSNamesystem (FSNamesystem.java:commitBlockSynchronization(2415)) - commitBlockSynchronization(lastblock=blk_1321803251510_83627, newgenerationstamp=85379, newlength=246505984, newtargets=[xx.xx.132.22:10010], closeFile=false, deleteBlock=false)
./hadoop-root-HANameNode-xx-xx-132-27.log.9:2011-11-20 18:18:15,869 INFO namenode.FSNamesystem (FSNamesystem.java:commitBlockSynchronization(2488)) - commitBlockSynchronization(blk_1321803251510_85379) successful
./hadoop-root-HANameNode-xx-xx-132-27.log.9:2011-11-20 18:18:45,496 WARN namenode.FSNamesystem (FSNamesystem.java:addStoredBlock(3708)) - Inconsistent size for block blk_1321803251510_85379 reported from xx.xx.132.22:10010 current size is 246505984 reported size is 255954944
./hadoop-root-HANameNode-xx-xx-132-27.log.9:2011-11-20 18:18:45,496 WARN hdfs.StateChange (FSNamesystem.java:addStoredBlock(3800)) - BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_1321803251510_85379 on xx.xx.132.22:10010 size 255954944
After DN restart:
./hadoop-root-HANameNode-xx-xx-132-27.log.7:2011-11-20 18:55:54,844 INFO hdfs.StateChange (FSNamesystem.java:rejectAddStoredBlock(3520)) - BLOCK* NameSystem.addStoredBlock: addStoredBlock request received for blk_1321803251510_83627 on xx.xx.132.22:10010 size 104428544 but was rejected: Reported as block being written but is a block of closed file.
From: Todd Lipcon [[EMAIL PROTECTED]]
Sent: Wednesday, November 23, 2011 6:27 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Blocks are getting corrupted under very high load
Can you look on the DN in question and see whether it was succesfully
finalized when the write finished? It doesn't sound like a successful
write -- should have moved it out of the bbw directory into current/
On Tue, Nov 22, 2011 at 3:16 AM, Uma Maheswara Rao G
<[EMAIL PROTECTED]> wrote:
> Hi All,
> I have backported HDFS-1779 to our Hadoop version which is based on 0.20-Append branch.
> We are running a load test, as usual. (We want to ensure the reliability of the system under heavy loads.)
> My cluster has 8 DataNodes and a Namenode
> Each machine has 16 CPUs and 12 hard disks, each having 2TB capacity.
> Clients are running along with Datanodes.
> Clients will upload some tar files containing 3-4 blocks, from 50 threads.
> Each block size is 256MB. replication factor is 3.
> Everything looks to be fine on a normal load.
> When the load is increased, lot of errors are happening.
> Many pipeline failures are happening also.
> All these are fine, except for the strange case of few blocks.
> Some blocks (around 30) are missing (FSCK report shows).
> When I tried to read that files, it fails saying that No Datanodes for this block
> Analysing the logs, we found that, for these blocks, pipeline recovery happened, write was successful to a single Datanode.
> Also, Datanode reported the block to Namenode in a blockReceived command.
> After some time (say, 30 minutes), the Datanode is getting restarted.
Software Engineer, Cloudera