So you know that a block is corrupted thanks to an external process which
in this case is checking the parity blocks. If a block is corrupted but
hasn't been detected by HDFS, you could delete the block from the local
filesystem (it's only a file) then HDFS will replicate the good remaining
replica of this block.
For performance reason (and that's what you want to do?), you might be able
to fix the corruption without needing to retrieve the good replica. It
might be possible by working directly with the local system by replacing
the corrupted block by the corrected block (which again are files). On
issue is that the corrected block might be different than the good replica.
If HDFS is able to tell (with CRC) it might be good else you will end up
with two different good replicas for the same block and that will not be
If the result is to be open source, you might want to check with Facebook
about their implementation and track the process within Apache JIRA. You
could gain additional feedbacks. One downside of HDFS RAID is that the less
replicas there is, the less read of the data for processing will be
'efficient/fast'. Reducing the number of replicas also diminishes the
number of supported node failures. I wouldn't say it's an easy ride.
On Mon, Jul 21, 2014 at 1:29 PM, Zesheng Wu <[EMAIL PROTECTED]> wrote: