-Re: DataBlockScanner's rate limit
Harsh J 2013-08-02, 03:27
The DataBlockScanner isn't responsible for the DN block reports at
startup, which is a wholly different thread/process - it is a NN
independent operation that merely verifies blocks in the background
for the DN's own health. Depending on what the outage caused, it is
likely that you are missing a mount and perhaps blocks of files with a
single replica. Run an fsck to identify what files these are and if
they used a single replication factor?
On Fri, Aug 2, 2013 at 7:25 AM, Davey Yan <[EMAIL PROTECTED]> wrote:
> I recently got a mini cluster corrupted after my inappropriate process.
> This mini cluster's dfs.replication was set to 1.
> After irregular restart of OS, I cannot wait to leave safemode, the block
> ratio is 0.9862, < 0.999.
> In the http://ip:50075/blockScannerReport, I notice there is rate limit to
> It will verify the blocks for long time.
> So I "hadoop dfdsadmin safemode leave", and then I got blocks missing.
> My question is: Why should we limit the rate in DataBlockScanner while the
> cluster is still starting up or still in safemode?
> I read the source code of DataBlockScanner.java, there is no parameter to
> change the rate limit.
> It seams to be 1MB to 8MB always.
> Davey Yan