Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: DataBlockScanner's rate limit

Copy link to this message
Re: DataBlockScanner's rate limit

The DataBlockScanner isn't responsible for the DN block reports at
startup, which is a wholly different thread/process - it is a NN
independent operation that merely verifies blocks in the background
for the DN's own health. Depending on what the outage caused, it is
likely that you are missing a mount and perhaps blocks of files with a
single replica. Run an fsck to identify what files these are and if
they used a single replication factor?

On Fri, Aug 2, 2013 at 7:25 AM, Davey Yan <[EMAIL PROTECTED]> wrote:
> I recently got a mini cluster corrupted after my inappropriate process.
> This mini cluster's dfs.replication was set to 1.
> After irregular restart of OS, I cannot wait to leave safemode, the block
> ratio is 0.9862, < 0.999.
> In the http://ip:50075/blockScannerReport, I notice there is rate limit to
> 1MB.
> It will verify the blocks for long time.
> So I "hadoop dfdsadmin safemode leave", and then I got blocks missing.
> My question is: Why should we limit the rate in DataBlockScanner while the
> cluster is still starting up or still in safemode?
> I read the source code of DataBlockScanner.java, there is no parameter to
> change the rate limit.
> It seams to be 1MB to 8MB always.
> --
> Davey Yan

Harsh J