MapReduce, mail # user - DataBlockScanner's rate limit
DataBlockScanner's rate limit
Davey Yan 2013-08-02, 01:55
I recently got a mini cluster corrupted after my inappropriate process.
This mini cluster's dfs.replication was set to 1. After irregular restart of OS, I cannot wait to leave safemode, the block ratio is 0.9862, < 0.999. In the http://ip:50075/blockScannerReport, I notice there is rate limit to 1MB. It will verify the blocks for long time.
So I "hadoop dfdsadmin safemode leave", and then I got blocks missing.
My question is: Why should we limit the rate in DataBlockScanner while the cluster is still starting up or still in safemode?
I read the source code of DataBlockScanner.java, there is no parameter to change the rate limit. It seams to be 1MB to 8MB always. -- Davey Yan
Re: DataBlockScanner's rate limit
Radim Kolar 2013-08-02, 06:02
Another questions: will a single replication factor offen lead to block missing? yes
All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext