Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: DataBlockScanner's rate limit


Copy link to this message
-
Re: DataBlockScanner's rate limit
Hi Harsh, thanks for reply.

Yes, dfs.replication was set to 1, but no missing mount.
Another questions: will a single replication factor offen lead to block
missing?
After the startup, the ratio reported in admin ui, e.g. 0.9826, will not
change?  Even the DataBlockScanner is still running?
On Fri, Aug 2, 2013 at 11:27 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> The DataBlockScanner isn't responsible for the DN block reports at
> startup, which is a wholly different thread/process - it is a NN
> independent operation that merely verifies blocks in the background
> for the DN's own health. Depending on what the outage caused, it is
> likely that you are missing a mount and perhaps blocks of files with a
> single replica. Run an fsck to identify what files these are and if
> they used a single replication factor?
>
> On Fri, Aug 2, 2013 at 7:25 AM, Davey Yan <[EMAIL PROTECTED]> wrote:
> > I recently got a mini cluster corrupted after my inappropriate process.
> >
> > This mini cluster's dfs.replication was set to 1.
> > After irregular restart of OS, I cannot wait to leave safemode, the block
> > ratio is 0.9862, < 0.999.
> > In the http://ip:50075/blockScannerReport, I notice there is rate limit
> to
> > 1MB.
> > It will verify the blocks for long time.
> >
> > So I "hadoop dfdsadmin safemode leave", and then I got blocks missing.
> >
> > My question is: Why should we limit the rate in DataBlockScanner while
> the
> > cluster is still starting up or still in safemode?
> >
> > I read the source code of DataBlockScanner.java, there is no parameter to
> > change the rate limit.
> > It seams to be 1MB to 8MB always.
> >
> >
> > --
> > Davey Yan
>
>
>
> --
> Harsh J
>

--
Davey Yan