Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - datanode tuning


Copy link to this message
-
Re: datanode tuning
Rita 2013-10-07, 21:09
For dfs.datanode.scan.period.hours, why isn't it documented here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Also, once these settings are in effect, how can I see that they are
active? is there a JSON page I can login to see them?

On Mon, Oct 7, 2013 at 10:50 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:

> Rita!
>
> 14-16 Tb is perhaps a big node. Even then the scalability limits of the
> Namenode in your case would depend on how many files (more accurately how
> many blocks) there are on HDFS.
>
> In any case, if you want the datanodes to be marked dead quickly when
> their heartbeats are lost, you should reduce the two parameters I told you
> about.
>
> The datanode block scanner is unfortunately hard coded to use a maximum of
> 8Mb/s and a minimum of 1 Mb/s. The only thing you can change is
> dfs.datanode.scan.period.hours
>
> HTH
> Ravi
>
>
> ________________________________
>  From: Rita <[EMAIL PROTECTED]>
> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>;
> Ravi Prakash <[EMAIL PROTECTED]>
> Sent: Monday, October 7, 2013 5:55 AM
> Subject: Re: datanode tuning
>
>
> Thanks Ravi. The number of nodes isn't a lot but the size is rather large.
> Each data node has about 14-16T (560-640T).
>
> For the datanode block scanner, how can increase its "Current scan rate
> limit KBps" ?
>
>
>
>
>
> On Sun, Oct 6, 2013 at 11:09 PM, Ravi Prakash <[EMAIL PROTECTED]> wrote:
>
> > Please look at dfs.heartbeat.interval and
> > dfs.namenode.heartbeat.recheck-interval
> >
> > 40 datanodes is not a large cluster IMHO and the Namenode is capable of
> > managing 100 times more datanodes.
> >
> >
> >
> >
> > ________________________________
> >  From: Rita <[EMAIL PROTECTED]>
> > To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
> > Sent: Sunday, October 6, 2013 9:49 AM
> > Subject: datanode tuning
> >
> >
> > I would like my 40 data nodes to aggressively report to namenode if they
> > are alive or not therefore I think I need to change these params
> >
> > dfs.block.access.token.lifetime : Default is 600 seconds. Can I decrease
> > this to 60?
> >
> >
> > dfs.block.access.key.update.interval: Default is 600 seconds. Can I
> > decrease this to 60?
> >
> > Also, what are some other turnings people do for datanodes in a
> relatively
> > large cluster?
> >
> >
> >
> > --
> > --- Get your facts first, then you can distort them as you please.--
> >
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
>

--
--- Get your facts first, then you can distort them as you please.--