Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> datanode tuning


+
Rita 2013-10-06, 14:49
+
Ravi Prakash 2013-10-07, 03:09
+
Rita 2013-10-07, 10:55
+
Ravi Prakash 2013-10-07, 14:50
Copy link to this message
-
Re: datanode tuning
For dfs.datanode.scan.period.hours, why isn't it documented here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Also, once these settings are in effect, how can I see that they are
active? is there a JSON page I can login to see them?

On Mon, Oct 7, 2013 at 10:50 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:

> Rita!
>
> 14-16 Tb is perhaps a big node. Even then the scalability limits of the
> Namenode in your case would depend on how many files (more accurately how
> many blocks) there are on HDFS.
>
> In any case, if you want the datanodes to be marked dead quickly when
> their heartbeats are lost, you should reduce the two parameters I told you
> about.
>
> The datanode block scanner is unfortunately hard coded to use a maximum of
> 8Mb/s and a minimum of 1 Mb/s. The only thing you can change is
> dfs.datanode.scan.period.hours
>
> HTH
> Ravi
>
>
> ________________________________
>  From: Rita <[EMAIL PROTECTED]>
> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>;
> Ravi Prakash <[EMAIL PROTECTED]>
> Sent: Monday, October 7, 2013 5:55 AM
> Subject: Re: datanode tuning
>
>
> Thanks Ravi. The number of nodes isn't a lot but the size is rather large.
> Each data node has about 14-16T (560-640T).
>
> For the datanode block scanner, how can increase its "Current scan rate
> limit KBps" ?
>
>
>
>
>
> On Sun, Oct 6, 2013 at 11:09 PM, Ravi Prakash <[EMAIL PROTECTED]> wrote:
>
> > Please look at dfs.heartbeat.interval and
> > dfs.namenode.heartbeat.recheck-interval
> >
> > 40 datanodes is not a large cluster IMHO and the Namenode is capable of
> > managing 100 times more datanodes.
> >
> >
> >
> >
> > ________________________________
> >  From: Rita <[EMAIL PROTECTED]>
> > To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
> > Sent: Sunday, October 6, 2013 9:49 AM
> > Subject: datanode tuning
> >
> >
> > I would like my 40 data nodes to aggressively report to namenode if they
> > are alive or not therefore I think I need to change these params
> >
> > dfs.block.access.token.lifetime : Default is 600 seconds. Can I decrease
> > this to 60?
> >
> >
> > dfs.block.access.key.update.interval: Default is 600 seconds. Can I
> > decrease this to 60?
> >
> > Also, what are some other turnings people do for datanodes in a
> relatively
> > large cluster?
> >
> >
> >
> > --
> > --- Get your facts first, then you can distort them as you please.--
> >
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
>

--
--- Get your facts first, then you can distort them as you please.--
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB