Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: How to set the handler count related parameters?


Copy link to this message
-
Re: How to set the handler count related parameters?
Thanks Nitin!

For dfs.namenode.handler.count, how many memory will be consumed for 1
thread?
2013/12/22 Nitin Pawar <[EMAIL PROTECTED]>

> In general when you have like large hardware machine like the one you have
> got
>
> you can set
> dfs.namenode.handler.count = 64 (you can increase this in proportion to
> your RAM)
>
> dfs.datanode.handler.count is defaulted to 3 but you can raise it to
> around 6 to 10. In some blogs I have read that increasing this will
> increase memory consumption but no performance gains
>
> mapreduce.jobtracker.handler.count this is basically you mean the number
> of server threads you want to run for jobtracker. From the default hadoop
> site xml it is recommended that you keep this size to around 4% of number
> of tasktrackers.
>
> mapred.child.java.opts i am not sure about this. by default its 200M but
> this is setting which is set that the child processes of tasktrackers will
> start with this option if you do not overwrite them from client side. I may
> be wrong in this.
>
>
>
>
>
> On Sun, Dec 22, 2013 at 6:50 PM, sam liu <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> We have 20 nodes cluster(1 namenode, 1 jobtracker, 18 datanodes).  Each
>> node has 20 cpu cores and 64 GB memory.
>>
>> How to set the values for following parameters?
>>
>> -
>>
>> *dfs.namenode.handler.count - dfs.datanode.handler.count- *
>> *mapreduce.jobtracker.handler.count*
>> - *mapred.child.java.opts*
>>
>> Thanks very much!
>>
>
>
>
> --
> Nitin Pawar
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB