Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> performance on memory usage


+
Kamil Rogoń 2012-07-24, 09:22
Copy link to this message
-
Re: performance on memory usage
hadoop will not use/hold on to memory unless its needed.

you push load on the cluster and the stats will automatically grow

On Tue, Jul 24, 2012 at 2:52 PM, Kamil Rogoń
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> Reading on the Internet best practices for selecting hardware for Hadoop I
> noticed there are always many RAM memory. On my Hadoop environment I have
> 16GB memory on all hardware, but I am worried about small utilization of it:
>
> $ free -m
>              total       used       free     shared buffers     cached
> Mem:         15997      15907         90          0 287      15064
> -/+ buffers/cache:        555      15442
> Swap:        15258        150      15108
>
> $ free -m
>              total       used       free     shared buffers     cached
> Mem:         16029      15937         92          0 228      14320
> -/+ buffers/cache:       1388      14641
> Swap:        15258       1017      14240
>
> As you see "buffers used" is below 10%. What options should I look closer? I
> changed "Heap Size" of Cluster, but utilization doesn't grow (Heap Size is
> 70.23 MB / 3.47 GB (1%)).
>
> Current config which can impact on memory:
>
> <name>fs.inmemory.size.mb</name>
> <value>200</value>
>
> <name>io.sort.mb</name>
> <value>200</value>
>
> <name>io.file.buffer.size</name>
> <value>131072</value>
>
> <name>dfs.block.size</name>
> <value>134217728</value>
>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1024M</value>
>
> export HADOOP_HEAPSIZE=4000
>
>
> Thanks for your reply,
> K.R.
>

--
Nitin Pawar
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB