Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> performance on memory usage


Copy link to this message
-
Re: performance on memory usage
hadoop will not use/hold on to memory unless its needed.

you push load on the cluster and the stats will automatically grow

On Tue, Jul 24, 2012 at 2:52 PM, Kamil Rogoń
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> Reading on the Internet best practices for selecting hardware for Hadoop I
> noticed there are always many RAM memory. On my Hadoop environment I have
> 16GB memory on all hardware, but I am worried about small utilization of it:
>
> $ free -m
>              total       used       free     shared buffers     cached
> Mem:         15997      15907         90          0 287      15064
> -/+ buffers/cache:        555      15442
> Swap:        15258        150      15108
>
> $ free -m
>              total       used       free     shared buffers     cached
> Mem:         16029      15937         92          0 228      14320
> -/+ buffers/cache:       1388      14641
> Swap:        15258       1017      14240
>
> As you see "buffers used" is below 10%. What options should I look closer? I
> changed "Heap Size" of Cluster, but utilization doesn't grow (Heap Size is
> 70.23 MB / 3.47 GB (1%)).
>
> Current config which can impact on memory:
>
> <name>fs.inmemory.size.mb</name>
> <value>200</value>
>
> <name>io.sort.mb</name>
> <value>200</value>
>
> <name>io.file.buffer.size</name>
> <value>131072</value>
>
> <name>dfs.block.size</name>
> <value>134217728</value>
>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1024M</value>
>
> export HADOOP_HEAPSIZE=4000
>
>
> Thanks for your reply,
> K.R.
>

--
Nitin Pawar