Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> NN Memory Jumps every 1 1/2 hours


Copy link to this message
-
Re: NN Memory Jumps every 1 1/2 hours
Ok so here is the latest.

http://imagebin.org/240392

I took a jmap on startup and one an hour after.

http://pastebin.com/xEkWid4f

I think the biggest deal is [B which may not be very helpful

num     #instances         #bytes  class name
----------------------------------------------
   1:      25094067     2319943656  [B
   2:      23720125     1518088000
org.apache.hadoop.hdfs.server.namenode.INodeFile
   3:      24460244     1174091712
org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo
   4:      25671649     1134707328  [Ljava.lang.Object;
   5:      31106937      995421984  java.util.HashMap$Entry
   6:      23725233      570829968
[Lorg.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo;
   7:          2934      322685952  [Ljava.util.HashMap$Entry;

 num     #instances         #bytes  class name
----------------------------------------------
   1:      24739690     3727511000  [B
   2:      23280668     1489962752
org.apache.hadoop.hdfs.server.namenode.INodeFile
   3:      24850044     1192802112
org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo
   4:      26124258     1157073272  [Ljava.lang.Object;
   5:      32142057     1028545824  java.util.HashMap$Entry
   6:      23307473      560625432
[Lorg.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo;

GC starts like this:

3.204: [GC 102656K->9625K(372032K), 0.0150300 secs]
3.519: [GC 112281K->21180K(372032K), 0.0741210 secs]
3.883: [GC 123836K->30729K(372032K), 0.0208900 secs]
4.194: [GC 132724K->45785K(372032K), 0.0293860 secs]
4.522: [GC 148441K->59282K(372032K), 0.0341330 secs]
4.844: [GC 161938K->70071K(372032K), 0.0284850 secs]
5.139: [GC 172727K->80624K(372032K), 0.0171910 secs]
5.338: [GC 183280K->90661K(372032K), 0.0184200 secs]
5.549: [GC 193317K->103126K(372032K), 0.0430080 secs]
5.775: [GC 205782K->113534K(372032K), 0.0359480 secs]
5.995: [GC 216190K->122832K(372032K), 0.0192900 secs]
6.192: [GC 225488K->131777K(372032K), 0.0183870 secs]

Then steadily increases

453.808: [GC 7482139K->7384396K(11240624K), 0.0208170 secs]
455.605: [GC 7487052K->7384177K(11240624K), 0.0206360 secs]
457.942: [GC 7486831K->7384131K(11240624K), 0.0189600 secs]
459.924: [GC 7486787K->7384141K(11240624K), 0.0193560 secs]
462.887: [GC 7486797K->7384151K(11240624K), 0.0189290 secs]
Until I triggered this full gc a moment ago.

6266.988: [GC 11255823K->10373641K(17194656K), 0.0331910 secs]
6280.083: [GC 11259721K->10373499K(17194656K), 0.0324870 secs]
6293.706: [GC 11259579K->10376656K(17194656K), 0.0324330 secs]
6309.781: [GC 11262736K->10376110K(17194656K), 0.0310330 secs]
6333.790: [GC 11262190K->10374348K(17194656K), 0.0297670 secs]
6333.934: [Full GC 10391746K->9722532K(17194656K), 63.9812940 secs]
6418.466: [GC 10608612K->9725743K(17201024K), 0.0339610 secs]
6421.420: [GC 10611823K->9760611K(17201024K), 0.1501610 secs]
6428.221: [GC 10646691K->9767236K(17201024K), 0.1503170 secs]
6437.431: [GC 10653316K->9734750K(17201024K), 0.0344960 secs]

Essentially gc sometimes clears some memory but not all and then the line
keeps rising. Delta is about 10-17 hours until the heap is exhaused.

On Sat, Dec 22, 2012 at 7:03 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> Blocks is ~26,000,000 Files is a bit higher ~27,000,000
>
> Currently running:
> [root@hnn217 ~]# java -version
> java version "1.7.0_09"
> Was running 1.6.0_23
>
> export JVM_OPTIONS="-XX:+UseCompressedOops -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly"
>
> I will grab the gc logs and the heap dump in a follow up.
>
>
>
> On Sat, Dec 22, 2012 at 1:32 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:
>
>> Please take a histo live dump when the memory is full. Note that this
>> causes full gc.
>> http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html
>>
>> What are the number of blocks you have on the system.
>>
>> Send the JVM options you are using. From earlier java versions which used
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB