Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Java Commited Virtual Memory significally larged then Heap Memory


Copy link to this message
-
Re: Java Commited Virtual Memory significally larged then Heap Memory
After starting the hbase in jroсkit found the same memory leakage.

After the launch

 Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head
Tue Jan 11 16:49:31 2011

   11 16:49:31 MSK 2011
   PID RSS VSZ% CPU
  7863 2547760 5576744 78.7

JR dumps:

 Total mapped 5576740KB (reserved = 2676404KB) - Java heap 2048000KB
(reserved = 1472176KB) - GC tables 68512KB - Thread stacks 37236KB (#
threads = 111) - Compiled code 1048576KB (used = 2599KB) - Internal
1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB (malloced
= 1110KB # 3285) - Java class data 20224KB (malloced = 20002KB # 15134
in 3285 classes) - Native memory tracking 1024KB (malloced = 325KB +10
KB # 20)

After running the mr which make high write load (~1hour)

 Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head
Tue Jan 11 17:08:56 2011

   11 17:08:56 MSK 2011
   PID RSS VSZ% CPU
  7863 4072396 5459572 100

JR said not important below specify why)

http://paste.ubuntu.com/552820/
 <http://paste.ubuntu.com/552820/>
7863:
Total mapped                  5742628KB +165888KB (reserved=1144000KB
-1532404KB)
-              Java heap      2048000KB           (reserved=0KB -1472176KB)
-              GC tables        68512KB
-          Thread stacks        38028KB    +792KB (#threads=114 +3)
-          Compiled code      1048576KB           (used=3376KB +776KB)
-               Internal         1480KB    +256KB
-                     OS       517944KB  -31744KB
-                  Other      1996792KB +195816KB
-            Classblocks         1280KB           (malloced=1156KB
+45KB #3421 +136)
-        Java class data        20992KB    +768KB (malloced=20843KB
+840KB #15774 +640 in 3421 classes)
- Native memory tracking         1024KB           (malloced=325KB +10KB #20)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    OS                          *java    r x 0x0000000000400000.(     76KB)
    OS                          *java    rw  0x0000000000612000 (      4KB)
    OS                        *[heap]    rw  0x0000000000613000.( 478712KB)
   INT                           Poll    r   0x000000007fffe000 (      4KB)
   INT                         Membar    rw  0x000000007ffff000.(      4KB)
   MSP              Classblocks (1/2)    rw  0x0000000082ec0000 (    768KB)
   MSP              Classblocks (2/2)    rw  0x0000000082f80000 (    512KB)
  HEAP                      Java heap    rw  0x0000000083000000.(2048000KB)
                                         rw  0x00007f2574000000 (  65500KB)
                                             0x00007f2577ff7000.(     36KB)
                                         rw  0x00007f2584000000 (  65492KB)
                                             0x00007f2587ff5000.(     44KB)
                                         rw  0x00007f258c000000 (  65500KB)
                                             0x00007f258fff7000 (     36KB)
                                         rw  0x00007f2590000000 (  65500KB)
                                             0x00007f2593ff7000 (     36KB)
                                         rw  0x00007f2594000000 (  65500KB)
                                             0x00007f2597ff7000 (     36KB)
                                         rw  0x00007f2598000000 ( 131036KB)
                                             0x00007f259fff7000 (     36KB)
                                         rw  0x00007f25a0000000 (  65528KB)
                                             0x00007f25a3ffe000 (      8KB)
                                         rw  0x00007f25a4000000 (  65496KB)
                                             0x00007f25a7ff6000 (     40KB)
                                         rw  0x00007f25a8000000 (  65496KB)
                                             0x00007f25abff6000 (     40KB)
                                         rw  0x00007f25ac000000 (  65504KB)

So, the difference was in the pieces of memory like this:

 rw 0x00007f2590000000 (65500KB)
     0x00007f2593ff7000 (36KB)
Looks like HLog allocates memory (looks like HLog, becase it is very similar
size)

If we count this blocks we get amount of lost memory:

65M * 32 + 132M = 2212M

So, it looks like HLog allcates to many memory, and question is: how to
restrict it?

2010/12/30 Andrey Stepachev <[EMAIL PROTECTED]>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB