Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Java Commited Virtual Memory significally larged then Heap Memory


Copy link to this message
-
Re: Java Commited Virtual Memory significally larged then Heap Memory
My bad. All things work. Thanks for  Todd Lipcon :)

2011/1/11 Andrey Stepachev <[EMAIL PROTECTED]>

> I tried to set MALLOC_ARENA_MAX=2. But still the same issue like in LZO
> problem thread. All those 65M blocks here. And JVM continues to eat memory
> on heavy write load. And yes, I use "improved" kernel
> Linux 2.6.34.7-0.5.
>
> 2011/1/11 Xavier Stevens <[EMAIL PROTECTED]>
>
> Are you using a newer linux kernel with the new and "improved" memory
>> allocator?
>>
>> If so try setting this in hadoop-env.sh:
>>
>> export MALLOC_ARENA_MAX=<number of cores you want to use>
>>
>> Maybe start by setting it to 4.  You can thank Todd Lipcon if this works
>> for you.
>>
>> Cheers,
>>
>>
>> -Xavier
>>
>> On 1/11/11 7:24 AM, Andrey Stepachev wrote:
>> > No. I don't use LZO. I tried even remove any native support (i.e. all
>> .so
>> > from class path)
>> > and use java gzip. But nothing.
>> >
>> >
>> > 2011/1/11 Friso van Vollenhoven <[EMAIL PROTECTED]>
>> >
>> >> Are you using LZO by any chance? If so, which version?
>> >>
>> >> Friso
>> >>
>> >>
>> >> On 11 jan 2011, at 15:57, Andrey Stepachev wrote:
>> >>
>> >>> After starting the hbase in jroсkit found the same memory leakage.
>> >>>
>> >>> After the launch
>> >>>
>> >>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head
>> >>> Tue Jan 11 16:49:31 2011
>> >>>
>> >>>   11 16:49:31 MSK 2011
>> >>>   PID RSS VSZ% CPU
>> >>>  7863 2547760 5576744 78.7
>> >>>
>> >>>
>> >>>
>> >>> JR dumps:
>> >>>
>> >>> Total mapped 5576740KB (reserved = 2676404KB) - Java heap 2048000KB
>> >>> (reserved = 1472176KB) - GC tables 68512KB - Thread stacks 37236KB (#
>> >>> threads = 111) - Compiled code 1048576KB (used = 2599KB) - Internal
>> >>> 1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB (malloced
>> >>> = 1110KB # 3285) - Java class data 20224KB (malloced = 20002KB # 15134
>> >>> in 3285 classes) - Native memory tracking 1024KB (malloced = 325KB +10
>> >>> KB # 20)
>> >>>
>> >>>
>> >>>
>> >>> After running the mr which make high write load (~1hour)
>> >>>
>> >>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head
>> >>> Tue Jan 11 17:08:56 2011
>> >>>
>> >>>   11 17:08:56 MSK 2011
>> >>>   PID RSS VSZ% CPU
>> >>>  7863 4072396 5459572 100
>> >>>
>> >>>
>> >>>
>> >>> JR said not important below specify why)
>> >>>
>> >>> http://paste.ubuntu.com/552820/
>> >>> <http://paste.ubuntu.com/552820/>
>> >>>
>> >>>
>> >>> 7863:
>> >>> Total mapped                  5742628KB +165888KB (reserved=1144000KB
>> >>> -1532404KB)
>> >>> -              Java heap      2048000KB           (reserved=0KB
>> >> -1472176KB)
>> >>> -              GC tables        68512KB
>> >>> -          Thread stacks        38028KB    +792KB (#threads=114 +3)
>> >>> -          Compiled code      1048576KB           (used=3376KB +776KB)
>> >>> -               Internal         1480KB    +256KB
>> >>> -                     OS       517944KB  -31744KB
>> >>> -                  Other      1996792KB +195816KB
>> >>> -            Classblocks         1280KB           (malloced=1156KB
>> >>> +45KB #3421 +136)
>> >>> -        Java class data        20992KB    +768KB (malloced=20843KB
>> >>> +840KB #15774 +640 in 3421 classes)
>> >>> - Native memory tracking         1024KB           (malloced=325KB
>> +10KB
>> >> #20)
>> >>>
>> >>>
>> >>
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>    OS                          *java    r x 0x0000000000400000.(
>> >> 76KB)
>> >>>    OS                          *java    rw  0x0000000000612000 (
>> >>  4KB)
>> >>>    OS                        *[heap]    rw  0x0000000000613000.(
>> >> 478712KB)
>> >>>   INT                           Poll    r   0x000000007fffe000 (
>> >>  4KB)
>> >>>   INT                         Membar    rw  0x000000007ffff000.(
>> >>  4KB)
>> >>>   MSP              Classblocks (1/2)    rw  0x0000000082ec0000 (
>> >>  768KB)
>> >>>   MSP              Classblocks (2/2)    rw  0x0000000082f80000 (
>> >>  512KB)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB