Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - memory leak


+
Yusup Ashrap 2012-11-21, 03:45
Copy link to this message
-
Re: memory leak
ramkrishna vasudevan 2012-11-21, 04:07
Hi Yusup

ARe you using Gzip compression for your storefiles by any chance?

Regards
Ram

On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:

> hi all, I am encountering with high memory usage problem in my production
> environment.I doubt this is caused by memory leak or something,
> and I hope someone could tell me what is going on , or what should I do to
> to keep a lower memory footprint, or any ways or tools to find out what is
> causing so much memory footprint.my cluster have 24 nodes, and here is some
> info from my random one node.
>
> hbase version:0.90.2
> ReadRequest AVG:1,716.02
> WriteRequest AVG:435.47
> Region count: 281
>
> *top:*
>
> top - 11:19:40 up 530 days, 12:29,  1 user,  load average: 4.10, 3.97, 4.28
> Tasks: 239 total,   2 running, 237 sleeping,   0 stopped,   0 zombie
> Cpu(s):  5.0%us,  1.2%sy,  0.0%ni, 85.3%id,  7.5%wa,  0.0%hi,  1.0%si,
>  0.0%st
> Mem:  24676836k total, 24599764k used,    77072k free,    30052k buffers
> Swap:  8385760k total,    20568k used,  8365192k free,  1954280k cached
>
> 11226 hbase     18   0 23.5g  21g  18m S 75.9 89.5   1219:37 java
>   (regionserver)
> 31579 hbase     19   0 2719m 171m  14m S 35.3  0.7  11502:28 java
>  (datanode)
>
>
> here is my regionserver configuration.
>
>  export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
> ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> +%Y%m%d-%H-%M`.log"
>
>
> I used google-perftools  to do the heap profiling and i got this.
>
> Total: 213.9 MB
>    154.4  72.2%  72.2%    154.4  72.2% os::malloc
>     18.8   8.8%  81.0%     20.6   9.7% CMSCollector::CMSCollector
>     13.0   6.1%  87.1%     13.0   6.1% ParNewGeneration::ParNewGeneration
>      9.6   4.5%  91.6%      9.6   4.5% ObjectSynchronizer::omAlloc
>      3.0   1.4%  93.0%      3.0   1.4% init
>      2.8   1.3%  94.3%      2.8   1.3% AllocateHeap
>      2.3   1.1%  95.3%      2.3   1.1% zcalloc
>      1.7   0.8%  96.1%      1.7   0.8% nmethod::nmethod
>      1.3   0.6%  96.7%      1.3   0.6% SymbolTable::basic_add
>      1.2   0.6%  97.3%      1.4   0.6%
> ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
>      1.1   0.5%  97.8%      1.1   0.5% Thread::operator new
>      0.8   0.4%  98.2%      0.8   0.4% ParkEvent::Allocate
>      0.6   0.3%  98.5%      0.6   0.3% readCEN
>      0.6   0.3%  98.7%      0.6   0.3% Arena::grow
>      0.4   0.2%  98.9%      0.4   0.2% Hashtable::new_entry
>      0.3   0.2%  99.1%      0.3   0.2% frame::oops_interpreted_do
>      0.3   0.1%  99.2%    150.8  70.5% JavaCalls::call
>      0.3   0.1%  99.3%      0.3   0.1% CHeapObj::operator new
>      0.2   0.1%  99.4%      0.2   0.1% Hashtable::Hashtable
>      0.2   0.1%  99.5%      0.2   0.1% JavaThread::initialize
>      0.1   0.1%  99.6%      0.2   0.1%
> Deoptimization::fetch_unroll_info_helper
>      0.1   0.1%  99.6%      0.1   0.1% os::create_thread
>      0.1   0.1%  99.7%      0.1   0.1% _dl_allocate_tls
>      0.1   0.0%  99.7%      0.1   0.0% addMetaName
>      0.1   0.0%  99.8%      0.1   0.0% vframeArray::allocate
>      0.1   0.0%  99.8%      1.8   0.8% ciEnv::register_method
>      0.1   0.0%  99.9%      1.6   0.7% Thread::Thread
>      0.1   0.0%  99.9%      0.1   0.0%
> CompactibleFreeListSpace::CompactibleFreeListSpace
>      0.0   0.0%  99.9%      0.1   0.0% WorkGang::allocate_worker
>      0.0   0.0%  99.9%      0.0   0.0% __libc_res_nsend
>      0.0   0.0%  99.9%      0.0   0.0%
> SystemDictionary::validate_protection_domain
>      0.0   0.0%  99.9%      0.0   0.0% _dl_new_object
>      0.0   0.0%  99.9%      0.0   0.0% growMetaNames
>      0.0   0.0%  99.9%      0.0   0.0% CompileBroker::allocate_task
>      0.0   0.0% 100.0%      0.0   0.0% allocZip
>      0.0   0.0% 100.0%      0.0   0.0% _dlerror_run
>      0.0   0.0% 100.0%      0.0   0.0% AdapterHandlerLibrary::get_adapter
>      0.0   0.0% 100.0%      0.0   0.0% instanceKlass:
+
宗丹 2012-11-21, 05:19
+
Yusup Ashrap 2012-11-21, 05:21
+
ramkrishna vasudevan 2012-11-21, 05:33
+
Yusup Ashrap 2012-11-21, 05:52
+
宗丹 2012-11-21, 05:51
+
Stack 2012-11-21, 05:52
+
Yusup Ashrap 2012-11-21, 06:05
+
Stack 2012-11-21, 07:30
+
Yusup Ashrap 2012-11-21, 08:16
+
Suraj Varma 2012-11-21, 08:26
+
Yusup Ashrap 2012-11-21, 08:48
+
Stack 2012-11-21, 17:31
+
宗丹 2012-11-21, 06:04