Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> memory leak


+
Yusup Ashrap 2012-11-21, 03:45
Hi Yusup

ARe you using Gzip compression for your storefiles by any chance?

Regards
Ram

On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:

> hi all, I am encountering with high memory usage problem in my production
> environment.I doubt this is caused by memory leak or something,
> and I hope someone could tell me what is going on , or what should I do to
> to keep a lower memory footprint, or any ways or tools to find out what is
> causing so much memory footprint.my cluster have 24 nodes, and here is some
> info from my random one node.
>
> hbase version:0.90.2
> ReadRequest AVG:1,716.02
> WriteRequest AVG:435.47
> Region count: 281
>
> *top:*
>
> top - 11:19:40 up 530 days, 12:29,  1 user,  load average: 4.10, 3.97, 4.28
> Tasks: 239 total,   2 running, 237 sleeping,   0 stopped,   0 zombie
> Cpu(s):  5.0%us,  1.2%sy,  0.0%ni, 85.3%id,  7.5%wa,  0.0%hi,  1.0%si,
>  0.0%st
> Mem:  24676836k total, 24599764k used,    77072k free,    30052k buffers
> Swap:  8385760k total,    20568k used,  8365192k free,  1954280k cached
>
> 11226 hbase     18   0 23.5g  21g  18m S 75.9 89.5   1219:37 java
>   (regionserver)
> 31579 hbase     19   0 2719m 171m  14m S 35.3  0.7  11502:28 java
>  (datanode)
>
>
> here is my regionserver configuration.
>
>  export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
> ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> +%Y%m%d-%H-%M`.log"
>
>
> I used google-perftools  to do the heap profiling and i got this.
>
> Total: 213.9 MB
>    154.4  72.2%  72.2%    154.4  72.2% os::malloc
>     18.8   8.8%  81.0%     20.6   9.7% CMSCollector::CMSCollector
>     13.0   6.1%  87.1%     13.0   6.1% ParNewGeneration::ParNewGeneration
>      9.6   4.5%  91.6%      9.6   4.5% ObjectSynchronizer::omAlloc
>      3.0   1.4%  93.0%      3.0   1.4% init
>      2.8   1.3%  94.3%      2.8   1.3% AllocateHeap
>      2.3   1.1%  95.3%      2.3   1.1% zcalloc
>      1.7   0.8%  96.1%      1.7   0.8% nmethod::nmethod
>      1.3   0.6%  96.7%      1.3   0.6% SymbolTable::basic_add
>      1.2   0.6%  97.3%      1.4   0.6%
> ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
>      1.1   0.5%  97.8%      1.1   0.5% Thread::operator new
>      0.8   0.4%  98.2%      0.8   0.4% ParkEvent::Allocate
>      0.6   0.3%  98.5%      0.6   0.3% readCEN
>      0.6   0.3%  98.7%      0.6   0.3% Arena::grow
>      0.4   0.2%  98.9%      0.4   0.2% Hashtable::new_entry
>      0.3   0.2%  99.1%      0.3   0.2% frame::oops_interpreted_do
>      0.3   0.1%  99.2%    150.8  70.5% JavaCalls::call
>      0.3   0.1%  99.3%      0.3   0.1% CHeapObj::operator new
>      0.2   0.1%  99.4%      0.2   0.1% Hashtable::Hashtable
>      0.2   0.1%  99.5%      0.2   0.1% JavaThread::initialize
>      0.1   0.1%  99.6%      0.2   0.1%
> Deoptimization::fetch_unroll_info_helper
>      0.1   0.1%  99.6%      0.1   0.1% os::create_thread
>      0.1   0.1%  99.7%      0.1   0.1% _dl_allocate_tls
>      0.1   0.0%  99.7%      0.1   0.0% addMetaName
>      0.1   0.0%  99.8%      0.1   0.0% vframeArray::allocate
>      0.1   0.0%  99.8%      1.8   0.8% ciEnv::register_method
>      0.1   0.0%  99.9%      1.6   0.7% Thread::Thread
>      0.1   0.0%  99.9%      0.1   0.0%
> CompactibleFreeListSpace::CompactibleFreeListSpace
>      0.0   0.0%  99.9%      0.1   0.0% WorkGang::allocate_worker
>      0.0   0.0%  99.9%      0.0   0.0% __libc_res_nsend
>      0.0   0.0%  99.9%      0.0   0.0%
> SystemDictionary::validate_protection_domain
>      0.0   0.0%  99.9%      0.0   0.0% _dl_new_object
>      0.0   0.0%  99.9%      0.0   0.0% growMetaNames
>      0.0   0.0%  99.9%      0.0   0.0% CompileBroker::allocate_task
>      0.0   0.0% 100.0%      0.0   0.0% allocZip
>      0.0   0.0% 100.0%      0.0   0.0% _dlerror_run
>      0.0   0.0% 100.0%      0.0   0.0% AdapterHandlerLibrary::get_adapter
>      0.0   0.0% 100.0%      0.0   0.0% instanceKlass:
+
宗丹 2012-11-21, 05:19
+
Yusup Ashrap 2012-11-21, 05:21
+
ramkrishna vasudevan 2012-11-21, 05:33
+
Yusup Ashrap 2012-11-21, 05:52
+
宗丹 2012-11-21, 05:51
+
Stack 2012-11-21, 05:52
+
Yusup Ashrap 2012-11-21, 06:05
+
Stack 2012-11-21, 07:30
+
Yusup Ashrap 2012-11-21, 08:16
+
Suraj Varma 2012-11-21, 08:26
+
Yusup Ashrap 2012-11-21, 08:48
+
Stack 2012-11-21, 17:31
+
宗丹 2012-11-21, 06:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB