Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> memory leak


hi all, I am encountering with high memory usage problem in my production
environment.I doubt this is caused by memory leak or something,
and I hope someone could tell me what is going on , or what should I do to
to keep a lower memory footprint, or any ways or tools to find out what is
causing so much memory footprint.my cluster have 24 nodes, and here is some
info from my random one node.

hbase version:0.90.2
ReadRequest AVG:1,716.02
WriteRequest AVG:435.47
Region count: 281

*top:*

top - 11:19:40 up 530 days, 12:29,  1 user,  load average: 4.10, 3.97, 4.28
Tasks: 239 total,   2 running, 237 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.0%us,  1.2%sy,  0.0%ni, 85.3%id,  7.5%wa,  0.0%hi,  1.0%si,
 0.0%st
Mem:  24676836k total, 24599764k used,    77072k free,    30052k buffers
Swap:  8385760k total,    20568k used,  8365192k free,  1954280k cached

11226 hbase     18   0 23.5g  21g  18m S 75.9 89.5   1219:37 java
  (regionserver)
31579 hbase     19   0 2719m 171m  14m S 35.3  0.7  11502:28 java
 (datanode)
here is my regionserver configuration.

 export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
-XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
+%Y%m%d-%H-%M`.log"
I used google-perftools  to do the heap profiling and i got this.

Total: 213.9 MB
   154.4  72.2%  72.2%    154.4  72.2% os::malloc
    18.8   8.8%  81.0%     20.6   9.7% CMSCollector::CMSCollector
    13.0   6.1%  87.1%     13.0   6.1% ParNewGeneration::ParNewGeneration
     9.6   4.5%  91.6%      9.6   4.5% ObjectSynchronizer::omAlloc
     3.0   1.4%  93.0%      3.0   1.4% init
     2.8   1.3%  94.3%      2.8   1.3% AllocateHeap
     2.3   1.1%  95.3%      2.3   1.1% zcalloc
     1.7   0.8%  96.1%      1.7   0.8% nmethod::nmethod
     1.3   0.6%  96.7%      1.3   0.6% SymbolTable::basic_add
     1.2   0.6%  97.3%      1.4   0.6%
ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
     1.1   0.5%  97.8%      1.1   0.5% Thread::operator new
     0.8   0.4%  98.2%      0.8   0.4% ParkEvent::Allocate
     0.6   0.3%  98.5%      0.6   0.3% readCEN
     0.6   0.3%  98.7%      0.6   0.3% Arena::grow
     0.4   0.2%  98.9%      0.4   0.2% Hashtable::new_entry
     0.3   0.2%  99.1%      0.3   0.2% frame::oops_interpreted_do
     0.3   0.1%  99.2%    150.8  70.5% JavaCalls::call
     0.3   0.1%  99.3%      0.3   0.1% CHeapObj::operator new
     0.2   0.1%  99.4%      0.2   0.1% Hashtable::Hashtable
     0.2   0.1%  99.5%      0.2   0.1% JavaThread::initialize
     0.1   0.1%  99.6%      0.2   0.1%
Deoptimization::fetch_unroll_info_helper
     0.1   0.1%  99.6%      0.1   0.1% os::create_thread
     0.1   0.1%  99.7%      0.1   0.1% _dl_allocate_tls
     0.1   0.0%  99.7%      0.1   0.0% addMetaName
     0.1   0.0%  99.8%      0.1   0.0% vframeArray::allocate
     0.1   0.0%  99.8%      1.8   0.8% ciEnv::register_method
     0.1   0.0%  99.9%      1.6   0.7% Thread::Thread
     0.1   0.0%  99.9%      0.1   0.0%
CompactibleFreeListSpace::CompactibleFreeListSpace
     0.0   0.0%  99.9%      0.1   0.0% WorkGang::allocate_worker
     0.0   0.0%  99.9%      0.0   0.0% __libc_res_nsend
     0.0   0.0%  99.9%      0.0   0.0%
SystemDictionary::validate_protection_domain
     0.0   0.0%  99.9%      0.0   0.0% _dl_new_object
     0.0   0.0%  99.9%      0.0   0.0% growMetaNames
     0.0   0.0%  99.9%      0.0   0.0% CompileBroker::allocate_task
     0.0   0.0% 100.0%      0.0   0.0% allocZip
     0.0   0.0% 100.0%      0.0   0.0% _dlerror_run
     0.0   0.0% 100.0%      0.0   0.0% AdapterHandlerLibrary::get_adapter
     0.0   0.0% 100.0%      0.0   0.0% instanceKlass::get_jmethod_id
     0.0   0.0% 100.0%      0.0   0.0% JVM_RawMonitorCreate
     0.0   0.0% 100.0%      0.0   0.0% PerfDataManager::create_counter
     0.0   0.0% 100.0%      0.0   0.0% newEntry
     0.0   0.0% 100.0%      0.0   0.0% PerfLong::PerfLong
     0.0   0.0% 100.0%      0.0   0.0% Java_java_util_zip_Inflater_init
     0.0   0.0% 100.0%      0.0   0.0% strdup
     0.0   0.0% 100.0%      0.0   0.0% Arguments::add_property
     0.0   0.0% 100.0%      0.0   0.0% JLI_MemAlloc
     0.0   0.0% 100.0%      0.0   0.0% _nl_intern_locale_data
     0.0   0.0% 100.0%      0.0   0.0%
PerfDataManager::create_string_constant
     0.0   0.0% 100.0%      0.0   0.0% SystemProperty::set_value
     0.0   0.0% 100.0%      0.0   0.0% PerfDataManager::create_variable
     0.0   0.0% 100.0%      0.7   0.3% ClassLoader::load_classfile
     0.0   0.0% 100.0%      0.0   0.0% BasicHashtable::BasicHashtable
     0.0   0.0% 100.0%      0.0   0.0% RuntimeStub::new_runtime_stub
     0.0   0.0% 100.0%      0.0   0.0% _dl_check_map_versions
     0.0   0.0% 100.0%      0.0   0.0%
SystemDictionary::add_loader_constraint
     0.0   0.0% 100.0%      0.0   0.0% _dl_map_object_deps
     0.0   0.0% 100.0%      0.0   0.0% StubCodeMark::StubCodeMark
     0.0   0.0% 100.0%      0.0   0.0% PerfDataManager::create_long_constant
     0.0   0.0% 100.0%      0.0   0.0% GCStatInfo::GCStatInfo
     0.0   0.0% 100.0%      0.0   0.0% NamedThread::set_name
     0.0   0.0% 100.0%      0.0   0.0% dl_open_worker
     0.0   0.0% 100.0%      0.0   0.0% nss_parse_service_list
     0.0   0.0% 100.0%      0.0   0.0% getpwuid
     0.0   0.0% 100.0%      0.0   0.0% PerfDataManager::create_long_variable
     0.0   0.0% 100.0%      0.0   0.0% ClassLoader::create_class_path_entry
     0.0   0.0% 100.0%      0.0   0.0% __fopen_internal
     0.0   0.0% 100.0%      0.0   0.0% expand_dynamic_string_token
     0.0   0.0% 100.0%      0.0   0.0%
PerfDataManager::create_string_variable
     0.0   0.0% 100.0%      0.0   0.0% os::format_boot_path
     0.0   0.0% 100.0%      0.0   0.0% MetaIndex::MetaIndex
     0.0   0.0% 100.0%      0.0   0.0% _dl_scope_free
     0.0   0.0% 100.0%      0.0   0.0% __nss_database_lookup
     0.0   0.0% 100.0%      0.0   0.0% __add_to_environ
     0.0   0.0% 100.0%      0.0   0.0% ResourceObj::operator new
  
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB