|
|
Yusup Ashrap 2012-11-21, 03:45
hi all, I am encountering with high memory usage problem in my production
environment.I doubt this is caused by memory leak or something,
and I hope someone could tell me what is going on , or what should I do to
to keep a lower memory footprint, or any ways or tools to find out what is
causing so much memory footprint.my cluster have 24 nodes, and here is some
info from my random one node.
hbase version:0.90.2
ReadRequest AVG：1,716.02
WriteRequest AVG：435.47
Region count: 281
*top:*
top - 11:19:40 up 530 days, 12:29, 1 user, load average: 4.10, 3.97, 4.28
Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.0%us, 1.2%sy, 0.0%ni, 85.3%id, 7.5%wa, 0.0%hi, 1.0%si,
0.0%st
Mem: 24676836k total, 24599764k used, 77072k free, 30052k buffers
Swap: 8385760k total, 20568k used, 8365192k free, 1954280k cached
11226 hbase 18 0 23.5g 21g 18m S 75.9 89.5 1219:37 java
(regionserver)
31579 hbase 19 0 2719m 171m 14m S 35.3 0.7 11502:28 java
(datanode)
here is my regionserver configuration.
export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
-XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
+%Y%m%d-%H-%M`.log"
I used google-perftools to do the heap profiling and i got this.
Total: 213.9 MB
154.4 72.2% 72.2% 154.4 72.2% os::malloc
18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
13.0 6.1% 87.1% 13.0 6.1% ParNewGeneration::ParNewGeneration
9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
3.0 1.4% 93.0% 3.0 1.4% init
2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
2.3 1.1% 95.3% 2.3 1.1% zcalloc
1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
1.2 0.6% 97.3% 1.4 0.6%
ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
1.1 0.5% 97.8% 1.1 0.5% Thread::operator new
0.8 0.4% 98.2% 0.8 0.4% ParkEvent::Allocate
0.6 0.3% 98.5% 0.6 0.3% readCEN
0.6 0.3% 98.7% 0.6 0.3% Arena::grow
0.4 0.2% 98.9% 0.4 0.2% Hashtable::new_entry
0.3 0.2% 99.1% 0.3 0.2% frame::oops_interpreted_do
0.3 0.1% 99.2% 150.8 70.5% JavaCalls::call
0.3 0.1% 99.3% 0.3 0.1% CHeapObj::operator new
0.2 0.1% 99.4% 0.2 0.1% Hashtable::Hashtable
0.2 0.1% 99.5% 0.2 0.1% JavaThread::initialize
0.1 0.1% 99.6% 0.2 0.1%
Deoptimization::fetch_unroll_info_helper
0.1 0.1% 99.6% 0.1 0.1% os::create_thread
0.1 0.1% 99.7% 0.1 0.1% _dl_allocate_tls
0.1 0.0% 99.7% 0.1 0.0% addMetaName
0.1 0.0% 99.8% 0.1 0.0% vframeArray::allocate
0.1 0.0% 99.8% 1.8 0.8% ciEnv::register_method
0.1 0.0% 99.9% 1.6 0.7% Thread::Thread
0.1 0.0% 99.9% 0.1 0.0%
CompactibleFreeListSpace::CompactibleFreeListSpace
0.0 0.0% 99.9% 0.1 0.0% WorkGang::allocate_worker
0.0 0.0% 99.9% 0.0 0.0% __libc_res_nsend
0.0 0.0% 99.9% 0.0 0.0%
SystemDictionary::validate_protection_domain
0.0 0.0% 99.9% 0.0 0.0% _dl_new_object
0.0 0.0% 99.9% 0.0 0.0% growMetaNames
0.0 0.0% 99.9% 0.0 0.0% CompileBroker::allocate_task
0.0 0.0% 100.0% 0.0 0.0% allocZip
0.0 0.0% 100.0% 0.0 0.0% _dlerror_run
0.0 0.0% 100.0% 0.0 0.0% AdapterHandlerLibrary::get_adapter
0.0 0.0% 100.0% 0.0 0.0% instanceKlass::get_jmethod_id
0.0 0.0% 100.0% 0.0 0.0% JVM_RawMonitorCreate
0.0 0.0% 100.0% 0.0 0.0% PerfDataManager::create_counter
0.0 0.0% 100.0% 0.0 0.0% newEntry
0.0 0.0% 100.0% 0.0 0.0% PerfLong::PerfLong
0.0 0.0% 100.0% 0.0 0.0% Java_java_util_zip_Inflater_init
0.0 0.0% 100.0% 0.0 0.0% strdup
0.0 0.0% 100.0% 0.0 0.0% Arguments::add_property
0.0 0.0% 100.0% 0.0 0.0% JLI_MemAlloc
0.0 0.0% 100.0% 0.0 0.0% _nl_intern_locale_data
0.0 0.0% 100.0% 0.0 0.0%
PerfDataManager::create_string_constant
0.0 0.0% 100.0% 0.0 0.0% SystemProperty::set_value
0.0 0.0% 100.0% 0.0 0.0% PerfDataManager::create_variable
0.0 0.0% 100.0% 0.7 0.3% ClassLoader::load_classfile
0.0 0.0% 100.0% 0.0 0.0% BasicHashtable::BasicHashtable
0.0 0.0% 100.0% 0.0 0.0% RuntimeStub::new_runtime_stub
0.0 0.0% 100.0% 0.0 0.0% _dl_check_map_versions
0.0 0.0% 100.0% 0.0 0.0%
SystemDictionary::add_loader_constraint
0.0 0.0% 100.0% 0.0 0.0% _dl_map_object_deps
0.0 0.0% 100.0% 0.0 0.0% StubCodeMark::StubCodeMark
0.0 0.0% 100.0% 0.0 0.0% PerfDataManager::create_long_constant
0.0 0.0% 100.0% 0.0 0.0% GCStatInfo::GCStatInfo
0.0 0.0% 100.0% 0.0 0.0% NamedThread::set_name
0.0 0.0% 100.0% 0.0 0.0% dl_open_worker
0.0 0.0% 100.0% 0.0 0.0% nss_parse_service_list
0.0 0.0% 100.0% 0.0 0.0% getpwuid
0.0 0.0% 100.0% 0.0 0.0% PerfDataManager::create_long_variable
0.0 0.0% 100.0% 0.0 0.0% ClassLoader::create_class_path_entry
0.0 0.0% 100.0% 0.0 0.0% __fopen_internal
0.0 0.0% 100.0% 0.0 0.0% expand_dynamic_string_token
0.0 0.0% 100.0% 0.0 0.0%
PerfDataManager::create_string_variable
0.0 0.0% 100.0% 0.0 0.0% os::format_boot_path
0.0 0.0% 100.0% 0.0 0.0% MetaIndex::MetaIndex
0.0 0.0% 100.0% 0.0 0.0% _dl_scope_free
0.0 0.0% 100.0% 0.0 0.0% __nss_database_lookup
0.0 0.0% 100.0% 0.0 0.0% __add_to_environ
0.0 0.0% 100.0% 0.0 0.0% ResourceObj::operator new
ramkrishna vasudevan 2012-11-21, 04:07
Hi Yusup
ARe you using Gzip compression for your storefiles by any chance?
Regards
Ram
On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
> hi all, I am encountering with high memory usage problem in my production
> environment.I doubt this is caused by memory leak or something,
> and I hope someone could tell me what is going on , or what should I do to
> to keep a lower memory footprint, or any ways or tools to find out what is
> causing so much memory footprint.my cluster have 24 nodes, and here is some
> info from my random one node.
>
> hbase version:0.90.2
> ReadRequest AVG：1,716.02
> WriteRequest AVG：435.47
> Region count: 281
>
> *top:*
>
> top - 11:19:40 up 530 days, 12:29, 1 user, load average: 4.10, 3.97, 4.28
> Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
> Cpu(s): 5.0%us, 1.2%sy, 0.0%ni, 85.3%id, 7.5%wa, 0.0%hi, 1.0%si,
> 0.0%st
> Mem: 24676836k total, 24599764k used, 77072k free, 30052k buffers
> Swap: 8385760k total, 20568k used, 8365192k free, 1954280k cached
>
> 11226 hbase 18 0 23.5g 21g 18m S 75.9 89.5 1219:37 java
> (regionserver)
> 31579 hbase 19 0 2719m 171m 14m S 35.3 0.7 11502:28 java
> (datanode)
>
>
> here is my regionserver configuration.
>
> export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
> ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> +%Y%m%d-%H-%M`.log"
>
>
> I used google-perftools to do the heap profiling and i got this.
>
> Total: 213.9 MB
> 154.4 72.2% 72.2% 154.4 72.2% os::malloc
> 18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
> 13.0 6.1% 87.1% 13.0 6.1% ParNewGeneration::ParNewGeneration
> 9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
> 3.0 1.4% 93.0% 3.0 1.4% init
> 2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
> 2.3 1.1% 95.3% 2.3 1.1% zcalloc
> 1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
> 1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
> 1.2 0.6% 97.3% 1.4 0.6%
> ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
> 1.1 0.5% 97.8% 1.1 0.5% Thread::operator new
> 0.8 0.4% 98.2% 0.8 0.4% ParkEvent::Allocate
> 0.6 0.3% 98.5% 0.6 0.3% readCEN
> 0.6 0.3% 98.7% 0.6 0.3% Arena::grow
> 0.4 0.2% 98.9% 0.4 0.2% Hashtable::new_entry
> 0.3 0.2% 99.1% 0.3 0.2% frame::oops_interpreted_do
> 0.3 0.1% 99.2% 150.8 70.5% JavaCalls::call
> 0.3 0.1% 99.3% 0.3 0.1% CHeapObj::operator new
> 0.2 0.1% 99.4% 0.2 0.1% Hashtable::Hashtable
> 0.2 0.1% 99.5% 0.2 0.1% JavaThread::initialize
> 0.1 0.1% 99.6% 0.2 0.1%
> Deoptimization::fetch_unroll_info_helper
> 0.1 0.1% 99.6% 0.1 0.1% os::create_thread
> 0.1 0.1% 99.7% 0.1 0.1% _dl_allocate_tls
> 0.1 0.0% 99.7% 0.1 0.0% addMetaName
> 0.1 0.0% 99.8% 0.1 0.0% vframeArray::allocate
> 0.1 0.0% 99.8% 1.8 0.8% ciEnv::register_method
> 0.1 0.0% 99.9% 1.6 0.7% Thread::Thread
> 0.1 0.0% 99.9% 0.1 0.0%
> CompactibleFreeListSpace::CompactibleFreeListSpace
> 0.0 0.0% 99.9% 0.1 0.0% WorkGang::allocate_worker
> 0.0 0.0% 99.9% 0.0 0.0% __libc_res_nsend
> 0.0 0.0% 99.9% 0.0 0.0%
> SystemDictionary::validate_protection_domain
> 0.0 0.0% 99.9% 0.0 0.0% _dl_new_object
> 0.0 0.0% 99.9% 0.0 0.0% growMetaNames
> 0.0 0.0% 99.9% 0.0 0.0% CompileBroker::allocate_task
> 0.0 0.0% 100.0% 0.0 0.0% allocZip
> 0.0 0.0% 100.0% 0.0 0.0% _dlerror_run
> 0.0 0.0% 100.0% 0.0 0.0% AdapterHandlerLibrary::get_adapter
> 0.0 0.0% 100.0% 0.0 0.0% instanceKlass:
5 tables on this node is using gzip compression.
At first I assumed this was caused by gzip compression, but as u can see from my google-perftools profiling result above that
there is no gzip related compression memory allocated.
On Wed, Nov 21, 2012 at 12:07 PM, ramkrishna vasudevan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Yusup
ARe you using Gzip compression for your storefiles by any chance?
Regards
Ram
On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> hi all, I am encountering with high memory usage problem in my production
> environment.I doubt this is caused by memory leak or something,
> and I hope someone could tell me what is going on , or what should I do to
> to keep a lower memory footprint, or any ways or tools to find out what is
> causing so much memory footprint.my cluster have 24 nodes, and here is some
> info from my random one node.
>
> hbase version:0.90.2
> ReadRequest AVG：1,716.02
> WriteRequest AVG：435.47
> Region count: 281
>
> *top:*
>
> top - 11:19:40 up 530 days, 12:29, 1 user, load average: 4.10, 3.97, 4.28
> Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
> Cpu(s): 5.0%us, 1.2%sy, 0.0%ni, 85.3%id, 7.5%wa, 0.0%hi, 1.0%si,
> 0.0%st
> Mem: 24676836k total, 24599764k used, 77072k free, 30052k buffers
> Swap: 8385760k total, 20568k used, 8365192k free, 1954280k cached
>
> 11226 hbase 18 0 23.5g 21g 18m S 75.9 89.5 1219:37 java
> (regionserver)
> 31579 hbase 19 0 2719m 171m 14m S 35.3 0.7 11502:28 java
> (datanode)
>
>
> here is my regionserver configuration.
>
> export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingO
> ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> +%Y%m%d-%H-%M`.log"
>
>
> I used google-perftools to do the heap profiling and i got this.
>
> Total: 213.9 MB
> 154.4 72.2% 72.2% 154.4 72.2% os::malloc
> 18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
> 13.0 6.1% 87.1% 13.0 6.1% ParNewGeneration::ParNewGeneration
> 9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
> 3.0 1.4% 93.0% 3.0 1.4% init
> 2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
> 2.3 1.1% 95.3% 2.3 1.1% zcalloc
> 1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
> 1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
> 1.2 0.6% 97.3% 1.4 0.6%
> ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
> 1.1 0.5% 97.8% 1.1 0.5% Thread::operator new
> 0.8 0.4% 98.2% 0.8 0.4% ParkEvent::Allocate
> 0.6 0.3% 98.5% 0.6 0.3% readCEN
> 0.6 0.3% 98.7% 0.6 0.3% Arena::grow
> 0.4 0.2% 98.9% 0.4 0.2% Hashtable::new_entry
> 0.3 0.2% 99.1% 0.3 0.2% frame::oops_interpreted_do
> 0.3 0.1% 99.2% 150.8 70.5% JavaCalls::call
> 0.3 0.1% 99.3% 0.3 0.1% CHeapObj::operator new
> 0.2 0.1% 99.4% 0.2 0.1% Hashtable::Hashtable
> 0.2 0.1% 99.5% 0.2 0.1% JavaThread::initialize
> 0.1 0.1% 99.6% 0.2 0.1%
> Deoptimization::fetch_unroll_info_helper
> 0.1 0.1% 99.6% 0.1 0.1% os::create_thread
> 0.1 0.1% 99.7% 0.1 0.1% _dl_allocate_tls
> 0.1 0.0% 99.7% 0.1 0.0% addMetaName
> 0.1 0.0% 99.8% 0.1 0.0% vframeArray::allocate
> 0.1 0.0% 99.8% 1.8 0.8% ciEnv::register_method
> 0.1 0.0% 99.9% 1.6 0.7% Thread::Thread
> 0.1 0.0% 99.9% 0.1 0.0%
> CompactibleFreeListSpace::CompactibleFreeListSpace
> 0.0 0.0% 99.9% 0.1 0.0% WorkGang::allocate_worker
> 0.0 0.0% 99.9% 0.0 0.0% __libc_res_nsend
> 0.0 0.0% 99.9% 0.0 0.0%
> SystemDictionary::validate_protection_domain
> 0.0 0.0% 99.9% 0.0 0.0% _dl_new_object
________________________________
This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Yusup Ashrap 2012-11-21, 05:21
hi Ramkrishna
5 tables on this node is using gzip compression.
At first I assumed this was caused by gzip compression, but as u can see
from my google-perftools profiling result above that
there is no gzip related compression memory allocated.(or am I misreading
profiling resutls?)
thanks,regards
On Wed, Nov 21, 2012 at 12:07 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:
> Hi Yusup
>
> ARe you using Gzip compression for your storefiles by any chance?
>
> Regards
> Ram
>
> On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
>
> > hi all, I am encountering with high memory usage problem in my production
> > environment.I doubt this is caused by memory leak or something,
> > and I hope someone could tell me what is going on , or what should I do
> to
> > to keep a lower memory footprint, or any ways or tools to find out what
> is
> > causing so much memory footprint.my cluster have 24 nodes, and here is
> some
> > info from my random one node.
> >
> > hbase version:0.90.2
> > ReadRequest AVG：1,716.02
> > WriteRequest AVG：435.47
> > Region count: 281
> >
> > *top:*
> >
> > top - 11:19:40 up 530 days, 12:29, 1 user, load average: 4.10, 3.97,
> 4.28
> > Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 5.0%us, 1.2%sy, 0.0%ni, 85.3%id, 7.5%wa, 0.0%hi, 1.0%si,
> > 0.0%st
> > Mem: 24676836k total, 24599764k used, 77072k free, 30052k buffers
> > Swap: 8385760k total, 20568k used, 8365192k free, 1954280k cached
> >
> > 11226 hbase 18 0 23.5g 21g 18m S 75.9 89.5 1219:37 java
> > (regionserver)
> > 31579 hbase 19 0 2719m 171m 14m S 35.3 0.7 11502:28 java
> > (datanode)
> >
> >
> > here is my regionserver configuration.
> >
> > export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> > -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingO
> > ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> > +%Y%m%d-%H-%M`.log"
> >
> >
> > I used google-perftools to do the heap profiling and i got this.
> >
> > Total: 213.9 MB
> > 154.4 72.2% 72.2% 154.4 72.2% os::malloc
> > 18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
> > 13.0 6.1% 87.1% 13.0 6.1% ParNewGeneration::ParNewGeneration
> > 9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
> > 3.0 1.4% 93.0% 3.0 1.4% init
> > 2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
> > 2.3 1.1% 95.3% 2.3 1.1% zcalloc
> > 1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
> > 1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
> > 1.2 0.6% 97.3% 1.4 0.6%
> > ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
> > 1.1 0.5% 97.8% 1.1 0.5% Thread::operator new
> > 0.8 0.4% 98.2% 0.8 0.4% ParkEvent::Allocate
> > 0.6 0.3% 98.5% 0.6 0.3% readCEN
> > 0.6 0.3% 98.7% 0.6 0.3% Arena::grow
> > 0.4 0.2% 98.9% 0.4 0.2% Hashtable::new_entry
> > 0.3 0.2% 99.1% 0.3 0.2% frame::oops_interpreted_do
> > 0.3 0.1% 99.2% 150.8 70.5% JavaCalls::call
> > 0.3 0.1% 99.3% 0.3 0.1% CHeapObj::operator new
> > 0.2 0.1% 99.4% 0.2 0.1% Hashtable::Hashtable
> > 0.2 0.1% 99.5% 0.2 0.1% JavaThread::initialize
> > 0.1 0.1% 99.6% 0.2 0.1%
> > Deoptimization::fetch_unroll_info_helper
> > 0.1 0.1% 99.6% 0.1 0.1% os::create_thread
> > 0.1 0.1% 99.7% 0.1 0.1% _dl_allocate_tls
> > 0.1 0.0% 99.7% 0.1 0.0% addMetaName
> > 0.1 0.0% 99.8% 0.1 0.0% vframeArray::allocate
> > 0.1 0.0% 99.8% 1.8 0.8% ciEnv::register_method
> > 0.1 0.0% 99.9% 1.6 0.7% Thread::Thread
> > 0.1 0.0% 99.9% 0.1 0.0%
> > CompactibleFreeListSpace::CompactibleFreeListSpace
> > 0.0 0.0% 99.9% 0.1 0.0% WorkGang::allocate_worker
*Best Regards*
*===================*
*Yusup Ashrap*
*cell:18611205204*
*--------------------------------------*
*do or don't, the is no try.*
*--------------------------------------*
ramkrishna vasudevan 2012-11-21, 05:33
Actually we once faced a memory leak issue with GZip and our profiler also
showed us the same.
Am not sure of this profiler what it is saying.
Can you try disabling the GZip and try doing it with NO compression. This
will try to figure out which is the actual problem creator.
Regards
Ram
On Wed, Nov 21, 2012 at 10:51 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
> hi Ramkrishna
> 5 tables on this node is using gzip compression.
> At first I assumed this was caused by gzip compression, but as u can see
> from my google-perftools profiling result above that
> there is no gzip related compression memory allocated.(or am I misreading
> profiling resutls?)
>
> thanks,regards
>
>
> On Wed, Nov 21, 2012 at 12:07 PM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Yusup
> >
> > ARe you using Gzip compression for your storefiles by any chance?
> >
> > Regards
> > Ram
> >
> > On Wed, Nov 21, 2012 at 9:15 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
> >
> > > hi all, I am encountering with high memory usage problem in my
> production
> > > environment.I doubt this is caused by memory leak or something,
> > > and I hope someone could tell me what is going on , or what should I do
> > to
> > > to keep a lower memory footprint, or any ways or tools to find out what
> > is
> > > causing so much memory footprint.my cluster have 24 nodes, and here is
> > some
> > > info from my random one node.
> > >
> > > hbase version:0.90.2
> > > ReadRequest AVG：1,716.02
> > > WriteRequest AVG：435.47
> > > Region count: 281
> > >
> > > *top:*
> > >
> > > top - 11:19:40 up 530 days, 12:29, 1 user, load average: 4.10, 3.97,
> > 4.28
> > > Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
> > > Cpu(s): 5.0%us, 1.2%sy, 0.0%ni, 85.3%id, 7.5%wa, 0.0%hi, 1.0%si,
> > > 0.0%st
> > > Mem: 24676836k total, 24599764k used, 77072k free, 30052k
> buffers
> > > Swap: 8385760k total, 20568k used, 8365192k free, 1954280k cached
> > >
> > > 11226 hbase 18 0 23.5g 21g 18m S 75.9 89.5 1219:37 java
> > > (regionserver)
> > > 31579 hbase 19 0 2719m 171m 14m S 35.3 0.7 11502:28 java
> > > (datanode)
> > >
> > >
> > > here is my regionserver configuration.
> > >
> > > export HBASE_REGIONSERVER_OPTS=" -Xms16g -Xmx16g -Xmn2g
> > > -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:CMSInitiatingO
> > > ccupancyFraction=75 -Xloggc:$HBASE_HOME/logs/gc-regionserver-`date
> > > +%Y%m%d-%H-%M`.log"
> > >
> > >
> > > I used google-perftools to do the heap profiling and i got this.
> > >
> > > Total: 213.9 MB
> > > 154.4 72.2% 72.2% 154.4 72.2% os::malloc
> > > 18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
> > > 13.0 6.1% 87.1% 13.0 6.1%
> ParNewGeneration::ParNewGeneration
> > > 9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
> > > 3.0 1.4% 93.0% 3.0 1.4% init
> > > 2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
> > > 2.3 1.1% 95.3% 2.3 1.1% zcalloc
> > > 1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
> > > 1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
> > > 1.2 0.6% 97.3% 1.4 0.6%
> > > ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration
> > > 1.1 0.5% 97.8% 1.1 0.5% Thread::operator new
> > > 0.8 0.4% 98.2% 0.8 0.4% ParkEvent::Allocate
> > > 0.6 0.3% 98.5% 0.6 0.3% readCEN
> > > 0.6 0.3% 98.7% 0.6 0.3% Arena::grow
> > > 0.4 0.2% 98.9% 0.4 0.2% Hashtable::new_entry
> > > 0.3 0.2% 99.1% 0.3 0.2% frame::oops_interpreted_do
> > > 0.3 0.1% 99.2% 150.8 70.5% JavaCalls::call
> > > 0.3 0.1% 99.3% 0.3 0.1% CHeapObj::operator new
> > > 0.2 0.1% 99.4% 0.2 0.1% Hashtable::Hashtable
> > > 0.2 0.1% 99.5% 0.2 0.1% JavaThread::initialize
> > > 0.1 0.1% 99.6% 0.2 0.1%
> > > Deoptimization::fetch_unroll_info_helper
I changed compression from gz to lzo ,now I'm waiting for the results.I
yet i still think there is some way to figure out what is causing this high memory footprint.
thanks , regards
On Wed, Nov 21, 2012 at 1:33 PM, ramkrishna vasudevan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Actually we once faced a memory leak issue with GZip and our profiler also
showed us the same.
Am not sure of this profiler what it is saying.
Can you try disabling the GZip and try doing it with NO compression. This
will try to figure out which is the actual problem creator.
Regards
Ram
________________________________
This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Yusup Ashrap 2012-11-21, 05:52
I changed compression from gz to lzo ,now I'm waiting for the results.
yet i still think there is some way to figure out what is causing this high
memory footprint.
thanks , regards
On Wed, Nov 21, 2012 at 1:33 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:
> Actually we once faced a memory leak issue with GZip and our profiler also
> showed us the same.
> Am not sure of this profiler what it is saying.
>
> Can you try disabling the GZip and try doing it with NO compression. This
> will try to figure out which is the actual problem creator.
>
> Regards
> Ram
>
>
On Tue, Nov 20, 2012 at 7:45 PM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
> hi all, I am encountering with high memory usage problem in my production
> environment.I doubt this is caused by memory leak or something,
> and I hope someone could tell me what is going on , or what should I do to
> to keep a lower memory footprint, or any ways or tools to find out what is
> causing so much memory footprint.my cluster have 24 nodes, and here is some
> info from my random one node.
>
> hbase version:0.90.2
Can you upgrade your hbase? This is an old version. What version of
hadoop are you running?
> I used google-perftools to do the heap profiling and i got this.
>
> Total: 213.9 MB
> 154.4 72.2% 72.2% 154.4 72.2% os::malloc
> 18.8 8.8% 81.0% 20.6 9.7% CMSCollector::CMSCollector
> 13.0 6.1% 87.1% 13.0 6.1% ParNewGeneration::ParNewGeneration
> 9.6 4.5% 91.6% 9.6 4.5% ObjectSynchronizer::omAlloc
> 3.0 1.4% 93.0% 3.0 1.4% init
> 2.8 1.3% 94.3% 2.8 1.3% AllocateHeap
> 2.3 1.1% 95.3% 2.3 1.1% zcalloc
> 1.7 0.8% 96.1% 1.7 0.8% nmethod::nmethod
> 1.3 0.6% 96.7% 1.3 0.6% SymbolTable::basic_add
> 1.2 0.6% 97.3% 1.4 0.6%
...
I've not used this tool before. Its output includes that of the JVM
itself which would tend to occlude any picture of what hbase is doing
inside in the JVM.
Are you OOMEing?
St.Ack
>Can you upgrade your hbase? This is an old version. What version of
>hadoop are you running?
it's in production environment, and we could not take upgrading as an option for now.
hadoop version is 0.20.2
>I've not used this tool before. Its output includes that of the JVM
>itself which would tend to occlude any picture of what hbase is doing
>inside in the JVM.
any perf tools or ways to recommend to detect memory leak?
>Are you OOMEing?
nope,and I cant still see a Full Gc coming from gc log file.and I don't have any particular exceptions in regionserver log file.
thanks , regards
________________________________
This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Yusup Ashrap 2012-11-21, 06:05
>Can you upgrade your hbase? This is an old version. What version of
>hadoop are you running?
it's in production environment, and we could not take upgrading as an
option for now.
hadoop version is 0.20.2
>I've not used this tool before. Its output includes that of the JVM
>itself which would tend to occlude any picture of what hbase is doing
>inside in the JVM.
any perf tools or ways to recommend to detect memory leak?
>Are you OOMEing?
nope,and I cant still see a Full Gc coming from gc log file.and I don't
have any particular exceptions in regionserver log file.
thanks , regards
--
*Best Regards*
*===================*
*Yusup Ashrap*
*cell:18611205204*
*--------------------------------------*
*do or don't, the is no try.*
*--------------------------------------*
On Tue, Nov 20, 2012 at 10:05 PM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
>>Can you upgrade your hbase? This is an old version. What version of
>>hadoop are you running?
>
> it's in production environment, and we could not take upgrading as an
> option for now.
> hadoop version is 0.20.2
>
Does your hadoop support append/sync?
>>I've not used this tool before. Its output includes that of the JVM
>>itself which would tend to occlude any picture of what hbase is doing
>>inside in the JVM.
>
> any perf tools or ways to recommend to detect memory leak?
>
Yeah, there is a whole industry of tools for looking at running apps.
You could try the profiler in eclipse or one of the commercial tools.
>>Are you OOMEing?
> nope,and I cant still see a Full Gc coming from gc log file.and I don't
> have any particular exceptions in regionserver log file.
>
So, what is the worry? You are using CMS? It carries lots of
garbage. If you are really concerned, then maybe dump a heap -- this
can take a while to write if your heap is large and you will freeze
the daemon while the heap is being written (be sure you write the heap
to a place that has space) -- and use one of the aforementioned tools
to inspect it.
Yusup Ashrap 2012-11-21, 08:16
>Does your hadoop support append/sync?
current version of hadoop supports append/sync ,but we have not trun on
explicitely in hdfs-site.xml.
>Yeah, there is a whole industry of tools for looking at running apps.
>You could try the profiler in eclipse or one of the commercial tools.
I will google that.
>So, what is the worry? You are using CMS? It carries lots of
>garbage. If you are really concerned, then maybe dump a heap -- this
>can take a while to write if your heap is large and you will freeze
>the daemon while the heap is being written (be sure you write the heap
>to a place that has space) -- and use one of the aforementioned tools
>to inspect it.
it's kinda delicate to dump heap now, and i tried once and regionserver
process crashed .
dumping heap process ended simultaneously.
Suraj Varma 2012-11-21, 08:26
Have you turned on mslab in your hbase-site.xml?
--Suraj
On Wed, Nov 21, 2012 at 12:16 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
>>Does your hadoop support append/sync?
> current version of hadoop supports append/sync ,but we have not trun on
> explicitely in hdfs-site.xml.
>
>
>>Yeah, there is a whole industry of tools for looking at running apps.
>>You could try the profiler in eclipse or one of the commercial tools.
> I will google that.
>
>
>>So, what is the worry? You are using CMS? It carries lots of
>>garbage. If you are really concerned, then maybe dump a heap -- this
>>can take a while to write if your heap is large and you will freeze
>>the daemon while the heap is being written (be sure you write the heap
>>to a place that has space) -- and use one of the aforementioned tools
>>to inspect it.
>
> it's kinda delicate to dump heap now, and i tried once and regionserver
> process crashed .
> dumping heap process ended simultaneously.
Yusup Ashrap 2012-11-21, 08:48
Hi Suraj
yes , mslab is on.
thanks , regards
On Wed, Nov 21, 2012 at 4:26 PM, Suraj Varma <[EMAIL PROTECTED]> wrote:
> Have you turned on mslab in your hbase-site.xml?
On Wed, Nov 21, 2012 at 12:16 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:
>>Does your hadoop support append/sync?
> current version of hadoop supports append/sync ,but we have not trun on
> explicitely in hdfs-site.xml.
>
Then you will lose data.
>>So, what is the worry? You are using CMS? It carries lots of
>>garbage. If you are really concerned, then maybe dump a heap -- this
>>can take a while to write if your heap is large and you will freeze
>>the daemon while the heap is being written (be sure you write the heap
>>to a place that has space) -- and use one of the aforementioned tools
>>to inspect it.
>
> it's kinda delicate to dump heap now, and i tried once and regionserver
> process crashed .
> dumping heap process ended simultaneously.
Yes. I suppose. If it takes a long time to dump it could hose the
regionserver. Did it make a dump file that you could use?
St.Ack
|
|
