Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Multiple regionservers on a single node


Copy link to this message
-
答复: 答复: Multiple regionservers on a single node
I am just a hbase&hotspot vm newbie:)

1)Before look into GC detail, we should turn ontracing flags, e.g. -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:xxxx -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintPromotionFailure ...

2)dive into GC log during each run, figure out the longest STW root cause, statistic GC total time/GC total count, etc. Here're some usual safepoint cause:  GC 、Revoke Biasedlock 、Deoptimize、FindDeadlocks、PrintJNI, etc..

3)If ParNew costs too much, we can reduce Xmn, adjust survivorRatio/TargetSurvivorRatio/PretenureSizeThreshold...

4)If CMS initial mak&remark are expesive, please notice : UseCMSCompactAtFullCollection/CMSInitiatingOccupancyFraction + UseCMSInitiatingOccupancyOnly/CMSParallelRemarkEnabled/CMSClassUnloadingEnabled/CMSMaxAbortablePrecleanTime/CMSWaitDuration/CMSScavengeBeforeRemark/

5)Multi-thread concurrent is a key as well, if running on modern hareware, e.g: CMSConcurrentMTEnabled/ParallelGCThreads/ConcGCThreads/...

at last, RTFC of right hotspot vm or ask help from hotspot-gc mail list should be the best choice for GC issue

Help it helpful for you,
Liang
________________________________________
发件人: Adrien Mogenet [[EMAIL PROTECTED]]
发送时间: 2012年12月11日 6:21
收件人: [EMAIL PROTECTED]
主题: Re: 答复: Multiple regionservers on a single node

On Fri, Dec 7, 2012 at 10:58 AM, 谢良 <[EMAIL PROTECTED]> wrote:

> Emm, have you tried to tune your GC deeply? please provide the exactly VM
> options and jdk version and GC logs..
> In our test cluster this week, i managed to reduce the longest STW from
> 22+ seconds(Xmx20G) to 1.1s(Xmx48G) under a very heavy YCSB stress
> long-term-testing.
>

Do you have any further explanation on your specific case ? Looks
interesting :-)
>
> Also it would be better to ask help from hotspot-gc-use/hotspot-gc-dev
> mail list:)
> And the G1GC within jdk7u4+ is a potential solution for large-heap senario
> as well:)
> ________________________________________
> > On Mon, Dec 3, 2012 at 3:39 PM, Ishan Chhabra <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > > Has anybody tried to run multiple RegionServers on a single physical
> > > node? Are there deep technical issues or minor impediments that would
> > > hinder this?
> > >
> > > We are trying to do this because we are facing a lot of GC pauses on
> the
> > > large heap sizes (~70G) that we are using, which leads to a lot of
> > timeouts
> > > in our latency critical application. More processes with smaller heaps
> > > would help in mitigating this issue.
> > >
> > > Any experience or thoughts on this would help.
> > > Thanks!
> > >
> > > --
> > > *Ishan Chhabra *| Rocket Scientist | Rocketfuel Inc. | *m *650 556
> 6803
> > >
> >
> >
> >
> > --
> >
> > Robert Dyer
> > [EMAIL PROTECTED]
> >
>

--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me