-答复: 答复: Multiple regionservers on a single node
谢良 2012-12-11, 08:21
I am just a hbase&hotspot vm newbie:)
1)Before look into GC detail, we should turn ontracing flags, e.g. -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:xxxx -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintPromotionFailure ...
2)dive into GC log during each run, figure out the longest STW root cause, statistic GC total time/GC total count, etc. Here're some usual safepoint cause: GC 、Revoke Biasedlock 、Deoptimize、FindDeadlocks、PrintJNI, etc..
3)If ParNew costs too much, we can reduce Xmn, adjust survivorRatio/TargetSurvivorRatio/PretenureSizeThreshold...
4)If CMS initial mak&remark are expesive, please notice : UseCMSCompactAtFullCollection/CMSInitiatingOccupancyFraction + UseCMSInitiatingOccupancyOnly/CMSParallelRemarkEnabled/CMSClassUnloadingEnabled/CMSMaxAbortablePrecleanTime/CMSWaitDuration/CMSScavengeBeforeRemark/
5)Multi-thread concurrent is a key as well, if running on modern hareware, e.g: CMSConcurrentMTEnabled/ParallelGCThreads/ConcGCThreads/...
at last, RTFC of right hotspot vm or ask help from hotspot-gc mail list should be the best choice for GC issue
Help it helpful for you,
发件人: Adrien Mogenet [[EMAIL PROTECTED]]
发送时间: 2012年12月11日 6:21
收件人: [EMAIL PROTECTED]
主题: Re: 答复: Multiple regionservers on a single node
On Fri, Dec 7, 2012 at 10:58 AM, 谢良 <[EMAIL PROTECTED]> wrote:
> Emm, have you tried to tune your GC deeply? please provide the exactly VM
> options and jdk version and GC logs..
> In our test cluster this week, i managed to reduce the longest STW from
> 22+ seconds(Xmx20G) to 1.1s(Xmx48G) under a very heavy YCSB stress
Do you have any further explanation on your specific case ? Looks
> Also it would be better to ask help from hotspot-gc-use/hotspot-gc-dev
> mail list:)
> And the G1GC within jdk7u4+ is a potential solution for large-heap senario
> as well:)
> > On Mon, Dec 3, 2012 at 3:39 PM, Ishan Chhabra <[EMAIL PROTECTED]
> > >wrote:
> > > Hi,
> > > Has anybody tried to run multiple RegionServers on a single physical
> > > node? Are there deep technical issues or minor impediments that would
> > > hinder this?
> > >
> > > We are trying to do this because we are facing a lot of GC pauses on
> > > large heap sizes (~70G) that we are using, which leads to a lot of
> > timeouts
> > > in our latency critical application. More processes with smaller heaps
> > > would help in mitigating this issue.
> > >
> > > Any experience or thoughts on this would help.
> > > Thanks!
> > >
> > > --
> > > *Ishan Chhabra *| Rocket Scientist | Rocketfuel Inc. | *m *650 556
> > >
> > --
> > Robert Dyer
> > [EMAIL PROTECTED]