Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Multiple regionservers on a single node


Copy link to this message
-
答复: 答复: Multiple regionservers on a single node
I am just a hbase&hotspot vm newbie:)

1)Before look into GC detail, we should turn ontracing flags, e.g. -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:xxxx -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintPromotionFailure ...

2)dive into GC log during each run, figure out the longest STW root cause, statistic GC total time/GC total count, etc. Here're some usual safepoint cause:  GC 、Revoke Biasedlock 、Deoptimize、FindDeadlocks、PrintJNI, etc..

3)If ParNew costs too much, we can reduce Xmn, adjust survivorRatio/TargetSurvivorRatio/PretenureSizeThreshold...

4)If CMS initial mak&remark are expesive, please notice : UseCMSCompactAtFullCollection/CMSInitiatingOccupancyFraction + UseCMSInitiatingOccupancyOnly/CMSParallelRemarkEnabled/CMSClassUnloadingEnabled/CMSMaxAbortablePrecleanTime/CMSWaitDuration/CMSScavengeBeforeRemark/

5)Multi-thread concurrent is a key as well, if running on modern hareware, e.g: CMSConcurrentMTEnabled/ParallelGCThreads/ConcGCThreads/...

at last, RTFC of right hotspot vm or ask help from hotspot-gc mail list should be the best choice for GC issue

Help it helpful for you,
Liang
________________________________________
发件人: Adrien Mogenet [[EMAIL PROTECTED]]
发送时间: 2012年12月11日 6:21
收件人: [EMAIL PROTECTED]
主题: Re: 答复: Multiple regionservers on a single node

On Fri, Dec 7, 2012 at 10:58 AM, 谢良 <[EMAIL PROTECTED]> wrote:

> Emm, have you tried to tune your GC deeply? please provide the exactly VM
> options and jdk version and GC logs..
> In our test cluster this week, i managed to reduce the longest STW from
> 22+ seconds(Xmx20G) to 1.1s(Xmx48G) under a very heavy YCSB stress
> long-term-testing.
>

Do you have any further explanation on your specific case ? Looks
interesting :-)
>
> Also it would be better to ask help from hotspot-gc-use/hotspot-gc-dev
> mail list:)
> And the G1GC within jdk7u4+ is a potential solution for large-heap senario
> as well:)
> ________________________________________
> > On Mon, Dec 3, 2012 at 3:39 PM, Ishan Chhabra <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > > Has anybody tried to run multiple RegionServers on a single physical
> > > node? Are there deep technical issues or minor impediments that would
> > > hinder this?
> > >
> > > We are trying to do this because we are facing a lot of GC pauses on
> the
> > > large heap sizes (~70G) that we are using, which leads to a lot of
> > timeouts
> > > in our latency critical application. More processes with smaller heaps
> > > would help in mitigating this issue.
> > >
> > > Any experience or thoughts on this would help.
> > > Thanks!
> > >
> > > --
> > > *Ishan Chhabra *| Rocket Scientist | Rocketfuel Inc. | *m *650 556
> 6803
> > >
> >
> >
> >
> > --
> >
> > Robert Dyer
> > [EMAIL PROTECTED]
> >
>

--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB