Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC recommendations for large Region Server heaps


Copy link to this message
-
Re: GC recommendations for large Region Server heaps
Hi Stack:
Yes, we do hit full GC in the 8G heap. I see ... so, you are saying that it
would just be linearly proportional ... so, I should expect a 3x pause
increase with 24G _if_ the full GC hits. I agree with you ... with much
more head room (and mslab etc enabled by default) we shouldn't normally hit
a full GC.

I can surely try this out on a test cluster ... thanks for the pointer on
the 3x increase if it hits full gc. That quantifies it for me much better.
Thanks
--Suraj

On Mon, Jul 8, 2013 at 11:56 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Jul 8, 2013 at 11:09 AM, Suraj Varma <[EMAIL PROTECTED]> wrote:
>
> > Hello:
> > We have an HBase cluster with region servers running on 8GB heap size
> with
> > a 0.6 block cache (it is a read heavy cluster, with bursty write traffic
> > via MR jobs). (version: hbase-0.94.6.1)
> >
> > During HBaseCon, while speaking to a few attendees, I heard some folks
> were
> > running region servers as high as 24GB and some others in the 16GB range.
> >
> > So - question: Are there any special GC recommendations (tuning
> parameters,
> > flags, etc) that folks who run at these large heaps can recommend while
> > moving up from an 8GB heap? i.e. for 16GB and for 24GB RS heaps ... ?
> >
> > I'm especially concerned about long pauses causing zk session timeouts
> and
> > consequent RS shutdowns. Our boxes do have a lot of RAM and we are
> > exploring how we can use more of it for the cluster while maintaining
> > overall stability.
> >
> > Also - if there are clusters running multiple region servers per host,
> I'd
> > be very interested to know what RS heap sizes those are being run at ...
> > and whether this was chosen as an alternative to running a single RS with
> > large heap.
> >
> > (I know I'll have to test the GC stuff out on my cluster and for my
> > workloads anyway ... but just trying to get a feel of what sort of tuning
> > options had to be used to have a stable HBase cluster with 16 or 24GB RS
> > heaps).
> >
>
>
> You hit full GC in this 8G heap Suraj?  Can you try running one server at
> 24G to see how it does (with GC logging enabled so you can watch it over
> time)?  On one hand, more heap may make it so you avoid full GC -- if you
> are hitting them now at 8G -- because application has more head room.  On
> other hand, yes, if a full GC hits, it will be gone for proportionally
> longer than for your 8G heap.
>
> St.Ack
>