Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - GC and High CPU


Copy link to this message
-
Re: GC and High CPU
Jack Levin 2011-05-18, 00:09
Well, our issue was that we take an 8 core box, doing 18MB ParNew GC,
do it super fast (10x1sec), receive RPC calls to RS at the same time,
pile them up due to CPU contention, do GC even faster, run out of user
CPU.  This is basically the cycle of cpu death.

It seems that larger RAM boxes doing this:

      4 2011-05-17T16:52
      4 2011-05-17T16:53
      4 2011-05-17T16:54
      4 2011-05-17T16:55

4 times per minute, is just great for me:

[ParNew: 191744K->21248K(191744K), 0.0361360 secs]

-Jack
-jack

On Tue, May 17, 2011 at 3:39 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> My rule of thumb is "as big as possible but no bigger". The young gen size
> determines the pause time and frequency of your young generation
> collections, which are stop-the-world. In my experience the speed is about
> 0.5-1 second per GB on modern hardware.
>
> You get better throughput and less fragmentatoin with a larger young
> genreation, but the pause times go up too.
>
> So, if you have a throughput oriented app, sure, let the JVM use its
> adaptive sizing - otherwise, constrain based on your latency SLAs.
>
> -Todd
>
> On Tue, May 17, 2011 at 7:27 AM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>
>> Isn't a 20mb young generation pretty small these days?  For a 4gb heap, you
>> might want to try giving 5-10% of it to the young generation by setting
>>  -XX:NewSize=256M
>>
>> Some of the older
>> docs<
>> http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#1.1.Sizing%20the%20Generations%7Coutline
>> >have
>> a ratio of 1/3 between young/old, so that would be 1gb young, but that
>> is probably bigger than you need.  I'm far from an expert though... what
>> size do other people use?
>>
>> Matt
>>
>>
>> On Tue, May 17, 2011 at 12:55 AM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>
>> > This is the way I read it. "Low processors" == "high CPU tasks, e.g. high
>> > load".   So, Incremental takes GC down a number of notches when it comes
>> to
>> > competing with CPU for APP threads.  That being the case the deadlock is
>> > less likely.  It would be useful to add code to the RS that will start
>> > blocking any RPC calls should this condition be detected, if we block say
>> > for 10 seconds, GC could finish doing it 'dirty' work and release CPU :)
>> >
>> >
>> > -Jack
>> >
>> >
>> > On May 16, 2011 5:53 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>> > > I don't understand what of the below made a difference though the
>> > > difference is plain from the GC logs you show.
>> > >
>> > > See below:
>> > >
>> > > On Mon, May 16, 2011 at 5:06 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> > >> Those are the lines I added:
>> > >>
>> > >> -XX:+CMSIncrementalMode \  <--------
>> > >
>> > >
>> > > From the doc., it says about i-cms and java6 "This feature is useful
>> > > when applications that need the low pause times provided by the
>> > > concurrent collector are run on machines with small numbers of
>> > > processors (e.g., 1 or 2)." [See
>> > >
>> >
>> http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms
>> > ]
>> > > Don't you have > 2 processors per machine?
>> > >
>> > > St.Ack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>