Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC and High CPU


Copy link to this message
-
Re: GC and High CPU
Somewhere like imageshack :)
http://img221.imageshack.us/img221/761/screenshot20110515at537p.png

-Jack

On Sun, May 15, 2011 at 7:09 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Jack:
> Your attachment got filtered.
> Can you upload somewhere ?
>
> Thanks
>
> On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>> But we had same issue.  Whats strange is that the load on the RS would
>> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
>> and kills all of the user CPU.  I have another jstack:
>> http://pastebin.com/4n3MvmrX from a different server.   How do people
>> deal with this?  We have HEAP of 12G, block cace at 5G, and upper
>> memstore at 1G.  Is hlog roll that intensive?  I though 1000 rpc calls
>> per RS should be easy?  The server is not at all busy pre-spike, you
>> can tell by attached graph.
>>
>> -Jack
>>
>> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> > Greetings all -- We had some issues with huge GC counts that drive CPU
>> > to 100% and rendering RS basically dead, here is some data:
>> >
>> >  We are normal here, about 1 ParNew per second.
>> >
>> >    67 2011-05-14T15:35
>> >     75 2011-05-14T15:36
>> >     70 2011-05-14T15:37
>> >     74 2011-05-14T15:38
>> >     73 2011-05-14T15:39
>> >     73 2011-05-14T15:40
>> >     69 2011-05-14T15:41
>> >     68 2011-05-14T15:42
>> >     76 2011-05-14T15:43
>> >     70 2011-05-14T15:44
>> >     68 2011-05-14T15:45
>> >    322 2011-05-14T15:46
>> >    562 2011-05-14T15:47  <------------ This is where its about 10 per
>> second.
>> >    567 2011-05-14T15:48
>> >    563 2011-05-14T15:49
>> >    562 2011-05-14T15:50
>> >    561 2011-05-14T15:51
>> >    513 2011-05-14T15:52
>> >
>> > here is the jstack from the time of this:
>> >
>> > http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.
>> >
>> > This is config for hbase-env:
>> >
>> > export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
>> > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> > -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>> >
>> > Does any have any suggestions on how to manage this?
>> >
>> > -Jack
>> >
>>
>