Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC and High CPU


Copy link to this message
-
Re: GC and High CPU
Somewhere like imageshack :)
http://img221.imageshack.us/img221/761/screenshot20110515at537p.png

-Jack

On Sun, May 15, 2011 at 7:09 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Jack:
> Your attachment got filtered.
> Can you upload somewhere ?
>
> Thanks
>
> On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>> But we had same issue.  Whats strange is that the load on the RS would
>> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
>> and kills all of the user CPU.  I have another jstack:
>> http://pastebin.com/4n3MvmrX from a different server.   How do people
>> deal with this?  We have HEAP of 12G, block cace at 5G, and upper
>> memstore at 1G.  Is hlog roll that intensive?  I though 1000 rpc calls
>> per RS should be easy?  The server is not at all busy pre-spike, you
>> can tell by attached graph.
>>
>> -Jack
>>
>> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> > Greetings all -- We had some issues with huge GC counts that drive CPU
>> > to 100% and rendering RS basically dead, here is some data:
>> >
>> >  We are normal here, about 1 ParNew per second.
>> >
>> >    67 2011-05-14T15:35
>> >     75 2011-05-14T15:36
>> >     70 2011-05-14T15:37
>> >     74 2011-05-14T15:38
>> >     73 2011-05-14T15:39
>> >     73 2011-05-14T15:40
>> >     69 2011-05-14T15:41
>> >     68 2011-05-14T15:42
>> >     76 2011-05-14T15:43
>> >     70 2011-05-14T15:44
>> >     68 2011-05-14T15:45
>> >    322 2011-05-14T15:46
>> >    562 2011-05-14T15:47  <------------ This is where its about 10 per
>> second.
>> >    567 2011-05-14T15:48
>> >    563 2011-05-14T15:49
>> >    562 2011-05-14T15:50
>> >    561 2011-05-14T15:51
>> >    513 2011-05-14T15:52
>> >
>> > here is the jstack from the time of this:
>> >
>> > http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.
>> >
>> > This is config for hbase-env:
>> >
>> > export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
>> > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> > -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>> >
>> > Does any have any suggestions on how to manage this?
>> >
>> > -Jack
>> >
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB