Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - GC and High CPU


Copy link to this message
-
Re: GC and High CPU
Jack Levin 2011-05-16, 19:39
There HEAP would be 8G used out of 12G total.  The gc-log would be
full of ParNew, no FULL GC at all, ParNew just starts at hight rate
(10-15 lines per second).  Even if you remove queries, the JVM will
not recover, and user CPU will remain 100%.  Only thing left to do is
to kill the process.  We are not writing in nearly as fast to case
such an issue.  This looks like a systemic problem, e.g. low enough
load when it comes to QPS, and the problem effectively kills HBASE.
There should be a way to tune things so that ParNew does not jump to
an inappropriate levels.

-Jack

On Mon, May 16, 2011 at 12:02 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>
> Does the big CPU usage correlate to a Full GC?  Do you  notice that in
> your logs Jack?
>
> I took a look at your thread dumps.  Nothing untoward (blocking on hdfs access).
>
> St.Ack
>