Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC and High CPU


Copy link to this message
-
Re: GC and High CPU
There HEAP would be 8G used out of 12G total.  The gc-log would be
full of ParNew, no FULL GC at all, ParNew just starts at hight rate
(10-15 lines per second).  Even if you remove queries, the JVM will
not recover, and user CPU will remain 100%.  Only thing left to do is
to kill the process.  We are not writing in nearly as fast to case
such an issue.  This looks like a systemic problem, e.g. low enough
load when it comes to QPS, and the problem effectively kills HBASE.
There should be a way to tune things so that ParNew does not jump to
an inappropriate levels.

-Jack

On Mon, May 16, 2011 at 12:02 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>
> Does the big CPU usage correlate to a Full GC?  Do you  notice that in
> your logs Jack?
>
> I took a look at your thread dumps.  Nothing untoward (blocking on hdfs access).
>
> St.Ack
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB