Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC and High CPU


Copy link to this message
-
GC and High CPU
Greetings all -- We had some issues with huge GC counts that drive CPU
to 100% and rendering RS basically dead, here is some data:

 We are normal here, about 1 ParNew per second.

    67 2011-05-14T15:35
     75 2011-05-14T15:36
     70 2011-05-14T15:37
     74 2011-05-14T15:38
     73 2011-05-14T15:39
     73 2011-05-14T15:40
     69 2011-05-14T15:41
     68 2011-05-14T15:42
     76 2011-05-14T15:43
     70 2011-05-14T15:44
     68 2011-05-14T15:45
    322 2011-05-14T15:46
    562 2011-05-14T15:47  <------------ This is where its about 10 per second.
    567 2011-05-14T15:48
    563 2011-05-14T15:49
    562 2011-05-14T15:50
    561 2011-05-14T15:51
    513 2011-05-14T15:52

here is the jstack from the time of this:

http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.

This is config for hbase-env:

export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log"

Does any have any suggestions on how to manage this?

-Jack
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB