Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - GC and High CPU


Copy link to this message
-
Re: GC and High CPU
Jack Levin 2011-05-16, 00:37
I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log"

But we had same issue.  Whats strange is that the load on the RS would
be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
and kills all of the user CPU.  I have another jstack:
http://pastebin.com/4n3MvmrX from a different server.   How do people
deal with this?  We have HEAP of 12G, block cace at 5G, and upper
memstore at 1G.  Is hlog roll that intensive?  I though 1000 rpc calls
per RS should be easy?  The server is not at all busy pre-spike, you
can tell by attached graph.

-Jack

On Sat, May 14, 2011 at 7:12 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
> Greetings all -- We had some issues with huge GC counts that drive CPU
> to 100% and rendering RS basically dead, here is some data:
>
>  We are normal here, about 1 ParNew per second.
>
>    67 2011-05-14T15:35
>     75 2011-05-14T15:36
>     70 2011-05-14T15:37
>     74 2011-05-14T15:38
>     73 2011-05-14T15:39
>     73 2011-05-14T15:40
>     69 2011-05-14T15:41
>     68 2011-05-14T15:42
>     76 2011-05-14T15:43
>     70 2011-05-14T15:44
>     68 2011-05-14T15:45
>    322 2011-05-14T15:46
>    562 2011-05-14T15:47  <------------ This is where its about 10 per second.
>    567 2011-05-14T15:48
>    563 2011-05-14T15:49
>    562 2011-05-14T15:50
>    561 2011-05-14T15:51
>    513 2011-05-14T15:52
>
> here is the jstack from the time of this:
>
> http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.
>
> This is config for hbase-env:
>
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>
> Does any have any suggestions on how to manage this?
>
> -Jack
>