Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> High cpu usage on a region server


Copy link to this message
-
Re: High cpu usage on a region server
You might have run into HBASE-9428

-- Lars

________________________________
 From: OpenSource Dev <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, September 11, 2013 1:49 PM
Subject: High cpu usage on a region server
 

Hi,

I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no
issues with writes/puts. System is handles upto 800k puts per seconds
without issue. On average we do 250k puts per second.

I am having the problem with Reads, I've also isolated where the
problem is but not been able to find the root cause.

I have 16 machines running hbase-region server, each has ~35 regions.
Once in a while cpu goes flatout 80% in 1 region server. These are the
things i've noticed in ganglia:

hbase.regionserver.request - evenly distributed. Not seeing any spikes
on the busy server
hbase.regionserver.blockCacheSize - between 500MB and 1000MB
hbase.regionserver.compactionQueueSize - avg 2 or less
hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other nodes
JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC

I've noticed the system load moves to a different region, sometimes
within a minute, if the busy region is restarted.

Any suggestion what could be causing the load and/or what other
metrics should I check ?
Thank you!
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB