Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> High cpu usage on a region server


+
OpenSource Dev 2013-09-11, 20:49
+
Stack 2013-09-11, 21:28
+
OpenSource Dev 2013-09-12, 15:15
+
lars hofhansl 2013-09-12, 15:36
+
OpenSource Dev 2013-09-12, 16:52
+
lars hofhansl 2013-09-12, 17:25
+
Jean-Daniel Cryans 2013-09-12, 17:33
+
OpenSource Dev 2013-09-15, 06:21
+
lars hofhansl 2013-09-15, 06:30
+
OpenSource Dev 2013-09-12, 02:11
+
Ted Yu 2013-09-11, 20:57
+
OpenSource Dev 2013-09-11, 21:02
+
lars hofhansl 2013-09-11, 20:55
+
OpenSource Dev 2013-09-12, 03:59
Copy link to this message
-
Re: High cpu usage on a region server
It might be a larger scan (maybe gathering many data points for a metric) hitting many regions, in that case you'd see only a single region server being busy at a given time, since HBase scans only a region at a time for a single client scan.
A thread dump would give us a better idea. J-D specifically mentions OpenTSDB in that jira.
-- Lars

________________________________
 From: OpenSource Dev <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Wednesday, September 11, 2013 8:59 PM
Subject: Re: High cpu usage on a region server
 

Hi Lars,

All the read & write requests are equally distributed across all region-servers.

If it is caused by the HBASE-9428 bug, any idea why it would impact
only 1 reason server at a given time ?

Thx
On Wed, Sep 11, 2013 at 1:55 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> You might have run into HBASE-9428
>
> -- Lars
>
>
>
> ________________________________
>  From: OpenSource Dev <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, September 11, 2013 1:49 PM
> Subject: High cpu usage on a region server
>
>
> Hi,
>
> I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no
> issues with writes/puts. System is handles upto 800k puts per seconds
> without issue. On average we do 250k puts per second.
>
> I am having the problem with Reads, I've also isolated where the
> problem is but not been able to find the root cause.
>
> I have 16 machines running hbase-region server, each has ~35 regions.
> Once in a while cpu goes flatout 80% in 1 region server. These are the
> things i've noticed in ganglia:
>
> hbase.regionserver.request - evenly distributed. Not seeing any spikes
> on the busy server
> hbase.regionserver.blockCacheSize - between 500MB and 1000MB
> hbase.regionserver.compactionQueueSize - avg 2 or less
> hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other nodes
>
>
> JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
>
> I've noticed the system load moves to a different region, sometimes
> within a minute, if the busy region is restarted.
>
> Any suggestion what could be causing the load and/or what other
> metrics should I check ?
>
>
> Thank you!
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB