Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - High cpu usage on a region server


+
OpenSource Dev 2013-09-11, 20:49
+
Stack 2013-09-11, 21:28
+
OpenSource Dev 2013-09-12, 15:15
+
lars hofhansl 2013-09-12, 15:36
+
OpenSource Dev 2013-09-12, 16:52
+
lars hofhansl 2013-09-12, 17:25
+
Jean-Daniel Cryans 2013-09-12, 17:33
+
OpenSource Dev 2013-09-15, 06:21
+
lars hofhansl 2013-09-15, 06:30
+
OpenSource Dev 2013-09-12, 02:11
+
Ted Yu 2013-09-11, 20:57
+
OpenSource Dev 2013-09-11, 21:02
+
lars hofhansl 2013-09-11, 20:55
+
OpenSource Dev 2013-09-12, 03:59
Copy link to this message
-
Re: High cpu usage on a region server
lars hofhansl 2013-09-12, 04:04
It might be a larger scan (maybe gathering many data points for a metric) hitting many regions, in that case you'd see only a single region server being busy at a given time, since HBase scans only a region at a time for a single client scan.
A thread dump would give us a better idea. J-D specifically mentions OpenTSDB in that jira.
-- Lars

________________________________
 From: OpenSource Dev <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Wednesday, September 11, 2013 8:59 PM
Subject: Re: High cpu usage on a region server
 

Hi Lars,

All the read & write requests are equally distributed across all region-servers.

If it is caused by the HBASE-9428 bug, any idea why it would impact
only 1 reason server at a given time ?

Thx
On Wed, Sep 11, 2013 at 1:55 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> You might have run into HBASE-9428
>
> -- Lars
>
>
>
> ________________________________
>  From: OpenSource Dev <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, September 11, 2013 1:49 PM
> Subject: High cpu usage on a region server
>
>
> Hi,
>
> I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no
> issues with writes/puts. System is handles upto 800k puts per seconds
> without issue. On average we do 250k puts per second.
>
> I am having the problem with Reads, I've also isolated where the
> problem is but not been able to find the root cause.
>
> I have 16 machines running hbase-region server, each has ~35 regions.
> Once in a while cpu goes flatout 80% in 1 region server. These are the
> things i've noticed in ganglia:
>
> hbase.regionserver.request - evenly distributed. Not seeing any spikes
> on the busy server
> hbase.regionserver.blockCacheSize - between 500MB and 1000MB
> hbase.regionserver.compactionQueueSize - avg 2 or less
> hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other nodes
>
>
> JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
>
> I've noticed the system load moves to a different region, sometimes
> within a minute, if the busy region is restarted.
>
> Any suggestion what could be causing the load and/or what other
> metrics should I check ?
>
>
> Thank you!