Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Region Server Spinning on JMX requests


+
Buckley,Ron 2013-04-23, 13:57
+
lars hofhansl 2013-04-23, 22:57
+
Kevin Odell 2013-04-24, 03:04
Copy link to this message
-
RE: HBase Region Server Spinning on JMX requests
I created https://issues.apache.org/jira/browse/HBASE-8416

We're not using OpenTSDB, but we do have something similar grabbing the
jmx data on a regular basis.

Eventually, we moved all the regions off of that region server.  We left
it spinning overnight, going to try to look at it this morning.
-----Original Message-----
From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 23, 2013 11:04 PM
To: [EMAIL PROTECTED]; lars hofhansl
Subject: Re: HBase Region Server Spinning on JMX requests

Hi Ron,

  Are you using OpenTSDB?  I have seen:

https://issues.apache.org/jira/browse/HBASE-6602 (which should be
addressed in your build).  One possibility is that the Tcollector is
leaving lots of connections open and causing the spin.  Unfortunately,
we have not been able to nail it down further.  We are thinking
Metrics2 in trunk might inadvertently take care of this issue.

On Tue, Apr 23, 2013 at 6:57 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Hmm... That's not good. Would you mind filing a ticket here:
https://issues.apache.org/jira/browse/HBASE ?
>
> -- Lars
>
>
> ________________________________
>  From: "Buckley,Ron" <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Tuesday, April 23, 2013 6:57 AM
> Subject: HBase Region Server Spinning on JMX requests
>
>
> This is with HBase 0.94.4 & CDH 4.1.1
>
> This morning one our region servers (we have 44) stopped responding to
> the '/jmx' request. (It's working for regular activity.)
Additionally,
> the region server is now using all the CPU on the host, running all 8
> cores at 100%.
>
> I've got several jstacks, they all look like this:
> http://pastebin.com/dGTmTEN7
>
> If I do a wget of the /jmx url, it starts responding, but never
> completes, always stopping at the same point:
> http://pastebin.com/qhNvxrQK
>
> Has anyone ever seen this before? If so, Is there a way out of it?
> (other than bouncing the region server).
>
> BTW: There's nothing relevant in the region server log and the garbage
> collector log is normal.
>
>
> ----------------------------------------------------------------------
> Ron Buckley

--
Kevin O'Dell
Systems Engineer, Cloudera
+
Andrew Purtell 2013-04-24, 16:31
+
Elliott Clark 2013-04-24, 18:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB