Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Region Server Spinning on JMX requests


Copy link to this message
-
Re: HBase Region Server Spinning on JMX requests
Andrew Purtell 2013-04-24, 16:31
This is probably unprotected concurrent access to a HashMap in Hadoop
metrics. See comments on https://issues.apache.org/jira/browse/HBASE-8416
On Wed, Apr 24, 2013 at 4:37 AM, Buckley,Ron <[EMAIL PROTECTED]> wrote:

> I created https://issues.apache.org/jira/browse/HBASE-8416
>
> We're not using OpenTSDB, but we do have something similar grabbing the
> jmx data on a regular basis.
>
> Eventually, we moved all the regions off of that region server.  We left
> it spinning overnight, going to try to look at it this morning.
>
>
> -----Original Message-----
> From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 23, 2013 11:04 PM
> To: [EMAIL PROTECTED]; lars hofhansl
> Subject: Re: HBase Region Server Spinning on JMX requests
>
> Hi Ron,
>
>   Are you using OpenTSDB?  I have seen:
>
> https://issues.apache.org/jira/browse/HBASE-6602 (which should be
> addressed in your build).  One possibility is that the Tcollector is
> leaving lots of connections open and causing the spin.  Unfortunately,
> we have not been able to nail it down further.  We are thinking
> Metrics2 in trunk might inadvertently take care of this issue.
>
> On Tue, Apr 23, 2013 at 6:57 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> > Hmm... That's not good. Would you mind filing a ticket here:
> https://issues.apache.org/jira/browse/HBASE ?
> >
> > -- Lars
> >
> >
> > ________________________________
> >  From: "Buckley,Ron" <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Tuesday, April 23, 2013 6:57 AM
> > Subject: HBase Region Server Spinning on JMX requests
> >
> >
> > This is with HBase 0.94.4 & CDH 4.1.1
> >
> > This morning one our region servers (we have 44) stopped responding to
> > the '/jmx' request. (It's working for regular activity.)
> Additionally,
> > the region server is now using all the CPU on the host, running all 8
> > cores at 100%.
> >
> > I've got several jstacks, they all look like this:
> > http://pastebin.com/dGTmTEN7
> >
> > If I do a wget of the /jmx url, it starts responding, but never
> > completes, always stopping at the same point:
> > http://pastebin.com/qhNvxrQK
> >
> > Has anyone ever seen this before? If so, Is there a way out of it?
> > (other than bouncing the region server).
> >
> > BTW: There's nothing relevant in the region server log and the garbage
> > collector log is normal.
> >
> >
> > ----------------------------------------------------------------------
> > Ron Buckley
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
>
>
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)