Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RS not processing any requests


Copy link to this message
-
Re: RS not processing any requests
Backport has been done in HBASE-6724

Cheers

On Wed, Sep 5, 2012 at 5:09 PM, Himanshu Vashishtha <[EMAIL PROTECTED]
> wrote:

> It usually happens in a long running setup (at least for me). Can you
> throttle your load?
>
> Replication is evolving; I'd say update if you can (or backport the
> jiras?).
>
> Himanshu
>
>
> On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> > hmm.  So if we are on 0.92 what suggestion would you have to prevent the
> > problem?
> >
> > ~Jeff
> >
> >
> > On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
> >>
> >> Number of PRI handlers are governed by
> >> "hbase.regionserver.metahandler.count"; default is 10.
> >>
> >> Increasing their number will not solve it, but will delay its
> >> occurring (i don't know about your load etc).
> >>
> >> Another related jira is HBase-6550.
> >>
> >> Some more context for your use case:
> >>
> >>
> http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements
> >>
> >>
> >> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> It looks like that is problem we are having.  We are on 0.92 so we
> don't
> >>> get
> >>> the patch.  But one solution seems to be increasing the privileged
> >>> handlers.
> >>> How do we increase the number of privilege handlers?
> >>>
> >>>
> >>> ~Jeff
> >>>
> >>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote:
> >>>>
> >>>> Your RS priority handlers are blocked on meta lookup, so it becomes
> >>>> unresponsive. Looks like you hitting
> >>>> https://issues.apache.org/jira/browse/HBASE-6165
> >>>> You running HBase replication? just confirming.
> >>>>
> >>>> Himanshu
> >>>>
> >>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>>>>
> >>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook
> >>>>> <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>
> >>>>>> We ran a jstack on the both the RS process and the hbase shell
> process
> >>>>>> trying to do the scan.
> >>>>>>
> >>>>>> Jstack log for RS:
> >>>>>> http://pastebin.com/9Y9t5ERE
> >>>>>>
> >>>>> What JVM (I don't know what (20.10-b01 mixed mode) is).
> >>>>>
> >>>>> I see a bunch of this:
> >>>>>
> >>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
> >>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry
> >>>>> [0x000000004ab0f000]
> >>>>>      java.lang.Thread.State: BLOCKED (on object monitor)
> >>>>>           at .....
> >>>>>
> >>>>> But when I go to look for other instances of the object monitor, I
> >>>>> don't find any.  I see this for each instance of BLOCKED (Or at
> least,
> >>>>> the two or three I checked).
> >>>>>
> >>>>> Whats your OS?
> >>>>>
> >>>>> St.Ack
> >>>
> >>>
> >>> --
> >>> Jeff Whiting
> >>> Qualtrics Senior Software Engineer
> >>> [EMAIL PROTECTED]
> >>>
> >>>
> >>>
> >
> > --
> > Jeff Whiting
> > Qualtrics Senior Software Engineer
> > [EMAIL PROTECTED]
> >
> >
> >
>