Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RS not processing any requests


Copy link to this message
-
Re: RS not processing any requests
It usually happens in a long running setup (at least for me). Can you
throttle your load?

Replication is evolving; I'd say update if you can (or backport the jiras?).

Himanshu
On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> hmm.  So if we are on 0.92 what suggestion would you have to prevent the
> problem?
>
> ~Jeff
>
>
> On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
>>
>> Number of PRI handlers are governed by
>> "hbase.regionserver.metahandler.count"; default is 10.
>>
>> Increasing their number will not solve it, but will delay its
>> occurring (i don't know about your load etc).
>>
>> Another related jira is HBase-6550.
>>
>> Some more context for your use case:
>>
>> http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements
>>
>>
>> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
>>>
>>> It looks like that is problem we are having.  We are on 0.92 so we don't
>>> get
>>> the patch.  But one solution seems to be increasing the privileged
>>> handlers.
>>> How do we increase the number of privilege handlers?
>>>
>>>
>>> ~Jeff
>>>
>>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote:
>>>>
>>>> Your RS priority handlers are blocked on meta lookup, so it becomes
>>>> unresponsive. Looks like you hitting
>>>> https://issues.apache.org/jira/browse/HBASE-6165
>>>> You running HBase replication? just confirming.
>>>>
>>>> Himanshu
>>>>
>>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook
>>>>> <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> We ran a jstack on the both the RS process and the hbase shell process
>>>>>> trying to do the scan.
>>>>>>
>>>>>> Jstack log for RS:
>>>>>> http://pastebin.com/9Y9t5ERE
>>>>>>
>>>>> What JVM (I don't know what (20.10-b01 mixed mode) is).
>>>>>
>>>>> I see a bunch of this:
>>>>>
>>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry
>>>>> [0x000000004ab0f000]
>>>>>      java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>           at .....
>>>>>
>>>>> But when I go to look for other instances of the object monitor, I
>>>>> don't find any.  I see this for each instance of BLOCKED (Or at least,
>>>>> the two or three I checked).
>>>>>
>>>>> Whats your OS?
>>>>>
>>>>> St.Ack
>>>
>>>
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> [EMAIL PROTECTED]
>>>
>>>
>>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> [EMAIL PROTECTED]
>
>
>