Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> RS not processing any requests


+
Nathaniel Cook 2012-09-05, 21:58
+
Stack 2012-09-05, 22:39
+
Jeff Whiting 2012-09-05, 22:51
+
Himanshu Vashishtha 2012-09-05, 22:47
+
Jeff Whiting 2012-09-05, 23:18
+
Himanshu Vashishtha 2012-09-05, 23:23
+
Jeff Whiting 2012-09-05, 23:53
Copy link to this message
-
Re: RS not processing any requests
It usually happens in a long running setup (at least for me). Can you
throttle your load?

Replication is evolving; I'd say update if you can (or backport the jiras?).

Himanshu
On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> hmm.  So if we are on 0.92 what suggestion would you have to prevent the
> problem?
>
> ~Jeff
>
>
> On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
>>
>> Number of PRI handlers are governed by
>> "hbase.regionserver.metahandler.count"; default is 10.
>>
>> Increasing their number will not solve it, but will delay its
>> occurring (i don't know about your load etc).
>>
>> Another related jira is HBase-6550.
>>
>> Some more context for your use case:
>>
>> http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements
>>
>>
>> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
>>>
>>> It looks like that is problem we are having.  We are on 0.92 so we don't
>>> get
>>> the patch.  But one solution seems to be increasing the privileged
>>> handlers.
>>> How do we increase the number of privilege handlers?
>>>
>>>
>>> ~Jeff
>>>
>>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote:
>>>>
>>>> Your RS priority handlers are blocked on meta lookup, so it becomes
>>>> unresponsive. Looks like you hitting
>>>> https://issues.apache.org/jira/browse/HBASE-6165
>>>> You running HBase replication? just confirming.
>>>>
>>>> Himanshu
>>>>
>>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook
>>>>> <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> We ran a jstack on the both the RS process and the hbase shell process
>>>>>> trying to do the scan.
>>>>>>
>>>>>> Jstack log for RS:
>>>>>> http://pastebin.com/9Y9t5ERE
>>>>>>
>>>>> What JVM (I don't know what (20.10-b01 mixed mode) is).
>>>>>
>>>>> I see a bunch of this:
>>>>>
>>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry
>>>>> [0x000000004ab0f000]
>>>>>      java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>           at .....
>>>>>
>>>>> But when I go to look for other instances of the object monitor, I
>>>>> don't find any.  I see this for each instance of BLOCKED (Or at least,
>>>>> the two or three I checked).
>>>>>
>>>>> Whats your OS?
>>>>>
>>>>> St.Ack
>>>
>>>
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> [EMAIL PROTECTED]
>>>
>>>
>>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> [EMAIL PROTECTED]
>
>
>
+
Ted Yu 2012-09-06, 00:49
+
Jeff Whiting 2012-09-06, 15:04
+
Jeff Whiting 2012-09-06, 15:04
+
Jeff Whiting 2012-09-05, 22:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB