Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor Increments


Copy link to this message
-
Re: Coprocessor Increments
Ok…

So…  I see you followed up to your earlier post.

Lets walk through this to make sure we're on the same page.

You put()  row 1 in table 1.
The post_put() wants to insert a value in table 2.

Now suppose I have an htable pool of connections in my Region Observer.
(Is this overkill? I'm treating the RO as if its a client connecting to HBase.)

RO performs a put() into the second table.

The RPC handlers are a Region server resource, yes?

So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests.

Is that the case?
On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <[EMAIL PROTECTED]> wrote:

> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming RPC calls can be accepted because
> everyone is connected to everyone in a spaghetti way and waiting on RPC calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10 handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .    
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Michael Segel [[EMAIL PROTECTED]]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: [EMAIL PROTECTED]
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <[EMAIL PROTECTED]> wrote:
>
>> Michael,
>> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post  (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>>
>>
>>
>> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>>> Can we just take a quick pause…
>>>
>>> John you wrote the following:
>>> "We have been running into an RPC deadlock issue on HBase and from
>>> investigation, we believe the root of the issue is in us doing cross
>>> region increments from a coprocessor. After some further searching and