Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Coprocessor Increments


Copy link to this message
-
Re: Coprocessor Increments
Tom Brown 2013-10-10, 20:20
We used to do our updates through coprocessors, but it was not scalable. We
extracted the update code into a separate system, added row transaction
IDs, and haven't looked back.

For each incoming message, we compute the set of updates that message will
generate. With a batch of messages, we merge updates to the same row
together (for example, 3 increments of 1 each becomes 1 increment of 3).

This means fewer overall writes to HBase, and no risk of cross-regionserver
communication deadlocks.

--Tom
On Thu, Oct 10, 2013 at 1:23 PM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10?
> by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming
> RPC calls can be accepted because
>    everyone is connected to everyone in a spaghetti way and waiting on RPC
> calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10
> handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP
> code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Michael Segel [[EMAIL PROTECTED]]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: [EMAIL PROTECTED]
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out
> of resources to handle an RPC connection while within your coprocessor and
> you're waiting for a resource to be free and it can't because another
> coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always
> try thing to limit the RPC call, which would delay updating the counter.
> (Which may not be a problem) or redesign the coprocessors so that the
> coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the
> Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <
> [EMAIL PROTECTED]> wrote:
>
> > Michael,
> > I would also really like to know how this issue is caused also. I can't
> even give a solid way to reproduce our deadlock. It *seems* to happen more
> under load, but nothing can be proved yet. While google-ing and looking for
> an answer I came across that old message post  (
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E).
> This seemed to line up with what we are doing, so we _hope_ this will be a
> fix for us, but we aren't entirely sure.
> >
> >
> >
> > On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
> >> Can we just take a quick pause…
> >>
> >> John you wrote the following:
> >> "We have been running into an RPC deadlock issue on HBase and from
> >> investigation, we believe the root of the issue is in us doing cross
> >> region increments from a coprocessor. After some further searching and
> >> reading over this
> >> <
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E