Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor Increments


Copy link to this message
-
Re: Coprocessor Increments
We used to do our updates through coprocessors, but it was not scalable. We
extracted the update code into a separate system, added row transaction
IDs, and haven't looked back.

For each incoming message, we compute the set of updates that message will
generate. With a batch of messages, we merge updates to the same row
together (for example, 3 increments of 1 each becomes 1 increment of 3).

This means fewer overall writes to HBase, and no risk of cross-regionserver
communication deadlocks.

--Tom
On Thu, Oct 10, 2013 at 1:23 PM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10?
> by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming
> RPC calls can be accepted because
>    everyone is connected to everyone in a spaghetti way and waiting on RPC
> calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10
> handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP
> code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Michael Segel [[EMAIL PROTECTED]]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: [EMAIL PROTECTED]
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out
> of resources to handle an RPC connection while within your coprocessor and
> you're waiting for a resource to be free and it can't because another
> coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always
> try thing to limit the RPC call, which would delay updating the counter.
> (Which may not be a problem) or redesign the coprocessors so that the
> coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the
> Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <
> [EMAIL PROTECTED]> wrote:
>
> > Michael,
> > I would also really like to know how this issue is caused also. I can't
> even give a solid way to reproduce our deadlock. It *seems* to happen more
> under load, but nothing can be proved yet. While google-ing and looking for
> an answer I came across that old message post  (
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E).
> This seemed to line up with what we are doing, so we _hope_ this will be a
> fix for us, but we aren't entirely sure.
> >
> >
> >
> > On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
> >> Can we just take a quick pause…
> >>
> >> John you wrote the following:
> >> "We have been running into an RPC deadlock issue on HBase and from
> >> investigation, we believe the root of the issue is in us doing cross
> >> region increments from a coprocessor. After some further searching and
> >> reading over this
> >> <
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB