Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor Increments

Copy link to this message
Re: Coprocessor Increments

A couple of things…

First, it looks like when tuning HBase, you're not accounting for the additional overhead caused by unknown coprocessors.  I guess even if you take a SWAG, you could end up still running out of resources…

Second… I'm a little confused.
In the OP's problem… is he trying to increment a counter in the same table, or is his counter in a different table?
This would then become a discussion of schema design.

Next, I would have to agree with you about the need to refactor HBase code. Reviewing and refactoring is always a good thing if you have time. ;-)

With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?

The reason I ask is that if I were to implement an RO that updates a row in a second table, upon instantiation, i would create an HTable object referencing the second table and then use that to handle the data. Am I wrong to assume that in doing so, that I wouldn't be using the internal RPC threads and that the update would look like an external client?

Since my mind is pretty much mush by now… (long week, no sleep),  is it better to accept the higher overhead by making the RO a client itself, or should I risk the potential of deadlock?

So what am I missing?


On Oct 10, 2013, at 6:52 PM, Vladimir Rodionov <[EMAIL PROTECTED]> wrote:

> Michael,
> RS has 3 different pools of threads (Readers (accept RPC), Handlers (process requests), third pool (do not remember name) of threads send request results back  to callers )
> All of them communicate to through internal Queues. Cumbersome and needs to be  refactored, of course.
> All these pools are configurable (max threads). Readers and writers are fine - they will never get into any deadlock situation.
> Handlers are different. In my example, I described the situation when ALL handlers in RS1 are waiting on RPC calls to RS2 and
> ALL handlers in RS2 are waiting on RPC calls to RS2. For the sake of simplicity:
> RS1 and RS2 - two region servers
> handlers count = 1 - just one thread in a pool.
> 1. RS2 coprocessor sends RPC to RS1. From single handler thread. Total available handlers in RS2 = 0.  
> 2. RS1 receives request from RS2 CP2 (coprocessor).
> 3. RS1 handler1 receives request. Total available handlers in RS1 = 0;
> 4. RS1 coprocessor makes RPC call back to RS2
> 5. RS2 Reader thread places request into Handler pool queue, but there is no handlers available (see 1). Deadlock.
> and the request MUST timeout (fail) eventually. If it does not fail - there is the issue in HBase RPC.
> But technically, its a deadlock.
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> ________________________________________
> From: Michael Segel [[EMAIL PROTECTED]]
> Sent: Thursday, October 10, 2013 2:57 PM
> Subject: Re: Coprocessor Increments
> Ok…
> So…  I see you followed up to your earlier post.
> Lets walk through this to make sure we're on the same page.
> You put()  row 1 in table 1.
> The post_put() wants to insert a value in table 2.
> Now suppose I have an htable pool of connections in my Region Observer.
> (Is this overkill? I'm treating the RO as if its a client connecting to HBase.)
> RO performs a put() into the second table.
> The RPC handlers are a Region server resource, yes?
> So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests.
> Is that the case?
> On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <[EMAIL PROTECTED]> wrote:
>> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>> Making RPC call from within handler's therad is: