Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor Increments


Copy link to this message
-
Re: Coprocessor Increments
I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.

Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.  

But the key is to really understanding and confirming what's causing the Deadlock in detail.

On Oct 10, 2013, at 11:15 AM, John Weatherford <[EMAIL PROTECTED]> wrote:

> Michael,
> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post  (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting
>>>> region, see the following:
>>>>
>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>
>>>>  if (!rowIsInRange(getRegionInfo(), row)) {
>>>>
>>>>    throw new WrongRegionException("Requested row out of range for " +
>>>>
>>>>        op + " on HRegion " + this + ", startKey='" +
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>> [EMAIL PROTECTED]
>>>> <mailto:[EMAIL PROTECTED]>> wrote:
>>>>
>>>>> First, Thank you everyone for the quick response.
>>>>>
>>>>> Ted:
>>>>> The custom split policy could be an interesting solution.
>>>>> Regarding the
>>>>> other option you sent, if I pull the end row key, I could construct
>>>>> an end
>>>>> key, but if I suffix the row key with the end key of the region,
>>>>> would that
>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>> suffixed key
>>>>> now be "California-total-California-**12346"  and the other region's
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB