Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Coprocessor Increments


+
John Weatherford 2013-10-10, 01:43
+
Vladimir Rodionov 2013-10-10, 02:28
+
Ted Yu 2013-10-10, 02:39
+
Ted Yu 2013-10-10, 02:43
+
John Weatherford 2013-10-10, 03:26
+
Ted Yu 2013-10-10, 03:43
+
John Weatherford 2013-10-10, 04:03
Copy link to this message
-
Re: Coprocessor Increments
Michael Segel 2013-10-10, 14:57
Can we just take a quick pause…

John you wrote the following:
"We have been running into an RPC deadlock issue on HBase and from investigation, we believe the root of the issue is in us doing cross region increments from a coprocessor. After some further searching and reading over this <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+[EMAIL PROTECTED]%3E> we think that we can solve this by doing the increments locally on the region. "

Which goes back to some thing Andrew wrote concerning an RPC deadlock.

Can either you or Andrew explain in detail what is meant by the RPC deadlock?

This goes back to rethink how to implement coprocessors.
On Oct 9, 2013, at 11:03 PM, John Weatherford <[EMAIL PROTECTED]> wrote:

> Thank you ted. I might have to rethink my key design to accomodate this. With this design and using the totals in keys as you suggested, I would have to scan the entire "California" group to find the totals. Thank you very much for your help.
>
> -John
>
> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>> John:
>> Suppose 'California-**12346' is the start key of region 1 and 'California-**
>> 95424' is the start key of region 2. You can choose 'California-**12346#total'
>> to be the row key where increment is done for region 1 and
>> 'California-**95424#total'
>> to be the row key where increment is done for region 2.
>>
>> w.r.t. verification of whether given row key is indeed inside the hosting
>> region, see the following:
>>
>>   void checkRow(final byte [] row, String op) throws IOException {
>>
>>     if (!rowIsInRange(getRegionInfo(), row)) {
>>
>>       throw new WrongRegionException("Requested row out of range for " +
>>
>>           op + " on HRegion " + this + ", startKey='" +
>>
>> Cheers
>>
>>
>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>> [EMAIL PROTECTED]> wrote:
>>
>>> First, Thank you everyone for the quick response.
>>>
>>> Ted:
>>>   The custom split policy could be an interesting solution. Regarding the
>>> other option you sent, if I pull the end row key, I could construct an end
>>> key, but if I suffix the row key with the end key of the region, would that
>>> actually solve my problem? In the contrived case, wouldn't the suffixed key
>>> now be "California-total-California-**12346"  and the other region's
>>> counter would be "California-total-California-**95424" and both of these
>>> keys would actually end up on the second region since the sorting of the
>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>
>>> Part of the question is that I don't understand if calling
>>> incrementColumnValue() on the region will always execute on the region
>>> called, regardles of rowkey. If so, what happens when those regions are
>>> merged?
>>>
>>> Thanks again for the help!
>>>
>>>
>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>
>>>> bq. 'California-total' row in each region
>>>>
>>>> Of course such row key needs to be suffixed with either start or end key
>>>> of
>>>> the corresponding region.
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>
>>>>  John:
>>>>> Looks like you need a 'California-total' row in each region.
>>>>>
>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>> [EMAIL PROTECTED]
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>>  Contrived Example
>>>>>>>>
>>>>>>>
>>>>>>  Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>> incrementColumnValue() with a rowkey of "California-total"  all on
>>>>>>>>
>>>>>>> Region 1.
>>>>>>
>>>>>>  This would likely be on an insert on the same region. But as the table
>>>>>>>> grows, this secondary insert could end up on another region. If it is
+
John Weatherford 2013-10-10, 16:15
+
Michael Segel 2013-10-10, 16:55
+
Vladimir Rodionov 2013-10-10, 19:09
+
Vladimir Rodionov 2013-10-10, 19:23
+
Tom Brown 2013-10-10, 20:20
+
Michael Segel 2013-10-10, 21:57
+
Vladimir Rodionov 2013-10-10, 23:52
+
Michael Segel 2013-10-11, 16:10
+
Vladimir Rodionov 2013-10-11, 17:26
+
Michael Segel 2013-10-11, 19:09
+
John Weatherford 2013-10-12, 01:06
+
Ted Yu 2013-10-12, 14:42
+
Michael Segel 2013-10-12, 23:04
+
anil gupta 2013-10-13, 04:27
+
Michael Segel 2013-10-13, 13:02
+
anil gupta 2013-10-13, 15:15
+
Michael Segel 2013-10-14, 14:50
+
Tom Brown 2013-10-14, 16:36
+
John Weatherford 2013-10-14, 20:39
+
anil gupta 2013-10-14, 22:25
+
Ted Yu 2013-10-14, 22:34
+
anil gupta 2013-10-15, 03:57
+
Michael Segel 2013-10-15, 18:12
+
anil gupta 2013-10-15, 20:47