Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Coprocessor / threading model


+
Adrien Mogenet 2013-01-13, 00:06
+
Andrew Purtell 2013-01-13, 02:39
+
Ted Yu 2013-01-13, 02:48
Copy link to this message
-
Re: Coprocessor / threading model
Yes, especially if the cross region communication is in process.

On Jan 12, 2013, at 6:48 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> bq. Coprocessors are meant to operate on the region to which they are
> associated.
>
> For Anoop's case, the secondary table(s) have their regions aligned with
> the corresponding region from primary table. Meaning, related regions are
> served by the same region server.
> Would writes to such regions of secondary table(s) be acceptable ?
>
> Thanks
>
> On Sat, Jan 12, 2013 at 6:39 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>
>>> In pre-put, I trigger another Put() in an external table (to build the
>> secondary index).
>>
>> We should probably call this a Coprocessor anti-pattern.
>>
>> Coprocessors are meant to operate on the region to which they are
>> associated. They are a way you can extend HBase function while it operates
>> in region on data for the region. Think of them as loadable kernel modules.
>> They are not a general purpose server side platform for programming as if
>> you are building a HBase client (with HTable, etc.). Just because you can
>> do this doesn't mean you should.
>>
>>
>> On Sat, Jan 12, 2013 at 4:06 PM, Adrien Mogenet <[EMAIL PROTECTED]
>>> wrote:
>>
>>> Hi there,
>>>
>>> I'm experiencing some issues with CP. I'm trying to implement an indexing
>>> solution (inspired by Annop's slides). In pre-put, I trigger another
>> Put()
>>> in an external table (to build the secondary index). It works perfect for
>>> one client, but when I'm inserting data from 2 separate clients, I met
>>> issues with HTable object (the one used in pre-Put()), because it's not
>>> thread-safe. I decided to move on TablePool and that fixed my issue.
>>>
>>> But if I increase the write-load (and concurrency) HBase is throwing a
>> OOM
>>> exception because it can't create new native threads. Looking at HBase
>>> metrics "threads count", I see that roughly 3500 threads are created.
>>>
>>> I'm looking for documentation about how CPs are working with threads :
>>> what/when should I protect against concurrency issues ? How may I solve
>> my
>>> issue ?
>>>
>>> Help is welcome :-)
>>>
>>> --
>>> Adrien Mogenet
>>> 06.59.16.64.22
>>> http://www.mogenet.me
>>
>>
>>
>> --
>> Best regards,
>>
>>   - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
+
ramkrishna vasudevan 2013-01-13, 10:04
+
Adrien Mogenet 2013-01-13, 10:42
+
Anoop John 2013-01-13, 16:12
+
Wei Tan 2013-01-15, 18:44
+
Varun Sharma 2013-01-15, 18:56
+
Andrew Purtell 2013-01-15, 19:20
+
Wei Tan 2013-01-15, 22:41
+
Anoop Sam John 2013-01-16, 04:39
+
Ted 2013-01-13, 01:38
+
anil gupta 2013-01-13, 02:30
+
Michel Segel 2013-01-13, 13:25