Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Coprocessor / threading model


Copy link to this message
-
Re: Coprocessor / threading model
ramkrishna vasudevan 2013-01-13, 10:04
In Anoop's soln its basicallly the put happens directly on the index region
rather than doing a put thro HTable.

Regards
Ram

On Sun, Jan 13, 2013 at 9:28 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> Yes, especially if the cross region communication is in process.
>
> On Jan 12, 2013, at 6:48 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > bq. Coprocessors are meant to operate on the region to which they are
> > associated.
> >
> > For Anoop's case, the secondary table(s) have their regions aligned with
> > the corresponding region from primary table. Meaning, related regions are
> > served by the same region server.
> > Would writes to such regions of secondary table(s) be acceptable ?
> >
> > Thanks
> >
> > On Sat, Jan 12, 2013 at 6:39 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> >
> >>> In pre-put, I trigger another Put() in an external table (to build the
> >> secondary index).
> >>
> >> We should probably call this a Coprocessor anti-pattern.
> >>
> >> Coprocessors are meant to operate on the region to which they are
> >> associated. They are a way you can extend HBase function while it
> operates
> >> in region on data for the region. Think of them as loadable kernel
> modules.
> >> They are not a general purpose server side platform for programming as
> if
> >> you are building a HBase client (with HTable, etc.). Just because you
> can
> >> do this doesn't mean you should.
> >>
> >>
> >> On Sat, Jan 12, 2013 at 4:06 PM, Adrien Mogenet <
> [EMAIL PROTECTED]
> >>> wrote:
> >>
> >>> Hi there,
> >>>
> >>> I'm experiencing some issues with CP. I'm trying to implement an
> indexing
> >>> solution (inspired by Annop's slides). In pre-put, I trigger another
> >> Put()
> >>> in an external table (to build the secondary index). It works perfect
> for
> >>> one client, but when I'm inserting data from 2 separate clients, I met
> >>> issues with HTable object (the one used in pre-Put()), because it's not
> >>> thread-safe. I decided to move on TablePool and that fixed my issue.
> >>>
> >>> But if I increase the write-load (and concurrency) HBase is throwing a
> >> OOM
> >>> exception because it can't create new native threads. Looking at HBase
> >>> metrics "threads count", I see that roughly 3500 threads are created.
> >>>
> >>> I'm looking for documentation about how CPs are working with threads :
> >>> what/when should I protect against concurrency issues ? How may I solve
> >> my
> >>> issue ?
> >>>
> >>> Help is welcome :-)
> >>>
> >>> --
> >>> Adrien Mogenet
> >>> 06.59.16.64.22
> >>> http://www.mogenet.me
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>