Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Coprocessor / threading model


Copy link to this message
-
Re: Coprocessor / threading model
Anoop John 2013-01-13, 16:12
In your CP methods you will get ObserverContext object from which you can
get HRS object.
ObserverContext.getEnvironment().getRegionServerServices()
>From this HRS you can get hold to any of the region served by that RS.
Then directly call methods on HRegion to insert data. :)
Good luck..
-Anoop-

On Sun, Jan 13, 2013 at 4:12 PM, Adrien Mogenet <[EMAIL PROTECTED]>wrote:

> Thanks for pointing me out the Jira, that's useful for my understanding.
> I'm using HBase 0.94.3, and regions of main and index table are co-located
> on the same RS as in Anoop's design. I'll browse the API tomorrow to find
> out how to not use HTable but inter-CPs communication.
>
>
> On Sun, Jan 13, 2013 at 11:04 AM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > In Anoop's soln its basicallly the put happens directly on the index
> region
> > rather than doing a put thro HTable.
> >
> > Regards
> > Ram
> >
> > On Sun, Jan 13, 2013 at 9:28 AM, Andrew Purtell <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Yes, especially if the cross region communication is in process.
> > >
> > > On Jan 12, 2013, at 6:48 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > >
> > > > bq. Coprocessors are meant to operate on the region to which they are
> > > > associated.
> > > >
> > > > For Anoop's case, the secondary table(s) have their regions aligned
> > with
> > > > the corresponding region from primary table. Meaning, related regions
> > are
> > > > served by the same region server.
> > > > Would writes to such regions of secondary table(s) be acceptable ?
> > > >
> > > > Thanks
> > > >
> > > > On Sat, Jan 12, 2013 at 6:39 PM, Andrew Purtell <[EMAIL PROTECTED]
> >
> > > wrote:
> > > >
> > > >>> In pre-put, I trigger another Put() in an external table (to build
> > the
> > > >> secondary index).
> > > >>
> > > >> We should probably call this a Coprocessor anti-pattern.
> > > >>
> > > >> Coprocessors are meant to operate on the region to which they are
> > > >> associated. They are a way you can extend HBase function while it
> > > operates
> > > >> in region on data for the region. Think of them as loadable kernel
> > > modules.
> > > >> They are not a general purpose server side platform for programming
> as
> > > if
> > > >> you are building a HBase client (with HTable, etc.). Just because
> you
> > > can
> > > >> do this doesn't mean you should.
> > > >>
> > > >>
> > > >> On Sat, Jan 12, 2013 at 4:06 PM, Adrien Mogenet <
> > > [EMAIL PROTECTED]
> > > >>> wrote:
> > > >>
> > > >>> Hi there,
> > > >>>
> > > >>> I'm experiencing some issues with CP. I'm trying to implement an
> > > indexing
> > > >>> solution (inspired by Annop's slides). In pre-put, I trigger
> another
> > > >> Put()
> > > >>> in an external table (to build the secondary index). It works
> perfect
> > > for
> > > >>> one client, but when I'm inserting data from 2 separate clients, I
> > met
> > > >>> issues with HTable object (the one used in pre-Put()), because it's
> > not
> > > >>> thread-safe. I decided to move on TablePool and that fixed my
> issue.
> > > >>>
> > > >>> But if I increase the write-load (and concurrency) HBase is
> throwing
> > a
> > > >> OOM
> > > >>> exception because it can't create new native threads. Looking at
> > HBase
> > > >>> metrics "threads count", I see that roughly 3500 threads are
> created.
> > > >>>
> > > >>> I'm looking for documentation about how CPs are working with
> threads
> > :
> > > >>> what/when should I protect against concurrency issues ? How may I
> > solve
> > > >> my
> > > >>> issue ?
> > > >>>
> > > >>> Help is welcome :-)
> > > >>>
> > > >>> --
> > > >>> Adrien Mogenet
> > > >>> 06.59.16.64.22
> > > >>> http://www.mogenet.me
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >>
> > > >>   - Andy
> > > >>
> > > >> Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > >> (via Tom White)
> > > >>
> > >
> >
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me