Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Coprocessor / threading model


+
Adrien Mogenet 2013-01-13, 00:06
+
Andrew Purtell 2013-01-13, 02:39
+
Ted Yu 2013-01-13, 02:48
+
Andrew Purtell 2013-01-13, 03:58
+
ramkrishna vasudevan 2013-01-13, 10:04
Copy link to this message
-
Re: Coprocessor / threading model
Adrien Mogenet 2013-01-13, 10:42
Thanks for pointing me out the Jira, that's useful for my understanding.
I'm using HBase 0.94.3, and regions of main and index table are co-located
on the same RS as in Anoop's design. I'll browse the API tomorrow to find
out how to not use HTable but inter-CPs communication.
On Sun, Jan 13, 2013 at 11:04 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> In Anoop's soln its basicallly the put happens directly on the index region
> rather than doing a put thro HTable.
>
> Regards
> Ram
>
> On Sun, Jan 13, 2013 at 9:28 AM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
>
> > Yes, especially if the cross region communication is in process.
> >
> > On Jan 12, 2013, at 6:48 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > bq. Coprocessors are meant to operate on the region to which they are
> > > associated.
> > >
> > > For Anoop's case, the secondary table(s) have their regions aligned
> with
> > > the corresponding region from primary table. Meaning, related regions
> are
> > > served by the same region server.
> > > Would writes to such regions of secondary table(s) be acceptable ?
> > >
> > > Thanks
> > >
> > > On Sat, Jan 12, 2013 at 6:39 PM, Andrew Purtell <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >>> In pre-put, I trigger another Put() in an external table (to build
> the
> > >> secondary index).
> > >>
> > >> We should probably call this a Coprocessor anti-pattern.
> > >>
> > >> Coprocessors are meant to operate on the region to which they are
> > >> associated. They are a way you can extend HBase function while it
> > operates
> > >> in region on data for the region. Think of them as loadable kernel
> > modules.
> > >> They are not a general purpose server side platform for programming as
> > if
> > >> you are building a HBase client (with HTable, etc.). Just because you
> > can
> > >> do this doesn't mean you should.
> > >>
> > >>
> > >> On Sat, Jan 12, 2013 at 4:06 PM, Adrien Mogenet <
> > [EMAIL PROTECTED]
> > >>> wrote:
> > >>
> > >>> Hi there,
> > >>>
> > >>> I'm experiencing some issues with CP. I'm trying to implement an
> > indexing
> > >>> solution (inspired by Annop's slides). In pre-put, I trigger another
> > >> Put()
> > >>> in an external table (to build the secondary index). It works perfect
> > for
> > >>> one client, but when I'm inserting data from 2 separate clients, I
> met
> > >>> issues with HTable object (the one used in pre-Put()), because it's
> not
> > >>> thread-safe. I decided to move on TablePool and that fixed my issue.
> > >>>
> > >>> But if I increase the write-load (and concurrency) HBase is throwing
> a
> > >> OOM
> > >>> exception because it can't create new native threads. Looking at
> HBase
> > >>> metrics "threads count", I see that roughly 3500 threads are created.
> > >>>
> > >>> I'm looking for documentation about how CPs are working with threads
> :
> > >>> what/when should I protect against concurrency issues ? How may I
> solve
> > >> my
> > >>> issue ?
> > >>>
> > >>> Help is welcome :-)
> > >>>
> > >>> --
> > >>> Adrien Mogenet
> > >>> 06.59.16.64.22
> > >>> http://www.mogenet.me
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
>

--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
+
Anoop John 2013-01-13, 16:12
+
Wei Tan 2013-01-15, 18:44
+
Varun Sharma 2013-01-15, 18:56
+
Andrew Purtell 2013-01-15, 19:20
+
Wei Tan 2013-01-15, 22:41
+
Anoop Sam John 2013-01-16, 04:39
+
Ted 2013-01-13, 01:38
+
anil gupta 2013-01-13, 02:30
+
Michel Segel 2013-01-13, 13:25