Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Coprocessor / threading model


+
Adrien Mogenet 2013-01-13, 00:06
+
Andrew Purtell 2013-01-13, 02:39
+
Ted Yu 2013-01-13, 02:48
+
Andrew Purtell 2013-01-13, 03:58
+
ramkrishna vasudevan 2013-01-13, 10:04
Copy link to this message
-
Re: Coprocessor / threading model
Thanks for pointing me out the Jira, that's useful for my understanding.
I'm using HBase 0.94.3, and regions of main and index table are co-located
on the same RS as in Anoop's design. I'll browse the API tomorrow to find
out how to not use HTable but inter-CPs communication.
On Sun, Jan 13, 2013 at 11:04 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> In Anoop's soln its basicallly the put happens directly on the index region
> rather than doing a put thro HTable.
>
> Regards
> Ram
>
> On Sun, Jan 13, 2013 at 9:28 AM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
>
> > Yes, especially if the cross region communication is in process.
> >
> > On Jan 12, 2013, at 6:48 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > bq. Coprocessors are meant to operate on the region to which they are
> > > associated.
> > >
> > > For Anoop's case, the secondary table(s) have their regions aligned
> with
> > > the corresponding region from primary table. Meaning, related regions
> are
> > > served by the same region server.
> > > Would writes to such regions of secondary table(s) be acceptable ?
> > >
> > > Thanks
> > >
> > > On Sat, Jan 12, 2013 at 6:39 PM, Andrew Purtell <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >>> In pre-put, I trigger another Put() in an external table (to build
> the
> > >> secondary index).
> > >>
> > >> We should probably call this a Coprocessor anti-pattern.
> > >>
> > >> Coprocessors are meant to operate on the region to which they are
> > >> associated. They are a way you can extend HBase function while it
> > operates
> > >> in region on data for the region. Think of them as loadable kernel
> > modules.
> > >> They are not a general purpose server side platform for programming as
> > if
> > >> you are building a HBase client (with HTable, etc.). Just because you
> > can
> > >> do this doesn't mean you should.
> > >>
> > >>
> > >> On Sat, Jan 12, 2013 at 4:06 PM, Adrien Mogenet <
> > [EMAIL PROTECTED]
> > >>> wrote:
> > >>
> > >>> Hi there,
> > >>>
> > >>> I'm experiencing some issues with CP. I'm trying to implement an
> > indexing
> > >>> solution (inspired by Annop's slides). In pre-put, I trigger another
> > >> Put()
> > >>> in an external table (to build the secondary index). It works perfect
> > for
> > >>> one client, but when I'm inserting data from 2 separate clients, I
> met
> > >>> issues with HTable object (the one used in pre-Put()), because it's
> not
> > >>> thread-safe. I decided to move on TablePool and that fixed my issue.
> > >>>
> > >>> But if I increase the write-load (and concurrency) HBase is throwing
> a
> > >> OOM
> > >>> exception because it can't create new native threads. Looking at
> HBase
> > >>> metrics "threads count", I see that roughly 3500 threads are created.
> > >>>
> > >>> I'm looking for documentation about how CPs are working with threads
> :
> > >>> what/when should I protect against concurrency issues ? How may I
> solve
> > >> my
> > >>> issue ?
> > >>>
> > >>> Help is welcome :-)
> > >>>
> > >>> --
> > >>> Adrien Mogenet
> > >>> 06.59.16.64.22
> > >>> http://www.mogenet.me
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
>

--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
+
Anoop John 2013-01-13, 16:12
+
Wei Tan 2013-01-15, 18:44
+
Varun Sharma 2013-01-15, 18:56
+
Andrew Purtell 2013-01-15, 19:20
+
Wei Tan 2013-01-15, 22:41
+
Anoop Sam John 2013-01-16, 04:39
+
Ted 2013-01-13, 01:38
+
anil gupta 2013-01-13, 02:30
+
Michel Segel 2013-01-13, 13:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB