Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor / threading model


Copy link to this message
-
Re: Coprocessor / threading model
There are a couple of different designs that you can use to perform the write to the secondary index.

I wouldn't call this an anti-pattern... (AP's comment)

Using htablepool wouldn't be my first choice, unless you are writing to a durable queue first which then uses the pool to write to the table.  This could work as part of a more general solution to handle indexing at a more general level. But that is a longer discussion.
Sent from a remote device. Please excuse any typos...

Mike Segel

On Jan 12, 2013, at 6:06 PM, Adrien Mogenet <[EMAIL PROTECTED]> wrote:

> Hi there,
>
> I'm experiencing some issues with CP. I'm trying to implement an indexing
> solution (inspired by Annop's slides). In pre-put, I trigger another Put()
> in an external table (to build the secondary index). It works perfect for
> one client, but when I'm inserting data from 2 separate clients, I met
> issues with HTable object (the one used in pre-Put()), because it's not
> thread-safe. I decided to move on TablePool and that fixed my issue.
>
> But if I increase the write-load (and concurrency) HBase is throwing a OOM
> exception because it can't create new native threads. Looking at HBase
> metrics "threads count", I see that roughly 3500 threads are created.
>
> I'm looking for documentation about how CPs are working with threads :
> what/when should I protect against concurrency issues ? How may I solve my
> issue ?
>
> Help is welcome :-)
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB