Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Coprocessor / threading model


Copy link to this message
-
Coprocessor / threading model
Adrien Mogenet 2013-01-13, 00:06
Hi there,

I'm experiencing some issues with CP. I'm trying to implement an indexing
solution (inspired by Annop's slides). In pre-put, I trigger another Put()
in an external table (to build the secondary index). It works perfect for
one client, but when I'm inserting data from 2 separate clients, I met
issues with HTable object (the one used in pre-Put()), because it's not
thread-safe. I decided to move on TablePool and that fixed my issue.

But if I increase the write-load (and concurrency) HBase is throwing a OOM
exception because it can't create new native threads. Looking at HBase
metrics "threads count", I see that roughly 3500 threads are created.

I'm looking for documentation about how CPs are working with threads :
what/when should I protect against concurrency issues ? How may I solve my
issue ?

Help is welcome :-)

--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me