Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - timeouts with lots of coprocessor puts on single row

Copy link to this message
Re: timeouts with lots of coprocessor puts on single row
Asaf Mesika 2013-08-26, 20:28
We did the same but on the client side, without any issue

On Monday, August 26, 2013, Olle Mårtensson wrote:

> Hi,
> I have developed a coprocessor that is extending BaseRegionObserver and
> implements the
> postPut method. The postPut method scans the columns of the row that the
> put was issued on and calculates an aggregated based on these values, when
> this is done a row in another table is updated with the aggregated value.
> This works out fine until I put some stress on one row, then the threads on
> the regionserver hosting the table will freeze on flushing the put on the
> aggregated value.
> The client application basically do 100 concurrent puts on one row in a
> tight loop( on the table where the coprocessor is activated ).
> After that the client sleeps for a while and tries to fetch the aggregated
> value and here the client freezes and periodically burps out exceptions.
> It works if I don't run so many put's in parallel.
> The HBASE environment is pseudo distributed 0.94.11 with one regionserver.
> I have tried using a connection pool in the coprocessor, bumped up the
> heapsize of the regionServer and also to up the number of RPC threads for
> the regionserver but without luck.
> The pseudo code postPut would be something like this:
> vals = env.getRegion().get(get).getFamilyMap().values()
> agg_val = aggregate(vals)
> agg_table = env.getTable("aggregates")
> agg_table.setAutoFlush(false)
> put = new Put()
> put.add(agg_val)
> agg_table.put(put)
> agg_table.flushCommits()
> agg_table.close()
> And the real clojure variant is:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj
> The hbase-site.xml:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml
> The regionserver stacktrace:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace
> The client exceptions:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions
> Thanks // Olle