Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> timeouts with lots of coprocessor puts on single row

Copy link to this message
Re: timeouts with lots of coprocessor puts on single row
On Mon, Aug 26, 2013 at 7:27 AM, Olle Mårtensson

> Hi,
> I have developed a coprocessor that is extending BaseRegionObserver and
> implements the
> postPut method. The postPut method scans the columns of the row that the
> put was issued on and calculates an aggregated based on these values, when
> this is done a row in another table is updated with the aggregated value.
This is an anti-pattern. It's not recommended to do put/deletes across
region servers like this. Try to move this aggregation on the client side
or at least outside RS. Here is the link for much detailed explanation why
this is not good: http://search-hadoop.com/m/XtAi5Fogw32

> This works out fine until I put some stress on one row, then the threads on
> the regionserver hosting the table will freeze on flushing the put on the
> aggregated value.
> The client application basically do 100 concurrent puts on one row in a
> tight loop( on the table where the coprocessor is activated ).
> After that the client sleeps for a while and tries to fetch the aggregated
> value and here the client freezes and periodically burps out exceptions.
> It works if I don't run so many put's in parallel.
> The HBASE environment is pseudo distributed 0.94.11 with one regionserver.
> I have tried using a connection pool in the coprocessor, bumped up the
> heapsize of the regionServer and also to up the number of RPC threads for
> the regionserver but without luck.
> The pseudo code postPut would be something like this:
> vals = env.getRegion().get(get).getFamilyMap().values()
> agg_val = aggregate(vals)
> agg_table = env.getTable("aggregates")
> agg_table.setAutoFlush(false)
> put = new Put()
> put.add(agg_val)
> agg_table.put(put)
> agg_table.flushCommits()
> agg_table.close()
> And the real clojure variant is:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj
> The hbase-site.xml:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml
> The regionserver stacktrace:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace
> The client exceptions:
> https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions
> Thanks // Olle

Thanks & Regards,
Anil Gupta