Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - timeouts with lots of coprocessor puts on single row


Copy link to this message
-
Re: timeouts with lots of coprocessor puts on single row
anil gupta 2013-08-26, 15:43
On Mon, Aug 26, 2013 at 7:27 AM, Olle Mårtensson
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I have developed a coprocessor that is extending BaseRegionObserver and
> implements the
> postPut method. The postPut method scans the columns of the row that the
> put was issued on and calculates an aggregated based on these values, when
> this is done a row in another table is updated with the aggregated value.
>
This is an anti-pattern. It's not recommended to do put/deletes across
region servers like this. Try to move this aggregation on the client side
or at least outside RS. Here is the link for much detailed explanation why
this is not good: http://search-hadoop.com/m/XtAi5Fogw32

> This works out fine until I put some stress on one row, then the threads on
> the regionserver hosting the table will freeze on flushing the put on the
> aggregated value.
> The client application basically do 100 concurrent puts on one row in a
> tight loop( on the table where the coprocessor is activated ).
> After that the client sleeps for a while and tries to fetch the aggregated
> value and here the client freezes and periodically burps out exceptions.
> It works if I don't run so many put's in parallel.
>
> The HBASE environment is pseudo distributed 0.94.11 with one regionserver.
>
> I have tried using a connection pool in the coprocessor, bumped up the
> heapsize of the regionServer and also to up the number of RPC threads for
> the regionserver but without luck.
>
> The pseudo code postPut would be something like this:
>
> vals = env.getRegion().get(get).getFamilyMap().values()
> agg_val = aggregate(vals)
> agg_table = env.getTable("aggregates")
> agg_table.setAutoFlush(false)
> put = new Put()
> put.add(agg_val)
> agg_table.put(put)
> agg_table.flushCommits()
> agg_table.close()
>
> And the real clojure variant is:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj
>
> The hbase-site.xml:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml
>
> The regionserver stacktrace:
>
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace
>
> The client exceptions:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions
>
> Thanks // Olle
>

--
Thanks & Regards,
Anil Gupta