Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> timeouts with lots of coprocessor puts on single row


Copy link to this message
-
Re: timeouts with lots of coprocessor puts on single row
We did the same but on the client side, without any issue

On Monday, August 26, 2013, Olle Mårtensson wrote:

> Hi,
>
> I have developed a coprocessor that is extending BaseRegionObserver and
> implements the
> postPut method. The postPut method scans the columns of the row that the
> put was issued on and calculates an aggregated based on these values, when
> this is done a row in another table is updated with the aggregated value.
> This works out fine until I put some stress on one row, then the threads on
> the regionserver hosting the table will freeze on flushing the put on the
> aggregated value.
> The client application basically do 100 concurrent puts on one row in a
> tight loop( on the table where the coprocessor is activated ).
> After that the client sleeps for a while and tries to fetch the aggregated
> value and here the client freezes and periodically burps out exceptions.
> It works if I don't run so many put's in parallel.
>
> The HBASE environment is pseudo distributed 0.94.11 with one regionserver.
>
> I have tried using a connection pool in the coprocessor, bumped up the
> heapsize of the regionServer and also to up the number of RPC threads for
> the regionserver but without luck.
>
> The pseudo code postPut would be something like this:
>
> vals = env.getRegion().get(get).getFamilyMap().values()
> agg_val = aggregate(vals)
> agg_table = env.getTable("aggregates")
> agg_table.setAutoFlush(false)
> put = new Put()
> put.add(agg_val)
> agg_table.put(put)
> agg_table.flushCommits()
> agg_table.close()
>
> And the real clojure variant is:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj
>
> The hbase-site.xml:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml
>
> The regionserver stacktrace:
>
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace
>
> The client exceptions:
>
> https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions
>
> Thanks // Olle
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB