Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Region server exception


Copy link to this message
-
Re: Region server exception
Kiru Pakkirisamy 2013-08-29, 20:56
Ted,
When there are more than 32 concurrent clients (in a 4 nodes x 8 core cluster). I keep getting responseTooSlow for my coprocessors.
Our app is built mainly using coprocessor and a few multi-get.

(responseTooSlow): {"processingtimems":10682,"call":"execCoprocessor([B@511c627c, getFoo({T_5208=0.004815409309791332,

.....(multiple values of T_<id>=<double value>)
20), rpc version=1, client version=0, methodsFingerPrint=0), rpc version=1, client version=29, methodsFingerPrint
=-1368823753","client":"10.149.5.56:38292","starttimems":1377808493508,"queuetimems":7,"class":"HRegionServer","responsesize":0,"method":"execCoprocessor"}

We do a orderby on the T_<number> and do calculations on the double. This finishes in 400 msec (total T_ values processed is around 600K)  when there is only one client. But takes 8000 or 10000 when the # of concurrent connections are increased to 32 or above.
 
Regards,
- kiru
________________________________
 From: Ted Yu <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Kiru Pakkirisamy <[EMAIL PROTECTED]>
Sent: Thursday, August 29, 2013 1:17 PM
Subject: Re: Region server exception
 

This exception means some other thread was holding the lock for extended
period of time.

Can you tell us more about your coprocessor ?

Thanks
On Thu, Aug 29, 2013 at 12:55 PM, Kiru Pakkirisamy <
[EMAIL PROTECTED]> wrote:

>
>
> This exception stack happens from within my coprocessor code on concurrent
> reads. Any ideas ?
>
> java.io.InterruptedIOException
> at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5894)
> at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5875)
> at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRe
> gion.java:5803)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(H
> Region.java:3852)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(H
> Region.java:3896)
>
>
>
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com