Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase client locks application during major compactions

Baugher,Bryan 2012-12-28, 17:40
Copy link to this message
Re: HBase client locks application during major compactions
Looks like there was socket timeout :

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/***:39752

Have you collected / checked GC log on the server referenced above ?

BTW Have you considered deploying 0.92.2 in your cluster ?

Thanks, glad to see Cerner using HBase.

On Fri, Dec 28, 2012 at 9:40 AM, Baugher,Bryan <[EMAIL PROTECTED]>wrote:

> Hi everyone,
> For the past month or so we have noticed that some of our applications
> become frozen about once a day and need to be restarted in order to bring
> them back. We eventually figured out that it was caused by/happening during
> major compactions.
> We have automated major compactions disabled and are running them manually
> on each table sequentially each day starting at 4am. We are running on
> CDH4.1.1 (Hbase Version : 0.92.1-cdh4.1.1). Interestingly enough this is
> only happening in our dev environment with each region server serving ~650
> regions.
> Looking at the logs in HBase show that the compactions are occurring and
> this warning repeatedly while the compactions are occurring,
> WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
> getHTableDescriptors(), rpc version=1, client version=29,
> methodsFingerPrint=400804878 from ***: output error
> Looking at our application logs we often see this error or a variation[1].
> I took a thread dump of our application while it was locked and saw that
> nearly all of the threads in the application were blocked by a single
> thread that was waiting on HBaseClient$Call[2].
> [1] - http://pastebin.com/P4skndEg
> [2] - http://pastebin.com/YLZn3SRK
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Baugher,Bryan 2012-12-28, 18:41
Ted Yu 2012-12-28, 18:45
Baugher,Bryan 2012-12-28, 19:39