Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase client locks application during major compactions


Copy link to this message
-
Re: HBase client locks application during major compactions
Looks like there was socket timeout :

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/***:39752
remote=***/***:60020]

Have you collected / checked GC log on the server referenced above ?

BTW Have you considered deploying 0.92.2 in your cluster ?

Thanks, glad to see Cerner using HBase.

On Fri, Dec 28, 2012 at 9:40 AM, Baugher,Bryan <[EMAIL PROTECTED]>wrote:

> Hi everyone,
>
> For the past month or so we have noticed that some of our applications
> become frozen about once a day and need to be restarted in order to bring
> them back. We eventually figured out that it was caused by/happening during
> major compactions.
>
> We have automated major compactions disabled and are running them manually
> on each table sequentially each day starting at 4am. We are running on
> CDH4.1.1 (Hbase Version : 0.92.1-cdh4.1.1). Interestingly enough this is
> only happening in our dev environment with each region server serving ~650
> regions.
>
> Looking at the logs in HBase show that the compactions are occurring and
> this warning repeatedly while the compactions are occurring,
>
> WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
> getHTableDescriptors(), rpc version=1, client version=29,
> methodsFingerPrint=400804878 from ***: output error
>
> Looking at our application logs we often see this error or a variation[1].
>
> I took a thread dump of our application while it was locked and saw that
> nearly all of the threads in the application were blocked by a single
> thread that was waiting on HBaseClient$Call[2].
>
> [1] - http://pastebin.com/P4skndEg
> [2] - http://pastebin.com/YLZn3SRK
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB