Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase client locks application during major compactions


+
Baugher,Bryan 2012-12-28, 17:40
+
Ted Yu 2012-12-28, 18:14
+
Baugher,Bryan 2012-12-28, 18:41
Copy link to this message
-
Re: HBase client locks application during major compactions
I was talking about the server which was anonymized:
***/***:60020

Cheers

On Fri, Dec 28, 2012 at 10:41 AM, Baugher,Bryan <[EMAIL PROTECTED]>wrote:

>
>
> On 12/28/12 12:14 PM, "Ted Yu" <[EMAIL PROTECTED]> wrote:
>
> >Looks like there was socket timeout :
> >
> >java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> >channel to be ready for read. ch :
> >java.nio.channels.SocketChannel[connected local=/***:39752
> >remote=***/***:60020]
> >
> >Have you collected / checked GC log on the server referenced above ?
>
> I am not sure exactly which server you are referring to. For the
> application server we don't currently collect gc logs. For hbase we do but
> the gc logs were truncated recently and won't help.
>
> >
> >BTW Have you considered deploying 0.92.2 in your cluster ?
>
> Not really. We have stuck with cloudera's distribution for a couple years
> now and I don't really see us going down that track.
>
> >
> >Thanks, glad to see Cerner using HBase.
> >
> >On Fri, Dec 28, 2012 at 9:40 AM, Baugher,Bryan
> ><[EMAIL PROTECTED]>wrote:
> >
> >> Hi everyone,
> >>
> >> For the past month or so we have noticed that some of our applications
> >> become frozen about once a day and need to be restarted in order to
> >>bring
> >> them back. We eventually figured out that it was caused by/happening
> >>during
> >> major compactions.
> >>
> >> We have automated major compactions disabled and are running them
> >>manually
> >> on each table sequentially each day starting at 4am. We are running on
> >> CDH4.1.1 (Hbase Version : 0.92.1-cdh4.1.1). Interestingly enough this is
> >> only happening in our dev environment with each region server serving
> >>~650
> >> regions.
> >>
> >> Looking at the logs in HBase show that the compactions are occurring and
> >> this warning repeatedly while the compactions are occurring,
> >>
> >> WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
> >> getHTableDescriptors(), rpc version=1, client version=29,
> >> methodsFingerPrint=400804878 from ***: output error
> >>
> >> Looking at our application logs we often see this error or a
> >>variation[1].
> >>
> >> I took a thread dump of our application while it was locked and saw that
> >> nearly all of the threads in the application were blocked by a single
> >> thread that was waiting on HBaseClient$Call[2].
> >>
> >> [1] - http://pastebin.com/P4skndEg
> >> [2] - http://pastebin.com/YLZn3SRK
> >>
> >>
> >> CONFIDENTIALITY NOTICE This message and any included attachments are
> >>from
> >> Cerner Corporation and are intended only for the addressee. The
> >>information
> >> contained in this message is confidential and may constitute inside or
> >> non-public information under international, federal, or state securities
> >> laws. Unauthorized forwarding, printing, copying, distribution, or use
> >>of
> >> such information is strictly prohibited and may be unlawful. If you are
> >>not
> >> the addressee, please promptly delete this message and notify the
> >>sender of
> >> the delivery error by e-mail or you may call Cerner's corporate offices
> >>in
> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
> >>
>
>
+
Baugher,Bryan 2012-12-28, 19:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB