Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper


Copy link to this message
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
Shrijeet Paliwal 2012-05-01, 17:28
Dhaval was hitting https://issues.apache.org/jira/browse/HBASE-4508 ,
setting hbase.connection.per.config to false did it.
On Mon, Apr 30, 2012 at 8:17 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:
>
> I took a quick look, but didn't find a smoking gun. Can you get a
> jstack of the ThrfitServer when it has the 8K threads active? I'd like
> to see what exactly is keeping the threads running.
>
> -Joey
>
> On Mon, Apr 30, 2012 at 9:29 PM, Dhaval Y Shah
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi Joey.. Did you get a chance to look at the code? I digged through this a
> > bit further but didn't get much.. I added logging to the TThreadPoolServer
> > class and the thread pool there is pretty small.. I looked through the
> > ThriftServer class and the Hbase.java file generated by thrift and there
> > aren't many places where HBase configurations are created..The ones where
> > they are created look exactly the same as CDH3B4 which works correctly which
> > is confusing the hell out of me now
> >
> >
> > Dhaval Y Shah wrote:
> >>
> >> Cool thanks Joey... What I also noticed is that the number of threads
> >> spawned by the Thrift server also goes to > 16000 when the number of file
> >> descriptors is going to > 32000.. Probably we are not re-using threads in
> >> a correct way and we are missing the option to limit the number of threads
> >> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
> >>
> >>
> >> Joey Echeverria-2 wrote:
> >>>
> >>> I don't have the code in front of me, but I bet we're not reusing the
> >>> Configuration object when getting the HTable handle. I'll take a look at
> >>> when the regression went in this evening.
> >>>
> >>> -Joey
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>>
> >>>> We have an app written in Ruby which uses HBase as the backing store..
> >>>> It
> >>>> uses
> >>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
> >>>> distro
> >>>> until
> >>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> >>>> (which
> >>>> is
> >>>> the latest stable CDH release at this point) and in a matter of hours
> >>>> all
> >>>> Thrift
> >>>> servers went down..
> >>>>
> >>>> Upon further investigation I realized that it was hitting the limit on
> >>>> the
> >>>> number
> >>>> of allowed file descriptors (which is pretty high at 32k).. This problem
> >>>> occurs if
> >>>> I use thrift in any configuration (hsha, framed transport, threadpool)
> >>>> except the
> >>>> nonblocking mode.. Digging further I realized a couple of things:
> >>>> 1. Even with light load (1-2 processes hitting the thrift server in
> >>>> quick
> >>>> succession), thrift is spinning up new threads and each of the threads
> >>>> is
> >>>> maintaining a socket connection to zookeeper.. In a matter on minutes
> >>>> (with
> >>>> this
> >>>> load test), thrift has > 32k open connections with > 8k threads having
> >>>> connection
> >>>> to zookeeper which do not seem to die even after a day..
> >>>> 2. The logs show approx 3-4 open connections (presumably for each
> >>>> thread):
> >>>> java    53588 hbase 4135r  FIFO                0,6             177426
> >>>> pipe
> >>>> java    53588 hbase 4136w  FIFO                0,6             177426
> >>>> pipe
> >>>> java    53588 hbase 4137r  0000               0,11         0   177427
> >>>> eventpoll
> >>>> java    53588 hbase 4138u  IPv4             177428                TCP
> >>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> >>>> ED)
> >>>>
> >>>> CDH3B4 with the exact same configurations and the exact same setup works
> >>>> fine but
> >>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
> >>>> option
> >>>> because
> >>>> of the low throughput and single threaded nature..
> >>>>
> >>>> Any help is greatly appreciated..
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html