Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper

Copy link to this message
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
>From top of my head see if you are hitting
https://issues.apache.org/jira/browse/HBASE-4508 (and hence
https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
3777 and has a new config which may help you. Of course it depends on if
the new hbase version you are running has 4508 or not.

On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:

> We have an app written in Ruby which uses HBase as the backing store.. It
> uses
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> (which is
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift
> servers went down..
> Upon further investigation I realized that it was hitting the limit on the
> number
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs
> if
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except
> the
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick
> succession), thrift is spinning up new threads and each of the threads is
> maintaining a socket connection to zookeeper.. In a matter on minutes
> (with this
> load test), thrift has > 32k open connections with > 8k threads having
> connection
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java    53588 hbase 4135r  FIFO                0,6             177426 pipe
> java    53588 hbase 4136w  FIFO                0,6             177426 pipe
> java    53588 hbase 4137r  0000               0,11         0   177427
> eventpoll
> java    53588 hbase 4138u  IPv4             177428                TCP
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
> CDH3B4 with the exact same configurations and the exact same setup works
> fine
> but
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because
> of the low throughput and single threaded nature..
> Can someone help please?