Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper


Copy link to this message
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
HBASE-4508 is in CDH3u3 (per
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.

On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java    53588 hbase 4135r  FIFO                0,6             177426 pipe
>> java    53588 hbase 4136w  FIFO                0,6             177426 pipe
>> java    53588 hbase 4137r  0000               0,11         0   177427
>> eventpoll
>> java    53588 hbase 4138u  IPv4             177428                TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB