Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper


+
Dhaval Y Shah 2012-04-27, 18:02
+
Joey Echeverria 2012-04-27, 18:41
+
Dhaval Y Shah 2012-04-28, 04:02
+
Dhaval Y Shah 2012-05-01, 02:29
+
Joey Echeverria 2012-05-01, 03:17
+
Shrijeet Paliwal 2012-05-01, 17:28
+
Dhaval Shah 2012-04-27, 16:04
+
Shrijeet Paliwal 2012-04-27, 16:35
+
Harsh J 2012-04-29, 15:52
Copy link to this message
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
But by default the config in 4508 is set to not share connections, I think.

Sent from Droid

----- Reply message -----
From: "Harsh J" <[EMAIL PROTECTED]>
Date: Sun, Apr 29, 2012 8:52 am
Subject: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
To: <[EMAIL PROTECTED]>

HBASE-4508 is in CDH3u3 (per
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.

On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java    53588 hbase 4135r  FIFO                0,6             177426 pipe
>> java    53588 hbase 4136w  FIFO                0,6             177426 pipe
>> java    53588 hbase 4137r  0000               0,11         0   177427
>> eventpoll
>> java    53588 hbase 4138u  IPv4             177428                TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>

--
Harsh J