Dhaval Y Shah
2012-04-27, 18:02
Joey Echeverria
2012-04-27, 18:41
Dhaval Y Shah
2012-04-28, 04:02
Dhaval Y Shah
2012-05-01, 02:29
Joey Echeverria
2012-05-01, 03:17
Shrijeet Paliwal
2012-05-01, 17:28
Dhaval Shah
2012-04-27, 16:04
Shrijeet Paliwal
2012-04-27, 16:35
Harsh J
2012-04-29, 15:52
Shrijeet Paliwal
2012-04-29, 16:15
-
HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperDhaval Y Shah 2012-04-27, 18:02
We have an app written in Ruby which uses HBase as the backing store.. It
uses
Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
until
now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which
is
the latest stable CDH release at this point) and in a matter of hours all
Thrift
servers went down..
Upon further investigation I realized that it was hitting the limit on the
number
of allowed file descriptors (which is pretty high at 32k).. This problem
occurs if
I use thrift in any configuration (hsha, framed transport, threadpool)
except the
nonblocking mode.. Digging further I realized a couple of things:
1. Even with light load (1-2 processes hitting the thrift server in quick
succession), thrift is spinning up new threads and each of the threads is
maintaining a socket connection to zookeeper.. In a matter on minutes (with
this
load test), thrift has > 32k open connections with > 8k threads having
connection
to zookeeper which do not seem to die even after a day..
2. The logs show approx 3-4 open connections (presumably for each thread):
java 53588 hbase 4135r FIFO 0,6 177426 pipe
java 53588 hbase 4136w FIFO 0,6 177426 pipe
java 53588 hbase 4137r 0000 0,11 0 177427
eventpoll
java 53588 hbase 4138u IPv4 177428 TCP
njhaddev05:49729->njhaddev01:2181 (ESTABLISH
ED)
CDH3B4 with the exact same configurations and the exact same setup works
fine but
CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
because
of the low throughput and single threaded nature..
Any help is greatly appreciated..
--
View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
Sent from the HBase User mailing list archive at Nabble.com.
+
Dhaval Y Shah 2012-04-27, 18:02
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperJoey Echeverria 2012-04-27, 18:41
I don't have the code in front of me, but I bet we're not reusing the Configuration object when getting the HTable handle. I'll take a look at when the regression went in this evening.
-Joey
Sent from my iPhone
On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]> wrote:
>
> We have an app written in Ruby which uses HBase as the backing store.. It
> uses
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which
> is
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift
> servers went down..
>
> Upon further investigation I realized that it was hitting the limit on the
> number
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs if
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except the
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick
> succession), thrift is spinning up new threads and each of the threads is
> maintaining a socket connection to zookeeper.. In a matter on minutes (with
> this
> load test), thrift has > 32k open connections with > 8k threads having
> connection
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java 53588 hbase 4135r FIFO 0,6 177426 pipe
> java 53588 hbase 4136w FIFO 0,6 177426 pipe
> java 53588 hbase 4137r 0000 0,11 0 177427
> eventpoll
> java 53588 hbase 4138u IPv4 177428 TCP
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
>
> CDH3B4 with the exact same configurations and the exact same setup works
> fine but
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because
> of the low throughput and single threaded nature..
>
> Any help is greatly appreciated..
>
> --
> View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
-Joey
Sent from my iPhone
On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]> wrote:
>
> We have an app written in Ruby which uses HBase as the backing store.. It
> uses
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which
> is
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift
> servers went down..
>
> Upon further investigation I realized that it was hitting the limit on the
> number
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs if
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except the
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick
> succession), thrift is spinning up new threads and each of the threads is
> maintaining a socket connection to zookeeper.. In a matter on minutes (with
> this
> load test), thrift has > 32k open connections with > 8k threads having
> connection
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java 53588 hbase 4135r FIFO 0,6 177426 pipe
> java 53588 hbase 4136w FIFO 0,6 177426 pipe
> java 53588 hbase 4137r 0000 0,11 0 177427
> eventpoll
> java 53588 hbase 4138u IPv4 177428 TCP
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
>
> CDH3B4 with the exact same configurations and the exact same setup works
> fine but
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because
> of the low throughput and single threaded nature..
>
> Any help is greatly appreciated..
>
> --
> View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
+
Joey Echeverria 2012-04-27, 18:41
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperDhaval Y Shah 2012-04-28, 04:02
Cool thanks Joey... What I also noticed is that the number of threads spawned
by the Thrift server also goes to > 16000 when the number of file
descriptors is going to > 32000.. Probably we are not re-using threads in a
correct way and we are missing the option to limit the number of threads
(which I guess is fixed for 0.94 as per what I read in a jira ticket)
Joey Echeverria-2 wrote:
>
> I don't have the code in front of me, but I bet we're not reusing the
> Configuration object when getting the HTable handle. I'll take a look at
> when the regression went in this evening.
>
> -Joey
>
> Sent from my iPhone
>
> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
> wrote:
>
>>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
>> distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which
>> is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on
>> the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with
>> this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each
>> thread):
>> java 53588 hbase 4135r FIFO 0,6 177426
>> pipe
>> java 53588 hbase 4136w FIFO 0,6 177426
>> pipe
>> java 53588 hbase 4137r 0000 0,11 0 177427
>> eventpoll
>> java 53588 hbase 4138u IPv4 177428 TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Any help is greatly appreciated..
>>
>> --
>> View this message in context:
>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33761191.html
Sent from the HBase User mailing list archive at Nabble.com.
+
Dhaval Y Shah 2012-04-28, 04:02
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperDhaval Y Shah 2012-05-01, 02:29
Hi Joey.. Did you get a chance to look at the code? I digged through this a
bit further but didn't get much.. I added logging to the TThreadPoolServer
class and the thread pool there is pretty small.. I looked through the
ThriftServer class and the Hbase.java file generated by thrift and there
aren't many places where HBase configurations are created..The ones where
they are created look exactly the same as CDH3B4 which works correctly which
is confusing the hell out of me now
Dhaval Y Shah wrote:
>
> Cool thanks Joey... What I also noticed is that the number of threads
> spawned by the Thrift server also goes to > 16000 when the number of file
> descriptors is going to > 32000.. Probably we are not re-using threads in
> a correct way and we are missing the option to limit the number of threads
> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
>
>
> Joey Echeverria-2 wrote:
>>
>> I don't have the code in front of me, but I bet we're not reusing the
>> Configuration object when getting the HTable handle. I'll take a look at
>> when the regression went in this evening.
>>
>> -Joey
>>
>> Sent from my iPhone
>>
>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
>> wrote:
>>
>>>
>>> We have an app written in Ruby which uses HBase as the backing store..
>>> It
>>> uses
>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
>>> distro
>>> until
>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>>> (which
>>> is
>>> the latest stable CDH release at this point) and in a matter of hours
>>> all
>>> Thrift
>>> servers went down..
>>>
>>> Upon further investigation I realized that it was hitting the limit on
>>> the
>>> number
>>> of allowed file descriptors (which is pretty high at 32k).. This problem
>>> occurs if
>>> I use thrift in any configuration (hsha, framed transport, threadpool)
>>> except the
>>> nonblocking mode.. Digging further I realized a couple of things:
>>> 1. Even with light load (1-2 processes hitting the thrift server in
>>> quick
>>> succession), thrift is spinning up new threads and each of the threads
>>> is
>>> maintaining a socket connection to zookeeper.. In a matter on minutes
>>> (with
>>> this
>>> load test), thrift has > 32k open connections with > 8k threads having
>>> connection
>>> to zookeeper which do not seem to die even after a day..
>>> 2. The logs show approx 3-4 open connections (presumably for each
>>> thread):
>>> java 53588 hbase 4135r FIFO 0,6 177426
>>> pipe
>>> java 53588 hbase 4136w FIFO 0,6 177426
>>> pipe
>>> java 53588 hbase 4137r 0000 0,11 0 177427
>>> eventpoll
>>> java 53588 hbase 4138u IPv4 177428 TCP
>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>>> ED)
>>>
>>> CDH3B4 with the exact same configurations and the exact same setup works
>>> fine but
>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
>>> option
>>> because
>>> of the low throughput and single threaded nature..
>>>
>>> Any help is greatly appreciated..
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33763327.html
Sent from the HBase User mailing list archive at Nabble.com.
+
Dhaval Y Shah 2012-05-01, 02:29
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperJoey Echeverria 2012-05-01, 03:17
I took a quick look, but didn't find a smoking gun. Can you get a
jstack of the ThrfitServer when it has the 8K threads active? I'd like
to see what exactly is keeping the threads running.
-Joey
On Mon, Apr 30, 2012 at 9:29 PM, Dhaval Y Shah
<[EMAIL PROTECTED]> wrote:
>
> Hi Joey.. Did you get a chance to look at the code? I digged through this a
> bit further but didn't get much.. I added logging to the TThreadPoolServer
> class and the thread pool there is pretty small.. I looked through the
> ThriftServer class and the Hbase.java file generated by thrift and there
> aren't many places where HBase configurations are created..The ones where
> they are created look exactly the same as CDH3B4 which works correctly which
> is confusing the hell out of me now
>
>
> Dhaval Y Shah wrote:
>>
>> Cool thanks Joey... What I also noticed is that the number of threads
>> spawned by the Thrift server also goes to > 16000 when the number of file
>> descriptors is going to > 32000.. Probably we are not re-using threads in
>> a correct way and we are missing the option to limit the number of threads
>> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
>>
>>
>> Joey Echeverria-2 wrote:
>>>
>>> I don't have the code in front of me, but I bet we're not reusing the
>>> Configuration object when getting the HTable handle. I'll take a look at
>>> when the regression went in this evening.
>>>
>>> -Joey
>>>
>>> Sent from my iPhone
>>>
>>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>>
>>>> We have an app written in Ruby which uses HBase as the backing store..
>>>> It
>>>> uses
>>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
>>>> distro
>>>> until
>>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>>>> (which
>>>> is
>>>> the latest stable CDH release at this point) and in a matter of hours
>>>> all
>>>> Thrift
>>>> servers went down..
>>>>
>>>> Upon further investigation I realized that it was hitting the limit on
>>>> the
>>>> number
>>>> of allowed file descriptors (which is pretty high at 32k).. This problem
>>>> occurs if
>>>> I use thrift in any configuration (hsha, framed transport, threadpool)
>>>> except the
>>>> nonblocking mode.. Digging further I realized a couple of things:
>>>> 1. Even with light load (1-2 processes hitting the thrift server in
>>>> quick
>>>> succession), thrift is spinning up new threads and each of the threads
>>>> is
>>>> maintaining a socket connection to zookeeper.. In a matter on minutes
>>>> (with
>>>> this
>>>> load test), thrift has > 32k open connections with > 8k threads having
>>>> connection
>>>> to zookeeper which do not seem to die even after a day..
>>>> 2. The logs show approx 3-4 open connections (presumably for each
>>>> thread):
>>>> java 53588 hbase 4135r FIFO 0,6 177426
>>>> pipe
>>>> java 53588 hbase 4136w FIFO 0,6 177426
>>>> pipe
>>>> java 53588 hbase 4137r 0000 0,11 0 177427
>>>> eventpoll
>>>> java 53588 hbase 4138u IPv4 177428 TCP
>>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>>>> ED)
>>>>
>>>> CDH3B4 with the exact same configurations and the exact same setup works
>>>> fine but
>>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
>>>> option
>>>> because
>>>> of the low throughput and single threaded nature..
>>>>
>>>> Any help is greatly appreciated..
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33763327.html
> Sent from the HBase User mailing list archive at Nabble.com.
Joey Echeverria
Senior Solutions Architect
Cloudera, Inc.
jstack of the ThrfitServer when it has the 8K threads active? I'd like
to see what exactly is keeping the threads running.
-Joey
On Mon, Apr 30, 2012 at 9:29 PM, Dhaval Y Shah
<[EMAIL PROTECTED]> wrote:
>
> Hi Joey.. Did you get a chance to look at the code? I digged through this a
> bit further but didn't get much.. I added logging to the TThreadPoolServer
> class and the thread pool there is pretty small.. I looked through the
> ThriftServer class and the Hbase.java file generated by thrift and there
> aren't many places where HBase configurations are created..The ones where
> they are created look exactly the same as CDH3B4 which works correctly which
> is confusing the hell out of me now
>
>
> Dhaval Y Shah wrote:
>>
>> Cool thanks Joey... What I also noticed is that the number of threads
>> spawned by the Thrift server also goes to > 16000 when the number of file
>> descriptors is going to > 32000.. Probably we are not re-using threads in
>> a correct way and we are missing the option to limit the number of threads
>> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
>>
>>
>> Joey Echeverria-2 wrote:
>>>
>>> I don't have the code in front of me, but I bet we're not reusing the
>>> Configuration object when getting the HTable handle. I'll take a look at
>>> when the regression went in this evening.
>>>
>>> -Joey
>>>
>>> Sent from my iPhone
>>>
>>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>>
>>>> We have an app written in Ruby which uses HBase as the backing store..
>>>> It
>>>> uses
>>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
>>>> distro
>>>> until
>>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>>>> (which
>>>> is
>>>> the latest stable CDH release at this point) and in a matter of hours
>>>> all
>>>> Thrift
>>>> servers went down..
>>>>
>>>> Upon further investigation I realized that it was hitting the limit on
>>>> the
>>>> number
>>>> of allowed file descriptors (which is pretty high at 32k).. This problem
>>>> occurs if
>>>> I use thrift in any configuration (hsha, framed transport, threadpool)
>>>> except the
>>>> nonblocking mode.. Digging further I realized a couple of things:
>>>> 1. Even with light load (1-2 processes hitting the thrift server in
>>>> quick
>>>> succession), thrift is spinning up new threads and each of the threads
>>>> is
>>>> maintaining a socket connection to zookeeper.. In a matter on minutes
>>>> (with
>>>> this
>>>> load test), thrift has > 32k open connections with > 8k threads having
>>>> connection
>>>> to zookeeper which do not seem to die even after a day..
>>>> 2. The logs show approx 3-4 open connections (presumably for each
>>>> thread):
>>>> java 53588 hbase 4135r FIFO 0,6 177426
>>>> pipe
>>>> java 53588 hbase 4136w FIFO 0,6 177426
>>>> pipe
>>>> java 53588 hbase 4137r 0000 0,11 0 177427
>>>> eventpoll
>>>> java 53588 hbase 4138u IPv4 177428 TCP
>>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>>>> ED)
>>>>
>>>> CDH3B4 with the exact same configurations and the exact same setup works
>>>> fine but
>>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
>>>> option
>>>> because
>>>> of the low throughput and single threaded nature..
>>>>
>>>> Any help is greatly appreciated..
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33763327.html
> Sent from the HBase User mailing list archive at Nabble.com.
Joey Echeverria
Senior Solutions Architect
Cloudera, Inc.
+
Joey Echeverria 2012-05-01, 03:17
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperShrijeet Paliwal 2012-05-01, 17:28
Dhaval was hitting https://issues.apache.org/jira/browse/HBASE-4508 ,
setting hbase.connection.per.config to false did it.
On Mon, Apr 30, 2012 at 8:17 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:
>
> I took a quick look, but didn't find a smoking gun. Can you get a
> jstack of the ThrfitServer when it has the 8K threads active? I'd like
> to see what exactly is keeping the threads running.
>
> -Joey
>
> On Mon, Apr 30, 2012 at 9:29 PM, Dhaval Y Shah
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi Joey.. Did you get a chance to look at the code? I digged through this a
> > bit further but didn't get much.. I added logging to the TThreadPoolServer
> > class and the thread pool there is pretty small.. I looked through the
> > ThriftServer class and the Hbase.java file generated by thrift and there
> > aren't many places where HBase configurations are created..The ones where
> > they are created look exactly the same as CDH3B4 which works correctly which
> > is confusing the hell out of me now
> >
> >
> > Dhaval Y Shah wrote:
> >>
> >> Cool thanks Joey... What I also noticed is that the number of threads
> >> spawned by the Thrift server also goes to > 16000 when the number of file
> >> descriptors is going to > 32000.. Probably we are not re-using threads in
> >> a correct way and we are missing the option to limit the number of threads
> >> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
> >>
> >>
> >> Joey Echeverria-2 wrote:
> >>>
> >>> I don't have the code in front of me, but I bet we're not reusing the
> >>> Configuration object when getting the HTable handle. I'll take a look at
> >>> when the regression went in this evening.
> >>>
> >>> -Joey
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>>
> >>>> We have an app written in Ruby which uses HBase as the backing store..
> >>>> It
> >>>> uses
> >>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
> >>>> distro
> >>>> until
> >>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> >>>> (which
> >>>> is
> >>>> the latest stable CDH release at this point) and in a matter of hours
> >>>> all
> >>>> Thrift
> >>>> servers went down..
> >>>>
> >>>> Upon further investigation I realized that it was hitting the limit on
> >>>> the
> >>>> number
> >>>> of allowed file descriptors (which is pretty high at 32k).. This problem
> >>>> occurs if
> >>>> I use thrift in any configuration (hsha, framed transport, threadpool)
> >>>> except the
> >>>> nonblocking mode.. Digging further I realized a couple of things:
> >>>> 1. Even with light load (1-2 processes hitting the thrift server in
> >>>> quick
> >>>> succession), thrift is spinning up new threads and each of the threads
> >>>> is
> >>>> maintaining a socket connection to zookeeper.. In a matter on minutes
> >>>> (with
> >>>> this
> >>>> load test), thrift has > 32k open connections with > 8k threads having
> >>>> connection
> >>>> to zookeeper which do not seem to die even after a day..
> >>>> 2. The logs show approx 3-4 open connections (presumably for each
> >>>> thread):
> >>>> java 53588 hbase 4135r FIFO 0,6 177426
> >>>> pipe
> >>>> java 53588 hbase 4136w FIFO 0,6 177426
> >>>> pipe
> >>>> java 53588 hbase 4137r 0000 0,11 0 177427
> >>>> eventpoll
> >>>> java 53588 hbase 4138u IPv4 177428 TCP
> >>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> >>>> ED)
> >>>>
> >>>> CDH3B4 with the exact same configurations and the exact same setup works
> >>>> fine but
> >>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
> >>>> option
> >>>> because
> >>>> of the low throughput and single threaded nature..
> >>>>
> >>>> Any help is greatly appreciated..
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
setting hbase.connection.per.config to false did it.
On Mon, Apr 30, 2012 at 8:17 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:
>
> I took a quick look, but didn't find a smoking gun. Can you get a
> jstack of the ThrfitServer when it has the 8K threads active? I'd like
> to see what exactly is keeping the threads running.
>
> -Joey
>
> On Mon, Apr 30, 2012 at 9:29 PM, Dhaval Y Shah
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi Joey.. Did you get a chance to look at the code? I digged through this a
> > bit further but didn't get much.. I added logging to the TThreadPoolServer
> > class and the thread pool there is pretty small.. I looked through the
> > ThriftServer class and the Hbase.java file generated by thrift and there
> > aren't many places where HBase configurations are created..The ones where
> > they are created look exactly the same as CDH3B4 which works correctly which
> > is confusing the hell out of me now
> >
> >
> > Dhaval Y Shah wrote:
> >>
> >> Cool thanks Joey... What I also noticed is that the number of threads
> >> spawned by the Thrift server also goes to > 16000 when the number of file
> >> descriptors is going to > 32000.. Probably we are not re-using threads in
> >> a correct way and we are missing the option to limit the number of threads
> >> (which I guess is fixed for 0.94 as per what I read in a jira ticket)
> >>
> >>
> >> Joey Echeverria-2 wrote:
> >>>
> >>> I don't have the code in front of me, but I bet we're not reusing the
> >>> Configuration object when getting the HTable handle. I'll take a look at
> >>> when the regression went in this evening.
> >>>
> >>> -Joey
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On Apr 27, 2012, at 14:02, Dhaval Y Shah <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>>
> >>>> We have an app written in Ruby which uses HBase as the backing store..
> >>>> It
> >>>> uses
> >>>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
> >>>> distro
> >>>> until
> >>>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> >>>> (which
> >>>> is
> >>>> the latest stable CDH release at this point) and in a matter of hours
> >>>> all
> >>>> Thrift
> >>>> servers went down..
> >>>>
> >>>> Upon further investigation I realized that it was hitting the limit on
> >>>> the
> >>>> number
> >>>> of allowed file descriptors (which is pretty high at 32k).. This problem
> >>>> occurs if
> >>>> I use thrift in any configuration (hsha, framed transport, threadpool)
> >>>> except the
> >>>> nonblocking mode.. Digging further I realized a couple of things:
> >>>> 1. Even with light load (1-2 processes hitting the thrift server in
> >>>> quick
> >>>> succession), thrift is spinning up new threads and each of the threads
> >>>> is
> >>>> maintaining a socket connection to zookeeper.. In a matter on minutes
> >>>> (with
> >>>> this
> >>>> load test), thrift has > 32k open connections with > 8k threads having
> >>>> connection
> >>>> to zookeeper which do not seem to die even after a day..
> >>>> 2. The logs show approx 3-4 open connections (presumably for each
> >>>> thread):
> >>>> java 53588 hbase 4135r FIFO 0,6 177426
> >>>> pipe
> >>>> java 53588 hbase 4136w FIFO 0,6 177426
> >>>> pipe
> >>>> java 53588 hbase 4137r 0000 0,11 0 177427
> >>>> eventpoll
> >>>> java 53588 hbase 4138u IPv4 177428 TCP
> >>>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> >>>> ED)
> >>>>
> >>>> CDH3B4 with the exact same configurations and the exact same setup works
> >>>> fine but
> >>>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an
> >>>> option
> >>>> because
> >>>> of the low throughput and single threaded nature..
> >>>>
> >>>> Any help is greatly appreciated..
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
+
Shrijeet Paliwal 2012-05-01, 17:28
-
HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperDhaval Shah 2012-04-27, 16:04
We have an app written in Ruby which uses HBase as the backing store.. It uses
Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
until
now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which is
the latest stable CDH release at this point) and in a matter of hours all Thrift
servers went down..
Upon further investigation I realized that it was hitting the limit on the
number
of allowed file descriptors (which is pretty high at 32k).. This problem occurs
if
I use thrift in any configuration (hsha, framed transport, threadpool) except
the
nonblocking mode.. Digging further I realized a couple of things:
1. Even with light load (1-2 processes hitting the thrift server in quick
succession), thrift is spinning up new threads and each of the threads is
maintaining a socket connection to zookeeper.. In a matter on minutes (with this
load test), thrift has > 32k open connections with > 8k threads having
connection
to zookeeper which do not seem to die even after a day..
2. The logs show approx 3-4 open connections (presumably for each thread):
java 53588 hbase 4135r FIFO 0,6 177426 pipe
java 53588 hbase 4136w FIFO 0,6 177426 pipe
java 53588 hbase 4137r 0000 0,11 0 177427 eventpoll
java 53588 hbase 4138u IPv4 177428 TCP
njhaddev05:49729->njhaddev01:2181 (ESTABLISH
ED)
CDH3B4 with the exact same configurations and the exact same setup works fine
but
CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
because
of the low throughput and single threaded nature..
Can someone help please?
Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
until
now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which is
the latest stable CDH release at this point) and in a matter of hours all Thrift
servers went down..
Upon further investigation I realized that it was hitting the limit on the
number
of allowed file descriptors (which is pretty high at 32k).. This problem occurs
if
I use thrift in any configuration (hsha, framed transport, threadpool) except
the
nonblocking mode.. Digging further I realized a couple of things:
1. Even with light load (1-2 processes hitting the thrift server in quick
succession), thrift is spinning up new threads and each of the threads is
maintaining a socket connection to zookeeper.. In a matter on minutes (with this
load test), thrift has > 32k open connections with > 8k threads having
connection
to zookeeper which do not seem to die even after a day..
2. The logs show approx 3-4 open connections (presumably for each thread):
java 53588 hbase 4135r FIFO 0,6 177426 pipe
java 53588 hbase 4136w FIFO 0,6 177426 pipe
java 53588 hbase 4137r 0000 0,11 0 177427 eventpoll
java 53588 hbase 4138u IPv4 177428 TCP
njhaddev05:49729->njhaddev01:2181 (ESTABLISH
ED)
CDH3B4 with the exact same configurations and the exact same setup works fine
but
CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
because
of the low throughput and single threaded nature..
Can someone help please?
+
Dhaval Shah 2012-04-27, 16:04
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperShrijeet Paliwal 2012-04-27, 16:35
>From top of my head see if you are hitting
https://issues.apache.org/jira/browse/HBASE-4508 (and hence
https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
3777 and has a new config which may help you. Of course it depends on if
the new hbase version you are running has 4508 or not.
On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
> We have an app written in Ruby which uses HBase as the backing store.. It
> uses
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> (which is
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift
> servers went down..
>
> Upon further investigation I realized that it was hitting the limit on the
> number
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs
> if
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except
> the
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick
> succession), thrift is spinning up new threads and each of the threads is
> maintaining a socket connection to zookeeper.. In a matter on minutes
> (with this
> load test), thrift has > 32k open connections with > 8k threads having
> connection
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java 53588 hbase 4135r FIFO 0,6 177426 pipe
> java 53588 hbase 4136w FIFO 0,6 177426 pipe
> java 53588 hbase 4137r 0000 0,11 0 177427
> eventpoll
> java 53588 hbase 4138u IPv4 177428 TCP
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
>
> CDH3B4 with the exact same configurations and the exact same setup works
> fine
> but
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because
> of the low throughput and single threaded nature..
>
> Can someone help please?
>
>
https://issues.apache.org/jira/browse/HBASE-4508 (and hence
https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
3777 and has a new config which may help you. Of course it depends on if
the new hbase version you are running has 4508 or not.
On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
> We have an app written in Ruby which uses HBase as the backing store.. It
> uses
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
> (which is
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift
> servers went down..
>
> Upon further investigation I realized that it was hitting the limit on the
> number
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs
> if
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except
> the
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick
> succession), thrift is spinning up new threads and each of the threads is
> maintaining a socket connection to zookeeper.. In a matter on minutes
> (with this
> load test), thrift has > 32k open connections with > 8k threads having
> connection
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java 53588 hbase 4135r FIFO 0,6 177426 pipe
> java 53588 hbase 4136w FIFO 0,6 177426 pipe
> java 53588 hbase 4137r 0000 0,11 0 177427
> eventpoll
> java 53588 hbase 4138u IPv4 177428 TCP
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
>
> CDH3B4 with the exact same configurations and the exact same setup works
> fine
> but
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because
> of the low throughput and single threaded nature..
>
> Can someone help please?
>
>
+
Shrijeet Paliwal 2012-04-27, 16:35
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperHarsh J 2012-04-29, 15:52
HBASE-4508 is in CDH3u3 (per
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.
On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java 53588 hbase 4135r FIFO 0,6 177426 pipe
>> java 53588 hbase 4136w FIFO 0,6 177426 pipe
>> java 53588 hbase 4137r 0000 0,11 0 177427
>> eventpoll
>> java 53588 hbase 4138u IPv4 177428 TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>
--
Harsh J
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.
On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java 53588 hbase 4135r FIFO 0,6 177426 pipe
>> java 53588 hbase 4136w FIFO 0,6 177426 pipe
>> java 53588 hbase 4137r 0000 0,11 0 177427
>> eventpoll
>> java 53588 hbase 4138u IPv4 177428 TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>
--
Harsh J
+
Harsh J 2012-04-29, 15:52
-
Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to ZookeeperShrijeet Paliwal 2012-04-29, 16:15
But by default the config in 4508 is set to not share connections, I think.
Sent from Droid
----- Reply message -----
From: "Harsh J" <[EMAIL PROTECTED]>
Date: Sun, Apr 29, 2012 8:52 am
Subject: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
To: <[EMAIL PROTECTED]>
HBASE-4508 is in CDH3u3 (per
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.
On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java 53588 hbase 4135r FIFO 0,6 177426 pipe
>> java 53588 hbase 4136w FIFO 0,6 177426 pipe
>> java 53588 hbase 4137r 0000 0,11 0 177427
>> eventpoll
>> java 53588 hbase 4138u IPv4 177428 TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>
--
Harsh J
Sent from Droid
----- Reply message -----
From: "Harsh J" <[EMAIL PROTECTED]>
Date: Sun, Apr 29, 2012 8:52 am
Subject: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
To: <[EMAIL PROTECTED]>
HBASE-4508 is in CDH3u3 (per
http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html)
so this probably seems to be something else.
On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal
<[EMAIL PROTECTED]> wrote:
> From top of my head see if you are hitting
> https://issues.apache.org/jira/browse/HBASE-4508 (and hence
> https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of
> 3777 and has a new config which may help you. Of course it depends on if
> the new hbase version you are running has 4508 or not.
>
> On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:
>
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
>> until
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which is
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift
>> servers went down..
>>
>> Upon further investigation I realized that it was hitting the limit on the
>> number
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs
>> if
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except
>> the
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick
>> succession), thrift is spinning up new threads and each of the threads is
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with this
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each thread):
>> java 53588 hbase 4135r FIFO 0,6 177426 pipe
>> java 53588 hbase 4136w FIFO 0,6 177426 pipe
>> java 53588 hbase 4137r 0000 0,11 0 177427
>> eventpoll
>> java 53588 hbase 4138u IPv4 177428 TCP
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>>
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine
>> but
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because
>> of the low throughput and single threaded nature..
>>
>> Can someone help please?
>>
>>
--
Harsh J
+
Shrijeet Paliwal 2012-04-29, 16:15
