Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Potential memory leak in client RPC timeout mechanism


Copy link to this message
-
Re: Potential memory leak in client RPC timeout mechanism
Shrijeet Paliwal 2011-10-20, 02:00
Stack, I have created https://issues.apache.org/jira/browse/HBASE-4633
J-D, I had tried in past to get the dump in production (only
environment where this is reproducible) but failed. The application
freezes if any profiling activity is attempted.
I can try and get you a dump from dev environment. Thanks for writing.

-Shrijeet

On Wed, Oct 19, 2011 at 4:23 PM, Stack <[EMAIL PROTECTED]> wrote:
> And file an issue please Shrijeet so we don't forget about it.
> Thanks boss,
> St.Ack
>
> On Fri, Oct 14, 2011 at 12:12 PM, Shrijeet Paliwal
> <[EMAIL PROTECTED]> wrote:
>> Hi All,
>>
>> HBase version: 0.90.3 + Patches
>> Hadoop version: CDH3u0
>> Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
>> https://issues.apache.org/jira/browse/HBASE-4003
>>
>> We have been using the 'hbase.client.operation.timeout' knob
>> introduced in 2937 for quite some time now. It helps us enforce SLA.
>> We have two HBase clusters and two HBase client clusters. One of them
>> is much busier than the other.
>>
>> We have seen a deterministic behavior of clients running in busy
>> cluster. Their (client's) memory footprint increases consistently
>> after they have been up for roughly 24 hours.
>> This memory footprint almost doubles from its usual value (usual case
>> == RPC timeout disabled). After much investigation nothing concrete
>> came out and we had to put a hack
>> which keep heap size in control even when RPC timeout is enabled. Also
>> please note , the same behavior is not observed in 'not so busy
>> cluster.
>>
>> The patch is here : https://gist.github.com/1288023
>>
>> Can some one, who is also running RPC timeout in production under fair
>> load, please share the experience.
>>
>> -Shrijeet
>>
>