Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - FeedbackRe: Suspected memory leak


+
Gaojinchao 2011-12-04, 03:57
+
lars hofhansl 2011-12-04, 06:15
+
Gaojinchao 2011-12-04, 07:22
+
lars hofhansl 2011-12-04, 20:24
Copy link to this message
-
Re: FeedbackRe: Suspected memory leak
Shrijeet Paliwal 2011-12-04, 20:30
Did Gaojinchao attached the stack dump people received it (Lars?).
Could some one or Gaojinchao attach it to the jira.

-Shrijeet

On Sun, Dec 4, 2011 at 12:24 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
> Thanks. Now the question is: How many connection threads do we have?
>
> I think there is one per regionserver, which would indeed be a problem.
> Need to look at the code again (I'm only partially familiar with the client code).
>
> Either the client should chunk (like the server does), or there should be a limited number of thread that
> perform IO on behalf of the client (or both).
>
> -- Lars
>
>
> ----- Original Message -----
> From: Gaojinchao <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
> Cc: Chenjian <[EMAIL PROTECTED]>; wenzaohua <[EMAIL PROTECTED]>
> Sent: Saturday, December 3, 2011 11:22 PM
> Subject: Re: FeedbackRe: Suspected memory leak
>
> This is dump stack.
>
>
> -----邮件原件-----
> 发件人: lars hofhansl [mailto:[EMAIL PROTECTED]]
> 发送时间: 2011年12月4日 14:15
> 收件人: [EMAIL PROTECTED]
> 抄送: Chenjian; wenzaohua
> 主题: Re: FeedbackRe: Suspected memory leak
>
> Dropping user list.
>
> Could you (or somebody) point me to where the client is using NIO?
> I'm looking at HBaseClient and I do not see references to NIO, also it seems that all work is handed off to
> separate threads: HBaseClient.Connection, and the JDK will not cache more than 3 direct buffers per thread.
>
> It's possible (likely?) that I missed something in the code.
>
> Thanks.
>
> -- Lars
>
> ________________________________
> From: Gaojinchao <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc: Chenjian <[EMAIL PROTECTED]>; wenzaohua <[EMAIL PROTECTED]>
> Sent: Saturday, December 3, 2011 7:57 PM
> Subject: FeedbackRe: Suspected memory leak
>
> Thank you for your help.
>
> This issue appears to be a configuration problem:
> 1. HBase client uses NIO(socket) API that uses the direct memory.
> 2. Default -XXMaxDirectMemorySize value is equal to -Xmx value, So if there doesn't have "full gc", all direct memory can't reclaim. Unfortunately, using GC confiugre parameter of our client doesn't produce any "full gc".
>
> This is only a preliminary result,  All tests is running, If have any further results , we will be fed back.
> Finally , I will update our story to issue https://issues.apache.org/jira/browse/HBASE-4633.
>
> If our digging is crrect, whether we should set a default value for the "-XXMaxDirectMemorySize" to prevent this situation?
>
>
> Thanks
>
> -----邮件原件-----
> 发件人: bijieshan [mailto:[EMAIL PROTECTED]]
> 发送时间: 2011年12月2日 15:37
> 收件人: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> 抄送: Chenjian; wenzaohua
> 主题: Re: Suspected memory leak
>
> Thank you all.
> I think it's the same problem with the link provided by Stack. Because the heap-size is stabilized, but the non-heap size keep growing. So I think not the problem of the CMS GC bug.
> And we have known the content of the problem memory section, all the records contains the info like below:
> "|www.hostname00000000000002087075.comlhggmdjapwpfvkqvxgnskzzydiywoacjnpljkarlehrnzzbpbxc||||||460|||||||||||Agent||||";
> "BBZHtable_UFDR_058,048342220093168-02570"
> ........
>
> Jieshan.
>
> -----邮件原件-----
> 发件人: Kihwal Lee [mailto:[EMAIL PROTECTED]]
> 发送时间: 2011年12月2日 4:20
> 收件人: [EMAIL PROTECTED]
> 抄送: Ramakrishna s vasudevan; [EMAIL PROTECTED]
> 主题: Re: Suspected memory leak
>
> Adding to the excellent write-up by Jonathan:
> Since finalizer is involved, it takes two GC cycles to collect them.  Due to a bug/bugs in the CMS GC, collection may not happen and the heap can grow really big.  See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7112034 for details.
>
> Koji tried "-XX:-CMSConcurrentMTEnabled" and confirmed that all the socket related objects were being collected properly. This option forces the concurrent marker to be one thread. This was for HDFS, but I think the same applies here.
+
Ted Yu 2011-12-04, 20:43
+
lars hofhansl 2011-12-04, 21:12
+
Ted Yu 2011-12-04, 23:37
+
Gaojinchao 2011-12-05, 00:45
+
Gaojinchao 2011-12-05, 01:01
+
Ted Yu 2011-12-05, 01:05
+
Gaojinchao 2011-12-05, 01:17
+
Ted Yu 2011-12-05, 03:39
+
Gaojinchao 2011-12-05, 04:18
+
Sandy Pratt 2011-12-05, 21:54
+
Lars 2011-12-05, 06:08
+
Stack 2011-12-05, 17:47