Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> FeedbackRe: Suspected memory leak


Copy link to this message
-
Re: FeedbackRe: Suspected memory leak
Can you try specifying XX:MaxDirectMemorySize with moderate value and see
if the leak gets under control ?

Thanks

2011/12/4 Gaojinchao <[EMAIL PROTECTED]>

> I have attached the stack in
> https://issues.apache.org/jira/browse/HBASE-4633.
> I will update our story.
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:[EMAIL PROTECTED]]
> 发送时间: 2011年12月5日 7:37
> 收件人: [EMAIL PROTECTED]; lars hofhansl
> 主题: Re: FeedbackRe: Suspected memory leak
>
> I looked through TRUNK and 0.90 code but didn't find
> HBaseClient.Connection.setParam().
> The method should be sendParam().
>
> When I was in China I tried to access Jonathan's post but wasn't able to.
>
> If Jinchao's stack trace resonates with the one Jonathan posted, we should
> consider using netty for HBaseClient.
>
> Cheers
>
> On Sun, Dec 4, 2011 at 1:12 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
> > I think HBASE-4508 is unrelated.
> > The "connections" I referring to are HBaseClient.Connection objects (not
> > HConnections).
> > It turns out that HBaseClient.Connection.setParam is actually called
> > directly by the client threads, which means we can get
> > an unlimited amount of DirectByteBuffers (until we get a full GC).
> >
> > The JDK will cache 3 per thread with a size necessary to serve the IO. So
> > sending some large requests from many thread
> > will lead to OOM.
> >
> > I think that was a related thread that Stack forwarded a while back from
> > the asynchbase mailing lists.
> >
> > Jinchao, could you add a text version (not a png image, please :-) ) of
> > this to the jira?
> >
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Ted Yu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> > Cc: Gaojinchao <[EMAIL PROTECTED]>; Chenjian <
> [EMAIL PROTECTED]>;
> > wenzaohua <[EMAIL PROTECTED]>
> > Sent: Sunday, December 4, 2011 12:43 PM
> > Subject: Re: FeedbackRe: Suspected memory leak
> >
> > I think Jinchao wasn't using HBASE-4508 in his 0.90 distribution because
> > 0.90.5 hasn't been released.
> > Assuming the NIO consumption is related to the number of connections from
> > client side, it would help to perform benchmarking on 0.90.5
> >
> > Jinchao:
> > Please attach stack trace to HBASE-4633 so that we can verify our
> > assumptions.
> >
> > Thanks
> >
> > On Sun, Dec 4, 2011 at 12:24 PM, lars hofhansl <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Thanks. Now the question is: How many connection threads do we have?
> > >
> > > I think there is one per regionserver, which would indeed be a problem.
> > > Need to look at the code again (I'm only partially familiar with the
> > > client code).
> > >
> > > Either the client should chunk (like the server does), or there should
> be
> > > a limited number of thread that
> > > perform IO on behalf of the client (or both).
> > >
> > > -- Lars
> > >
> > >
> > > ----- Original Message -----
> > > From: Gaojinchao <[EMAIL PROTECTED]>
> > > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <
> > > [EMAIL PROTECTED]>
> > > Cc: Chenjian <[EMAIL PROTECTED]>; wenzaohua <
> [EMAIL PROTECTED]
> > >
> > > Sent: Saturday, December 3, 2011 11:22 PM
> > > Subject: Re: FeedbackRe: Suspected memory leak
> > >
> > > This is dump stack.
> > >
> > >
> > > -----邮件原件-----
> > > 发件人: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > > 发送时间: 2011年12月4日 14:15
> > > 收件人: [EMAIL PROTECTED]
> > > 抄送: Chenjian; wenzaohua
> > > 主题: Re: FeedbackRe: Suspected memory leak
> > >
> > > Dropping user list.
> > >
> > > Could you (or somebody) point me to where the client is using NIO?
> > > I'm looking at HBaseClient and I do not see references to NIO, also it
> > > seems that all work is handed off to
> > > separate threads: HBaseClient.Connection, and the JDK will not cache
> more
> > > than 3 direct buffers per thread.