Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> FeedbackRe: Suspected memory leak


Copy link to this message
-
Re: FeedbackRe: Suspected memory leak
Can you try specifying XX:MaxDirectMemorySize with moderate value and see
if the leak gets under control ?

Thanks

2011/12/4 Gaojinchao <[EMAIL PROTECTED]>

> I have attached the stack in
> https://issues.apache.org/jira/browse/HBASE-4633.
> I will update our story.
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:[EMAIL PROTECTED]]
> 发送时间: 2011年12月5日 7:37
> 收件人: [EMAIL PROTECTED]; lars hofhansl
> 主题: Re: FeedbackRe: Suspected memory leak
>
> I looked through TRUNK and 0.90 code but didn't find
> HBaseClient.Connection.setParam().
> The method should be sendParam().
>
> When I was in China I tried to access Jonathan's post but wasn't able to.
>
> If Jinchao's stack trace resonates with the one Jonathan posted, we should
> consider using netty for HBaseClient.
>
> Cheers
>
> On Sun, Dec 4, 2011 at 1:12 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
> > I think HBASE-4508 is unrelated.
> > The "connections" I referring to are HBaseClient.Connection objects (not
> > HConnections).
> > It turns out that HBaseClient.Connection.setParam is actually called
> > directly by the client threads, which means we can get
> > an unlimited amount of DirectByteBuffers (until we get a full GC).
> >
> > The JDK will cache 3 per thread with a size necessary to serve the IO. So
> > sending some large requests from many thread
> > will lead to OOM.
> >
> > I think that was a related thread that Stack forwarded a while back from
> > the asynchbase mailing lists.
> >
> > Jinchao, could you add a text version (not a png image, please :-) ) of
> > this to the jira?
> >
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Ted Yu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> > Cc: Gaojinchao <[EMAIL PROTECTED]>; Chenjian <
> [EMAIL PROTECTED]>;
> > wenzaohua <[EMAIL PROTECTED]>
> > Sent: Sunday, December 4, 2011 12:43 PM
> > Subject: Re: FeedbackRe: Suspected memory leak
> >
> > I think Jinchao wasn't using HBASE-4508 in his 0.90 distribution because
> > 0.90.5 hasn't been released.
> > Assuming the NIO consumption is related to the number of connections from
> > client side, it would help to perform benchmarking on 0.90.5
> >
> > Jinchao:
> > Please attach stack trace to HBASE-4633 so that we can verify our
> > assumptions.
> >
> > Thanks
> >
> > On Sun, Dec 4, 2011 at 12:24 PM, lars hofhansl <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Thanks. Now the question is: How many connection threads do we have?
> > >
> > > I think there is one per regionserver, which would indeed be a problem.
> > > Need to look at the code again (I'm only partially familiar with the
> > > client code).
> > >
> > > Either the client should chunk (like the server does), or there should
> be
> > > a limited number of thread that
> > > perform IO on behalf of the client (or both).
> > >
> > > -- Lars
> > >
> > >
> > > ----- Original Message -----
> > > From: Gaojinchao <[EMAIL PROTECTED]>
> > > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <
> > > [EMAIL PROTECTED]>
> > > Cc: Chenjian <[EMAIL PROTECTED]>; wenzaohua <
> [EMAIL PROTECTED]
> > >
> > > Sent: Saturday, December 3, 2011 11:22 PM
> > > Subject: Re: FeedbackRe: Suspected memory leak
> > >
> > > This is dump stack.
> > >
> > >
> > > -----邮件原件-----
> > > 发件人: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > > 发送时间: 2011年12月4日 14:15
> > > 收件人: [EMAIL PROTECTED]
> > > 抄送: Chenjian; wenzaohua
> > > 主题: Re: FeedbackRe: Suspected memory leak
> > >
> > > Dropping user list.
> > >
> > > Could you (or somebody) point me to where the client is using NIO?
> > > I'm looking at HBaseClient and I do not see references to NIO, also it
> > > seems that all work is handed off to
> > > separate threads: HBaseClient.Connection, and the JDK will not cache
> more
> > > than 3 direct buffers per thread.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB