Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Allocating more heap for endpoint coprocessors


Copy link to this message
-
Re: Allocating more heap for endpoint coprocessors
+1 HBASE-4047

On Friday, August 31, 2012, Gary Helmling wrote:

> Maybe we need to add a coprocessors section to the ref guide.  I think
> all the current documentation is in javadoc.  And if all the
> potentially destabilizing issues of in-process coprocessor usage are
> not yet called out (memory usage, cpu, etc), we could more explicitly
> detail that.
>
> In we want to really be able to support these cases, though, we
> probably need to push HBASE-4047 (external coprocessor host) forward.
> Without it there's not a lot we can do to prevent coprocessors from
> taking down region servers.
>
>
> On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > Maybe we should be better with marking improvement as such in jira, and
> then have a special list in the release announcements highlighting these
> (otherwise they just be in the noise of all the other bug fixes).
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Ramkrishna.S.Vasudevan <[EMAIL PROTECTED]<javascript:;>
> >
> > To: [EMAIL PROTECTED] <javascript:;>; 'lars hofhansl' <
> [EMAIL PROTECTED] <javascript:;>>
> > Sent: Thursday, August 30, 2012 9:59 PM
> > Subject: RE: Allocating more heap for endpoint coprocessors
> >
> > @Lars/@Gary
> >
> > Do we need to document such things.  Recently someone was asking me a
> > question like this, if my endpoint impl is so memory intensive it just
> > affects a running cluster and already the RS
> > has a huge memory heap associated with it.
> >
> > So its better we document saying if your endpoint consumes memory and
> > because it runs along with RS based on your need add that extra amount of
> > memory that will be taken up by the endpoint impl.
> >
> > Regards
> > Ram
> >
> >> -----Original Message-----
> >> From: lars hofhansl [mailto:[EMAIL PROTECTED] <javascript:;>]
> >> Sent: Friday, August 31, 2012 3:04 AM
> >> To: [EMAIL PROTECTED] <javascript:;>
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> >> you to share state between RegionObservers (and Endpoints) within the
> >> same RegionServer.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Gary Helmling <[EMAIL PROTECTED] <javascript:;>>
> >> To: [EMAIL PROTECTED] <javascript:;>
> >> Sent: Thursday, August 30, 2012 1:59 PM
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> Endpoint coprocessors are loaded and run within the HBase RegionServer
> >> process.  Your endpoint coprocessors will be running on the region
> >> servers hosting the regions for the table(s) on which the coprocessor
> >> is configured.
> >>
> >> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> >> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> >> the region server.
> >>
> >> Note that a separate coprocessor instance is loaded for each table
> >> region, so, say you want to allocate 10MB for your coprocessor, but
> >> each region server hosts 20 regions, you would want to increase the
> >> heap size by 200MB (20x10MB).
> >>
> >> --gh
> >>
> >> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]<javascript:;>
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > We have some memory intensive endpoint coprocessors running on our
> >> RegionServers. As a result, we want to allocate more heap for the
> >> coprocessors, but there doesn't seem to be much documentation on which
> >> Hbase processes are directly responsible for coprocessors. Does anyone
> >> happen to know or direct me to some resource that does?
> >> >
> >> > Thanks,
> >> > Young Kim
> >> >
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB