Hi,

We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?

Thanks,
Young Kim

Endpoint coprocessors are loaded and run within the HBase RegionServer
process.  Your endpoint coprocessors will be running on the region
servers hosting the regions for the table(s) on which the coprocessor
is configured.

So the way to allocate more memory is by setting either HBASE_HEAPSIZE
or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
the region server.

Note that a separate coprocessor instance is loaded for each table
region, so, say you want to allocate 10MB for your coprocessor, but
each region server hosts 20 regions, you would want to increase the
heap size by 200MB (20x10MB).

--gh

On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]> wrote:
> Hi,
>
> We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?
>
> Thanks,
> Young Kim
>

In the upcoming 0.94.2 release will also have HBASE-6505, which allows you to share state between RegionObservers (and Endpoints) within the same RegionServer.

-- Lars

________________________________
 From: Gary Helmling <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thursday, August 30, 2012 1:59 PM
Subject: Re: Allocating more heap for endpoint coprocessors
 
Endpoint coprocessors are loaded and run within the HBase RegionServer
process.  Your endpoint coprocessors will be running on the region
servers hosting the regions for the table(s) on which the coprocessor
is configured.

So the way to allocate more memory is by setting either HBASE_HEAPSIZE
or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
the region server.

Note that a separate coprocessor instance is loaded for each table
region, so, say you want to allocate 10MB for your coprocessor, but
each region server hosts 20 regions, you would want to increase the
heap size by 200MB (20x10MB).

--gh

On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]> wrote:
> Hi,
>
> We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?
>
> Thanks,
> Young Kim
>

@Lars/@Gary

Do we need to document such things.  Recently someone was asking me a
question like this, if my endpoint impl is so memory intensive it just
affects a running cluster and already the RS
has a huge memory heap associated with it.

So its better we document saying if your endpoint consumes memory and
because it runs along with RS based on your need add that extra amount of
memory that will be taken up by the endpoint impl.

Regards
Ram

> -----Original Message-----
> From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> Sent: Friday, August 31, 2012 3:04 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> you to share state between RegionObservers (and Endpoints) within the
> same RegionServer.
>
> -- Lars
>
>
>
> ________________________________
>  From: Gary Helmling <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thursday, August 30, 2012 1:59 PM
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> Endpoint coprocessors are loaded and run within the HBase RegionServer
> process.  Your endpoint coprocessors will be running on the region
> servers hosting the regions for the table(s) on which the coprocessor
> is configured.
>
> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> the region server.
>
> Note that a separate coprocessor instance is loaded for each table
> region, so, say you want to allocate 10MB for your coprocessor, but
> each region server hosts 20 regions, you would want to increase the
> heap size by 200MB (20x10MB).
>
> --gh
>
> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > We have some memory intensive endpoint coprocessors running on our
> RegionServers. As a result, we want to allocate more heap for the
> coprocessors, but there doesn't seem to be much documentation on which
> Hbase processes are directly responsible for coprocessors. Does anyone
> happen to know or direct me to some resource that does?
> >
> > Thanks,
> > Young Kim
> >

Maybe we should be better with marking improvement as such in jira, and then have a special list in the release announcements highlighting these (otherwise they just be in the noise of all the other bug fixes).

-- Lars

________________________________
 From: Ramkrishna.S.Vasudevan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; 'lars hofhansl' <[EMAIL PROTECTED]>
Sent: Thursday, August 30, 2012 9:59 PM
Subject: RE: Allocating more heap for endpoint coprocessors
 
@Lars/@Gary

Do we need to document such things.  Recently someone was asking me a
question like this, if my endpoint impl is so memory intensive it just
affects a running cluster and already the RS
has a huge memory heap associated with it.

So its better we document saying if your endpoint consumes memory and
because it runs along with RS based on your need add that extra amount of
memory that will be taken up by the endpoint impl.

Regards
Ram

> -----Original Message-----
> From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> Sent: Friday, August 31, 2012 3:04 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> you to share state between RegionObservers (and Endpoints) within the
> same RegionServer.
>
> -- Lars
>
>
>
> ________________________________
>  From: Gary Helmling <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thursday, August 30, 2012 1:59 PM
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> Endpoint coprocessors are loaded and run within the HBase RegionServer
> process.  Your endpoint coprocessors will be running on the region
> servers hosting the regions for the table(s) on which the coprocessor
> is configured.
>
> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> the region server.
>
> Note that a separate coprocessor instance is loaded for each table
> region, so, say you want to allocate 10MB for your coprocessor, but
> each region server hosts 20 regions, you would want to increase the
> heap size by 200MB (20x10MB).
>
> --gh
>
> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > We have some memory intensive endpoint coprocessors running on our
> RegionServers. As a result, we want to allocate more heap for the
> coprocessors, but there doesn't seem to be much documentation on which
> Hbase processes are directly responsible for coprocessors. Does anyone
> happen to know or direct me to some resource that does?
> >
> > Thanks,
> > Young Kim
> >

Maybe we need to add a coprocessors section to the ref guide.  I think
all the current documentation is in javadoc.  And if all the
potentially destabilizing issues of in-process coprocessor usage are
not yet called out (memory usage, cpu, etc), we could more explicitly
detail that.

In we want to really be able to support these cases, though, we
probably need to push HBASE-4047 (external coprocessor host) forward.
Without it there's not a lot we can do to prevent coprocessors from
taking down region servers.
On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Maybe we should be better with marking improvement as such in jira, and then have a special list in the release announcements highlighting these (otherwise they just be in the noise of all the other bug fixes).
>
> -- Lars
>
>
>
> ________________________________
>  From: Ramkrishna.S.Vasudevan <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; 'lars hofhansl' <[EMAIL PROTECTED]>
> Sent: Thursday, August 30, 2012 9:59 PM
> Subject: RE: Allocating more heap for endpoint coprocessors
>
> @Lars/@Gary
>
> Do we need to document such things.  Recently someone was asking me a
> question like this, if my endpoint impl is so memory intensive it just
> affects a running cluster and already the RS
> has a huge memory heap associated with it.
>
> So its better we document saying if your endpoint consumes memory and
> because it runs along with RS based on your need add that extra amount of
> memory that will be taken up by the endpoint impl.
>
> Regards
> Ram
>
>> -----Original Message-----
>> From: lars hofhansl [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, August 31, 2012 3:04 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Allocating more heap for endpoint coprocessors
>>
>> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
>> you to share state between RegionObservers (and Endpoints) within the
>> same RegionServer.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>  From: Gary Helmling <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Thursday, August 30, 2012 1:59 PM
>> Subject: Re: Allocating more heap for endpoint coprocessors
>>
>> Endpoint coprocessors are loaded and run within the HBase RegionServer
>> process.  Your endpoint coprocessors will be running on the region
>> servers hosting the regions for the table(s) on which the coprocessor
>> is configured.
>>
>> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
>> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
>> the region server.
>>
>> Note that a separate coprocessor instance is loaded for each table
>> region, so, say you want to allocate 10MB for your coprocessor, but
>> each region server hosts 20 regions, you would want to increase the
>> heap size by 200MB (20x10MB).
>>
>> --gh
>>
>> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > We have some memory intensive endpoint coprocessors running on our
>> RegionServers. As a result, we want to allocate more heap for the
>> coprocessors, but there doesn't seem to be much documentation on which
>> Hbase processes are directly responsible for coprocessors. Does anyone
>> happen to know or direct me to some resource that does?
>> >
>> > Thanks,
>> > Young Kim
>> >

+1 HBASE-4047

On Friday, August 31, 2012, Gary Helmling wrote:

> Maybe we need to add a coprocessors section to the ref guide.  I think
> all the current documentation is in javadoc.  And if all the
> potentially destabilizing issues of in-process coprocessor usage are
> not yet called out (memory usage, cpu, etc), we could more explicitly
> detail that.
>
> In we want to really be able to support these cases, though, we
> probably need to push HBASE-4047 (external coprocessor host) forward.
> Without it there's not a lot we can do to prevent coprocessors from
> taking down region servers.
>
>
> On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > Maybe we should be better with marking improvement as such in jira, and
> then have a special list in the release announcements highlighting these
> (otherwise they just be in the noise of all the other bug fixes).
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Ramkrishna.S.Vasudevan <[EMAIL PROTECTED]<javascript:;>
> >
> > To: [EMAIL PROTECTED] <javascript:;>; 'lars hofhansl' <
> [EMAIL PROTECTED] <javascript:;>>
> > Sent: Thursday, August 30, 2012 9:59 PM
> > Subject: RE: Allocating more heap for endpoint coprocessors
> >
> > @Lars/@Gary
> >
> > Do we need to document such things.  Recently someone was asking me a
> > question like this, if my endpoint impl is so memory intensive it just
> > affects a running cluster and already the RS
> > has a huge memory heap associated with it.
> >
> > So its better we document saying if your endpoint consumes memory and
> > because it runs along with RS based on your need add that extra amount of
> > memory that will be taken up by the endpoint impl.
> >
> > Regards
> > Ram
> >
> >> -----Original Message-----
> >> From: lars hofhansl [mailto:[EMAIL PROTECTED] <javascript:;>]
> >> Sent: Friday, August 31, 2012 3:04 AM
> >> To: [EMAIL PROTECTED] <javascript:;>
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> >> you to share state between RegionObservers (and Endpoints) within the
> >> same RegionServer.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Gary Helmling <[EMAIL PROTECTED] <javascript:;>>
> >> To: [EMAIL PROTECTED] <javascript:;>
> >> Sent: Thursday, August 30, 2012 1:59 PM
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> Endpoint coprocessors are loaded and run within the HBase RegionServer
> >> process.  Your endpoint coprocessors will be running on the region
> >> servers hosting the regions for the table(s) on which the coprocessor
> >> is configured.
> >>
> >> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> >> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> >> the region server.
> >>
> >> Note that a separate coprocessor instance is loaded for each table
> >> region, so, say you want to allocate 10MB for your coprocessor, but
> >> each region server hosts 20 regions, you would want to increase the
> >> heap size by 200MB (20x10MB).
> >>
> >> --gh
> >>
> >> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <[EMAIL PROTECTED]<javascript:;>
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > We have some memory intensive endpoint coprocessors running on our
> >> RegionServers. As a result, we want to allocate more heap for the
> >> coprocessors, but there doesn't seem to be much documentation on which
> >> Hbase processes are directly responsible for coprocessors. Does anyone
> >> happen to know or direct me to some resource that does?
> >> >
> >> > Thanks,
> >> > Young Kim
> >> >
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)