hi, all,
          We have some PCs(a dozens of them) with  4 cores CPU and 4GB ram
each, a hadoop instance is already running on these machines.

          Currently there is around 1.2GB ram left on every node.

          We want to setup a Hbase instance on these machines, with 3 quorum
servers and 5 region servers, but we kept getting "out of memory" error when
the table grows big on some nodes.

          I know HBase is really a memory eater, my question is can we set
up a Hbase cluster on these PC if we add more region servers? how much ram
should we reserve to make hbase run happily?

          An option is we put hbase regionservers and Hadoop
datanode/tasktracker on different node respectively(Not tried yet), but I
don't know how much this would influence the performance, as we can not
benefit from the data locality.

--
       best wishes.
                            steven

I would consider doing the following things:

- disable the block cache - you dont have ram to spare. 4gb is really
minimum (my laptop is 4gb ram and i decry how sucky I am occasionally)
- run between 1000-2000m, with the cache disabled you are in better shape
- consider adjusting the config values:
    <name>hbase.regionserver.global.memstore.upperLimit</name>
    <value>0.4</value>

This implies that memstore (the write buffer) can occupy 40% of heap.
This might be too aggressive with a small heap and CMS.  Try changing
that to 0.2 and:

    <name>hbase.regionserver.global.memstore.lowerLimit</name>
    <value>0.35</value>

to 0.10 or so.

Good luck!
-ryan

On Sun, Feb 28, 2010 at 11:59 PM, steven zhuang <[EMAIL PROTECTED]> wrote:
> hi, all,
>          We have some PCs(a dozens of them) with  4 cores CPU and 4GB ram
> each, a hadoop instance is already running on these machines.
>
>          Currently there is around 1.2GB ram left on every node.
>
>          We want to setup a Hbase instance on these machines, with 3 quorum
> servers and 5 region servers, but we kept getting "out of memory" error when
> the table grows big on some nodes.
>
>          I know HBase is really a memory eater, my question is can we set
> up a Hbase cluster on these PC if we add more region servers? how much ram
> should we reserve to make hbase run happily?
>
>          An option is we put hbase regionservers and Hadoop
> datanode/tasktracker on different node respectively(Not tried yet), but I
> don't know how much this would influence the performance, as we can not
> benefit from the data locality.
>
> --
>       best wishes.
>                            steven
>

thanks, Ryan,
            I think I can enlarge the heap size a little, and meanwhile
shrink the write buffer sounds good too, but I am not sure we can disable
the block cache, since the table is already built, anyway I can do that?

            I still have the question that if I add more regionservers, will
the load on current nodes be reduced to make the cluster run OK?

On Mon, Mar 1, 2010 at 4:08 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:

> I would consider doing the following things:
>
> - disable the block cache - you dont have ram to spare. 4gb is really
> minimum (my laptop is 4gb ram and i decry how sucky I am occasionally)
> - run between 1000-2000m, with the cache disabled you are in better shape
> - consider adjusting the config values:
>    <name>hbase.regionserver.global.memstore.upperLimit</name>
>    <value>0.4</value>
>
> This implies that memstore (the write buffer) can occupy 40% of heap.
> This might be too aggressive with a small heap and CMS.  Try changing
> that to 0.2 and:
>
>    <name>hbase.regionserver.global.memstore.lowerLimit</name>
>    <value>0.35</value>
>
> to 0.10 or so.
>
> Good luck!
> -ryan
>
> On Sun, Feb 28, 2010 at 11:59 PM, steven zhuang <[EMAIL PROTECTED]>
> wrote:
> > hi, all,
> >          We have some PCs(a dozens of them) with  4 cores CPU and 4GB ram
> > each, a hadoop instance is already running on these machines.
> >
> >          Currently there is around 1.2GB ram left on every node.
> >
> >          We want to setup a Hbase instance on these machines, with 3
> quorum
> > servers and 5 region servers, but we kept getting "out of memory" error
> when
> > the table grows big on some nodes.
> >
> >          I know HBase is really a memory eater, my question is can we set
> > up a Hbase cluster on these PC if we add more region servers? how much
> ram
> > should we reserve to make hbase run happily?
> >
> >          An option is we put hbase regionservers and Hadoop
> > datanode/tasktracker on different node respectively(Not tried yet), but I
> > don't know how much this would influence the performance, as we can not
> > benefit from the data locality.
> >
> > --
> >       best wishes.
> >                            steven
> >
>

--
       best wishes.
                            steven

The block cache is the memory limit used to keep blocks in the
regionserver memory to allow you to have faster operations. It's
literally a ram cache.

And yes, adding more regionservers helps :-)  Hooray for near-linear
scalability.

On Mon, Mar 1, 2010 at 12:26 AM, steven zhuang <[EMAIL PROTECTED]> wrote:
> thanks, Ryan,
>            I think I can enlarge the heap size a little, and meanwhile
> shrink the write buffer sounds good too, but I am not sure we can disable
> the block cache, since the table is already built, anyway I can do that?
>
>            I still have the question that if I add more regionservers, will
> the load on current nodes be reduced to make the cluster run OK?
>
>
>
> On Mon, Mar 1, 2010 at 4:08 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>
>> I would consider doing the following things:
>>
>> - disable the block cache - you dont have ram to spare. 4gb is really
>> minimum (my laptop is 4gb ram and i decry how sucky I am occasionally)
>> - run between 1000-2000m, with the cache disabled you are in better shape
>> - consider adjusting the config values:
>>    <name>hbase.regionserver.global.memstore.upperLimit</name>
>>    <value>0.4</value>
>>
>> This implies that memstore (the write buffer) can occupy 40% of heap.
>> This might be too aggressive with a small heap and CMS.  Try changing
>> that to 0.2 and:
>>
>>    <name>hbase.regionserver.global.memstore.lowerLimit</name>
>>    <value>0.35</value>
>>
>> to 0.10 or so.
>>
>> Good luck!
>> -ryan
>>
>> On Sun, Feb 28, 2010 at 11:59 PM, steven zhuang <[EMAIL PROTECTED]>
>> wrote:
>> > hi, all,
>> >          We have some PCs(a dozens of them) with  4 cores CPU and 4GB ram
>> > each, a hadoop instance is already running on these machines.
>> >
>> >          Currently there is around 1.2GB ram left on every node.
>> >
>> >          We want to setup a Hbase instance on these machines, with 3
>> quorum
>> > servers and 5 region servers, but we kept getting "out of memory" error
>> when
>> > the table grows big on some nodes.
>> >
>> >          I know HBase is really a memory eater, my question is can we set
>> > up a Hbase cluster on these PC if we add more region servers? how much
>> ram
>> > should we reserve to make hbase run happily?
>> >
>> >          An option is we put hbase regionservers and Hadoop
>> > datanode/tasktracker on different node respectively(Not tried yet), but I
>> > don't know how much this would influence the performance, as we can not
>> > benefit from the data locality.
>> >
>> > --
>> >       best wishes.
>> >                            steven
>> >
>>
>
>
>
> --
>       best wishes.
>                            steven
>