Hi
I try to use hbase 0.90 to store 100billion massage, and I have
setup hbase, and use API to store messages into hbase, It seems ok,
But, my mates tell me that , for hbase , max size for one region is
4GB , and for one server, the max number for region is 100, then for one
server, I can only store 400GB data, is it that ?

Or what about the max size for one region and what about the max
size of region for one server?

For 100billion message, I need to store more data in one server,
Could you help me

Thank you
--------------------
Tian Guanhua

This should help:
http://hbase.apache.org/book/important_configurations.html#bigger.regions

On Mon, Dec 17, 2012 at 9:11 AM, tgh <[EMAIL PROTECTED]> wrote:

>
>         Or what about the max size for one region and what about the max
> size of region for one server?
>

Thank you for your reply, and I visit the webpage, it is helpful,
And following it, I can use 500 region in ONE server, is it?
And then if I use 500 region in ONE server, and one region is 40GB, and one
server will store 20TB, it is ok , is it?

Thank you
---------------------
Tian Guanhua

-----邮件原件-----
发件人: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] 代表
Nicolas Liochon
发送时间: 2012年12月17日 16:28
收件人: [EMAIL PROTECTED]
主题: Re: what is the max size for one region and what is the max size of
region for one server

This should help:
http://hbase.apache.org/book/important_configurations.html#bigger.regions

On Mon, Dec 17, 2012 at 9:11 AM, tgh <[EMAIL PROTECTED]> wrote:

>
>         Or what about the max size for one region and what about the
> max size of region for one server?
>

You're reading correctly. It's a little bit extreme however. More extreme
than necessary imho: if you have 20TB of HBase data, this leads to 60TB of
hdfs data, plus the WALs. That's a lot for a single machine.

On Mon, Dec 17, 2012 at 9:56 AM, tgh <[EMAIL PROTECTED]> wrote:

> is it?
> And then if I use 500 region in ONE server, and one region
>

Thank you for your reply ,

but I write for I want to make sure, if the number of region in ONE server
exceed 300 or 500, the hbase will fail or something, or what is the max
number of region for ONE server?
And I use hbase 0.90,

Could you help me
Thank you
------------------------
Tian Guanhua

-----邮件原件-----
发件人: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] 代表
Nicolas Liochon
发送时间: 2012年12月17日 17:42
收件人: [EMAIL PROTECTED]
主题: Re: 答复: what is the max size for one region and what is the max size
of region for one server

You're reading correctly. It's a little bit extreme however. More extreme
than necessary imho: if you have 20TB of HBase data, this leads to 60TB of
hdfs data, plus the WALs. That's a lot for a single machine.

On Mon, Dec 17, 2012 at 9:56 AM, tgh <[EMAIL PROTECTED]> wrote:

> is it?
> And then if I use 500 region in ONE server, and one region
>

I think it's safer to use a newer version (0.94): there are a lot of things
around performances & volumes in the 0.92 & 0.94. As well, there are much
more bug fixes releases on the 0.94.

For the number of region, there is no maximum written in stone. Having too
many regions will essentially impact the performances. As I said, having
60TB of data per machine is not standard today (points are: that's a lot of
disk a single machine; what's the impact if you lose a node; what will be
the network load, ...). I suppose all this is documented in the usual books
on HBase.
On Mon, Dec 17, 2012 at 11:26 AM, tgh <[EMAIL PROTECTED]> wrote:

> number of region for ONE server?

Hi there,

When sizing your data, don't forget to read thisŠ

http://hbase.apache.org/book.html#schema.creation

and

http://hbase.apache.org/book.html#regions.arch

"9.7.5.4. KeyValue"

You need to understand how Hbase stores data internally on initial design
to avoid problems down the line.  Keep the keys as small as reasonable,
likewise CF name, and column names.
On 12/17/12 6:07 AM, "Nicolas Liochon" <[EMAIL PROTECTED]> wrote:

>I think it's safer to use a newer version (0.94): there are a lot of
>things
>around performances & volumes in the 0.92 & 0.94. As well, there are much
>more bug fixes releases on the 0.94.
>
>For the number of region, there is no maximum written in stone. Having too
>many regions will essentially impact the performances. As I said, having
>60TB of data per machine is not standard today (points are: that's a lot
>of
>disk a single machine; what's the impact if you lose a node; what will be
>the network load, ...). I suppose all this is documented in the usual
>books
>on HBase.
>
>
>On Mon, Dec 17, 2012 at 11:26 AM, tgh <[EMAIL PROTECTED]> wrote:
>
>> number of region for ONE server?

0.90.x supports up to 4GB region sizes max, not 40. You would need to upgrade to 0.92.x at least to go higher than that.

Sent from iPhone.

On Dec 17, 2012, at 9:31 AM, Doug Meil <[EMAIL PROTECTED]> wrote:

>
> Hi there,
>
> When sizing your data, don't forget to read thisŠ
>
> http://hbase.apache.org/book.html#schema.creation
>
> and
>
> http://hbase.apache.org/book.html#regions.arch
>
> "9.7.5.4. KeyValue"
>
> You need to understand how Hbase stores data internally on initial design
> to avoid problems down the line.  Keep the keys as small as reasonable,
> likewise CF name, and column names.
>
>
>
>
> On 12/17/12 6:07 AM, "Nicolas Liochon" <[EMAIL PROTECTED]> wrote:
>
>> I think it's safer to use a newer version (0.94): there are a lot of
>> things
>> around performances & volumes in the 0.92 & 0.94. As well, there are much
>> more bug fixes releases on the 0.94.
>>
>> For the number of region, there is no maximum written in stone. Having too
>> many regions will essentially impact the performances. As I said, having
>> 60TB of data per machine is not standard today (points are: that's a lot
>> of
>> disk a single machine; what's the impact if you lose a node; what will be
>> the network load, ...). I suppose all this is documented in the usual
>> books
>> on HBase.
>>
>>
>> On Mon, Dec 17, 2012 at 11:26 AM, tgh <[EMAIL PROTECTED]> wrote:
>>
>>> number of region for ONE server?
>

Don't use HBase 0.90. Our current release is 0.94. You will find the
community is able to help you much more satisfactorily if you start with
the current release.
On Mon, Dec 17, 2012 at 2:26 AM, tgh <[EMAIL PROTECTED]> wrote:

> Thank you for your reply ,
>
> but I write for I want to make sure, if the number of region in ONE server
> exceed 300 or 500, the hbase will fail or something, or what is the max
> number of region for ONE server?
> And I use hbase 0.90,
>
> Could you help me
>
>
> Thank you
> ------------------------
> Tian Guanhua
>
>
>
>
>
>
>
>
>
> -----邮件原件-----
> 发件人: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]] 代表
> Nicolas Liochon
> 发送时间: 2012年12月17日 17:42
> 收件人: [EMAIL PROTECTED]
> 主题: Re: 答复: what is the max size for one region and what is the max size
> of region for one server
>
> You're reading correctly. It's a little bit extreme however. More extreme
> than necessary imho: if you have 20TB of HBase data, this leads to 60TB of
> hdfs data, plus the WALs. That's a lot for a single machine.
>
> On Mon, Dec 17, 2012 at 9:56 AM, tgh <[EMAIL PROTECTED]> wrote:
>
> > is it?
> > And then if I use 500 region in ONE server, and one region
> >
>
>
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Here's some back of the envelope math:
Say you have 6 1T drives per machines. That gives you about 2T of usable space (considering HDFS 3-way replication).
A reasonable max size for regions is 20gb. That's 100 regions for 2T.
If you set the flushsize to 128mb, you'd need ~13gb RAM in the worst case, just for the memstores. (Not all of them will be full all the time, though.)
Then you also want a block cache, plus the normal amount of memory just needed to run HBase.

You'll reach the JVM's reasonable memory limit pretty quickly, in the case above you'd probably want a 24gb JVM heap at least.
I find that current JVM technology 6T per machines is about as much as you can reasonably do with HBase.
As Andy said, you should upgrade, in HBase < 0.92 the max region size is somewhere around 4gb.

-- Lars

________________________________
 From: Andrew Purtell <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc: tgh <[EMAIL PROTECTED]>
Sent: Monday, December 17, 2012 11:26 AM
Subject: Re: 答复: 答复: what is the max size for one region and what is the max size of region for one server
 
Don't use HBase 0.90. Our current release is 0.94. You will find the
community is able to help you much more satisfactorily if you start with
the current release.
On Mon, Dec 17, 2012 at 2:26 AM, tgh <[EMAIL PROTECTED]> wrote:

> Thank you for your reply ,
>
> but I write for I want to make sure, if the number of region in ONE server
> exceed 300 or 500, the hbase will fail or something, or what is the max
> number of region for ONE server?
> And I use hbase 0.90,
>
> Could you help me
>
>
> Thank you
> ------------------------
> Tian Guanhua
>
>
>
>
>
>
>
>
>
> -----邮件原件-----
> 发件人: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]] 代表
> Nicolas Liochon
> 发送时间: 2012年12月17日 17:42
> 收件人: [EMAIL PROTECTED]
> 主题: Re: 答复: what is the max size for one region and what is the max size
> of region for one server
>
> You're reading correctly. It's a little bit extreme however. More extreme
> than necessary imho: if you have 20TB of HBase data, this leads to 60TB of
> hdfs data, plus the WALs. That's a lot for a single machine.
>
> On Mon, Dec 17, 2012 at 9:56 AM, tgh <[EMAIL PROTECTED]> wrote:
>
> > is it?
> > And then if I use 500 region in ONE server, and one region
> >
>
>
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)