Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Re: 答复: 答复: what is the max size for one region and what is the max size of region for one server


+
Nicolas Liochon 2012-12-17, 11:07
+
Doug Meil 2012-12-17, 14:31
+
Bryan Beaudreault 2012-12-17, 14:39
+
Nicolas Liochon 2012-12-17, 09:42
+
tgh 2012-12-17, 10:26
+
Nicolas Liochon 2012-12-17, 08:27
+
tgh 2012-12-17, 08:56
+
tgh 2012-12-17, 08:11
+
Andrew Purtell 2012-12-17, 19:26
Copy link to this message
-
Re: 答复: 答复: what is the max size for one region and what is the max size of region for one server
Here's some back of the envelope math:
Say you have 6 1T drives per machines. That gives you about 2T of usable space (considering HDFS 3-way replication).
A reasonable max size for regions is 20gb. That's 100 regions for 2T.
If you set the flushsize to 128mb, you'd need ~13gb RAM in the worst case, just for the memstores. (Not all of them will be full all the time, though.)
Then you also want a block cache, plus the normal amount of memory just needed to run HBase.

You'll reach the JVM's reasonable memory limit pretty quickly, in the case above you'd probably want a 24gb JVM heap at least.
I find that current JVM technology 6T per machines is about as much as you can reasonably do with HBase.
As Andy said, you should upgrade, in HBase < 0.92 the max region size is somewhere around 4gb.

-- Lars

________________________________
 From: Andrew Purtell <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc: tgh <[EMAIL PROTECTED]>
Sent: Monday, December 17, 2012 11:26 AM
Subject: Re: 答复: 答复: what is the max size for one region and what is the max size of region for one server
 
Don't use HBase 0.90. Our current release is 0.94. You will find the
community is able to help you much more satisfactorily if you start with
the current release.
On Mon, Dec 17, 2012 at 2:26 AM, tgh <[EMAIL PROTECTED]> wrote:

> Thank you for your reply ,
>
> but I write for I want to make sure, if the number of region in ONE server
> exceed 300 or 500, the hbase will fail or something, or what is the max
> number of region for ONE server?
> And I use hbase 0.90,
>
> Could you help me
>
>
> Thank you
> ------------------------
> Tian Guanhua
>
>
>
>
>
>
>
>
>
> -----邮件原件-----
> 发件人: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]] 代表
> Nicolas Liochon
> 发送时间: 2012年12月17日 17:42
> 收件人: [EMAIL PROTECTED]
> 主题: Re: 答复: what is the max size for one region and what is the max size
> of region for one server
>
> You're reading correctly. It's a little bit extreme however. More extreme
> than necessary imho: if you have 20TB of HBase data, this leads to 60TB of
> hdfs data, plus the WALs. That's a lot for a single machine.
>
> On Mon, Dec 17, 2012 at 9:56 AM, tgh <[EMAIL PROTECTED]> wrote:
>
> > is it?
> > And then if I use 500 region in ONE server, and one region
> >
>
>
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB