-Re: Hbase heap size
Varun Sharma 2013-01-17, 23:24
Thanks for the info. I am looking for a balance where I have a write heavy
work load and need excellent read latency. So 40 % to block cache for
caching, 35 % to memstore.
But I would like to also reduce the number of HFiles and amount of
compaction activity. So, having few number of regions and much larger
memstore flush size - like 640M. Could a large memstore flush be a problem
in some sense ? Are updates blocked on memstore flush ? In my case, I would
expect a 600M sized memstore to materialize into a 200-300M sized HFile.
On Thu, Jan 17, 2013 at 2:31 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> A good rule of thumb that I found is to give each region server a Java
> help that is roughly 1/100th of the size of the disk space per region
> (that is assuming all the default setting: 10G regions, 128M memstores,
> 40% of heap for memstores, 20% of heap for block cache, 3-way replication)
> That is, if you give the region server a 10G heap, you can expect to be
> able to serve about 1T worth of disk space.
> That can be tweaked of course (increase the region size to 20G, if your
> load is mostly readonly you shrink the memstores, etc).
> That way you can reduce that ratio to 1/200 or even less.
> I'm sure other folks will have more detailed input.
> -- Lars
> From: Varun Sharma <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thursday, January 17, 2013 1:15 PM
> Subject: Hbase heap size
> I was wondering how much folks typical give to hbase and how much they
> leave for the file system cache for the region server. I am using hbase
> 0.94 and running only the region server and data node daemons. I have a
> system with 15G ram.