I increased the max region file size to 4gb so I should have fewer than 200 regions per node now, more like 25. With 2 column families that will be 50 memstores per node. 5.6gb would then flush files of 112mb. Still not close to the memstore limit but shouldn't I be much better off than before?
Inserting sequentially may or may not be an option for me. I am storing a live feed of data from an external source so it could prove tricky.
On Feb 6, 2012, at 3:56 PM, Jean-Daniel Cryans wrote:
> Good but...
> Keep in mind that if you just increase max filesize and memstore size
> without changing anything else then you'll be in the same situation
> except with 16GB it'll take just a bit more time to get there.
> Here's the math:
> 200 regions of 2 families means 400 memstores to fill. Assuming a
> completely random pattern between all the regions and families, it
> means that you're going to fill 400 memstores at the same rate. With
> 4GB you hit the memstore lower barrier at 0.35*4=1.4GB at which point
> the regions have around 3.5MB each and the bigger one will flush.
> Currently we flush whole regions not just families so it would flush 2
> files of about 3.5MB. About 7MB later, another region will flush like
> that and so on and so forth.
> Now with 16GB you have 5.6GB which is a lot more room but still you
> would flush files that will be 14MB... but it's going to flush before
> that unfortunately. By default HBase will keep a maximum of 32
> write-ahead-logs (WAL) each of about 64MB which is almost 2GB total.
> Since your pattern is random, each log will contain rows from almost
> each region meaning that in order to get rid of the older logs in
> order to make room for newer ones it will have to force flush ALL your
> regions. And it's gonna happen again 2GB later.
> This is why I recommended that you try to insert sequentially into
> only a few regions at a time as this will play more nicely with the
> Note that you could set to have bigger WALs or more of them in order
> to match the lower barrier (you'd tweak hbase.regionserver.maxlogs and
> hbase.regionserver.hlog.blocksize) but it's still not as good as
> having a few regions or using less of them at the same time.
> On Mon, Feb 6, 2012 at 3:18 PM, Bryan Keller <[EMAIL PROTECTED]> wrote:
>> Yes, insert pattern is random, and yes, the compactions are going through the roof. Thanks for pointing me in that direction. I am going to try increasing the region max filesize to 4gb (it was set to 512mb) and the memstore flush size to 512mb (it was 128mb). I'm also going to increase the heap to 16gb (right now it is 4gb).
>> On Feb 6, 2012, at 1:33 PM, Jean-Daniel Cryans wrote:
>>> Ok this helps, we're still missing your insert pattern regarding but I
>>> bet it's pretty random considering what's happening to your cluster.
>>> I'm guessing you didn't set up metrics else you would have told us
>>> that the compaction queues are through the roof during the import, but
>>> at this point I'm pretty sure it's the case.
>>> To solve this your choices are:
>>> - Do bulk uploads instead of brute forcing it so that you would be
>>> entirely skipping those issues. See
>>> - Get that number of regions down to something more manageable; you
>>> didn't say how much memory you gave to HBase so I can't say how many
>>> exactly you need but it's usually never more than 20. Then set the
>>> memstore flush size and max file size accordingly. The goal here is to
>>> flush/compact as less as possible.
>>> - Keep your current setup, but slow down the insert rate so that data
>>> can be compacted over and over again without overrunning your region
>>> - Use a more sequential pattern so that you hit only a few regions at
>>> a time, this is like the second solution but trying to make it work
>>> with your current setup. This might not be practical for you as it
>>> really depends on how easily you can sort your data source.