-Re: Bulk loading (and/or major compaction) causing OOM
Stack 2012-12-07, 21:48
On Fri, Dec 7, 2012 at 1:01 PM, Bryan Beaudreault
> We have a couple tables that had thousands of regions due to the size of
> the day in them. We recently changed them to have larger regions (nearly
> 4GB). We are trying to bulk load these in now, but every time we do our
> servers die with OOM.
You mean, you are reloading the data that once was in thousands of regions
instead into new regions of 4GB in size?
I'd be surprised if the actual bulk load brings on the OOME.
> The logs seem to show that there is always a major compaction happening
> when the OOM happens. This is among other normal usage from a variety of
> apps in our product, so the memstores, block cache, etc are all active
> during this time.
Could you turn off major compaction during the bulk load to see if that
> I was reading through the compaction code and it doesn't look like it
> should take up much memory (depending on how the Reader class works) .
Are there lots of storefiles under each region?
> Does anyone with more knowledge of these internals know how it bulk load
> and major compaction works with regard to memory?
> We are running on ec2 c1.xlarge servers with 5GB of heap, and on hbase
> version 0.90.4 (I know, I know, we're working to upgrade).
How much have you given hbase?
If you look at your cluster monitoring, are you swapping?
The regionservers are carrying how many regions per server?