Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Bulk loading (and/or major compaction) causing OOM


+
Bryan Beaudreault 2012-12-07, 21:01
+
Stack 2012-12-07, 21:48
+
Bryan Beaudreault 2012-12-08, 16:50
+
Marcos Ortiz 2012-12-08, 20:42
+
Bryan Beaudreault 2012-12-08, 21:08
Copy link to this message
-
Re: Bulk loading (and/or major compaction) causing OOM

On 12/07/2012 04:01 PM, Bryan Beaudreault wrote:
> We have a couple tables that had thousands of regions due to the size of
> the day in them.  We recently changed them to have larger regions (nearly
> 4GB).  We are trying to bulk load these in now, but every time we do our
> servers die with OOM.
When you are doing the bulk load, are you pre-split your regions?
What OS are you using and what version of Java?
>
> The logs seem to show that there is always a major compaction happening
> when the OOM happens.  This is among other normal usage from a variety of
> apps in our product, so the memstores, block cache, etc are all active
> during this time.
There are a good number of improvements in the new releases respect to
compactions.
>
> I was reading through the compaction code and it doesn't look like it
> should take up much memory (depending on how the Reader class works) .
>   Does anyone with more knowledge of these internals know how it bulk load
> and major compaction works with regard to memory?
>
> We are running on ec2 c1.xlarge servers with 5GB of heap, and on hbase
> version 0.90.4 (I know, I know, we're working to upgrade).
Yes, my friend. You should know all the benefits in the new stable
release (0.94.3), so
this is the first advice.
>
> Thanks.
>
>
>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci