Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Bulk loading (and/or major compaction) causing OOM


+
Bryan Beaudreault 2012-12-07, 21:01
+
Stack 2012-12-07, 21:48
+
Bryan Beaudreault 2012-12-08, 16:50
+
Marcos Ortiz 2012-12-08, 20:42
+
Bryan Beaudreault 2012-12-08, 21:08
Copy link to this message
-
Re: Bulk loading (and/or major compaction) causing OOM

On 12/07/2012 04:01 PM, Bryan Beaudreault wrote:
> We have a couple tables that had thousands of regions due to the size of
> the day in them.  We recently changed them to have larger regions (nearly
> 4GB).  We are trying to bulk load these in now, but every time we do our
> servers die with OOM.
When you are doing the bulk load, are you pre-split your regions?
What OS are you using and what version of Java?
>
> The logs seem to show that there is always a major compaction happening
> when the OOM happens.  This is among other normal usage from a variety of
> apps in our product, so the memstores, block cache, etc are all active
> during this time.
There are a good number of improvements in the new releases respect to
compactions.
>
> I was reading through the compaction code and it doesn't look like it
> should take up much memory (depending on how the Reader class works) .
>   Does anyone with more knowledge of these internals know how it bulk load
> and major compaction works with regard to memory?
>
> We are running on ec2 c1.xlarge servers with 5GB of heap, and on hbase
> version 0.90.4 (I know, I know, we're working to upgrade).
Yes, my friend. You should know all the benefits in the new stable
release (0.94.3), so
this is the first advice.
>
> Thanks.
>
>
>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB