Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space


+
Mesika, Asaf 2013-02-12, 05:46
+
lars hofhansl 2013-02-12, 05:50
+
Ted Yu 2013-02-12, 05:53
Copy link to this message
-
Re: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
I will, but I don't think its correlated since this issue was referring to PermGen.

On Feb 12, 2013, at 7:53 AM, Ted Yu wrote:

> It would be nice if you can try your improved code using 0.94.4 where
> HBASE-7205 <https://issues.apache.org/jira/browse/HBASE-7205> is included.
>
> Cheers
>
> On Mon, Feb 11, 2013 at 9:46 PM, Mesika, Asaf <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I wrote a RegionObserver which does preCompact.
>> I activated in pre-production, and then entire cluster dropped dead: One
>> RegionServer after another crashed on OutOfMemoryException: Heap Space.
>>
>> My preCompact method generates a KeyValue per each set of Column
>> Qualifiers it sees.
>> When I remove the coprocessor and restart the cluster, cluster remains
>> stable.
>> I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific
>> table I'm working on) per Region Server.
>> Running HBase 0.94.3
>>
>> The crash occur when the major compaction fires up, apparently cluster
>> wide.
>>
>>
>> My question is this: Where do the KV generated during the compaction
>> process queue up before being written to the disk? Is this buffer
>> configurable?
>> When I wrote the Region Observer my assumption was the the compaction
>> process works in Streaming fashion, thus even if I decide to generate a KV
>> per KV I see, it still shouldn't be a problem memory wise.
>>
>> Of course I'm trying to improve my code so it will generate much less new
>> KV (by simply altering the existing KVs received from the InternalScanner).
>>
>> Thank you,
>>
>> Asaf
+
Anoop Sam John 2013-02-12, 06:44
+
Mesika, Asaf 2013-02-12, 09:30
+
Mesika, Asaf 2013-02-12, 09:13
+
Anoop Sam John 2013-02-12, 09:22
+
Mesika, Asaf 2013-02-12, 14:41
+
James Taylor 2013-02-12, 15:32
+
Ted Yu 2013-02-12, 16:00
+
Anoop Sam John 2013-02-13, 05:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB