Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space


+
Mesika, Asaf 2013-02-12, 05:46
+
lars hofhansl 2013-02-12, 05:50
+
Ted Yu 2013-02-12, 05:53
Copy link to this message
-
Re: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
Mesika, Asaf 2013-02-12, 07:22
I will, but I don't think its correlated since this issue was referring to PermGen.

On Feb 12, 2013, at 7:53 AM, Ted Yu wrote:

> It would be nice if you can try your improved code using 0.94.4 where
> HBASE-7205 <https://issues.apache.org/jira/browse/HBASE-7205> is included.
>
> Cheers
>
> On Mon, Feb 11, 2013 at 9:46 PM, Mesika, Asaf <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I wrote a RegionObserver which does preCompact.
>> I activated in pre-production, and then entire cluster dropped dead: One
>> RegionServer after another crashed on OutOfMemoryException: Heap Space.
>>
>> My preCompact method generates a KeyValue per each set of Column
>> Qualifiers it sees.
>> When I remove the coprocessor and restart the cluster, cluster remains
>> stable.
>> I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific
>> table I'm working on) per Region Server.
>> Running HBase 0.94.3
>>
>> The crash occur when the major compaction fires up, apparently cluster
>> wide.
>>
>>
>> My question is this: Where do the KV generated during the compaction
>> process queue up before being written to the disk? Is this buffer
>> configurable?
>> When I wrote the Region Observer my assumption was the the compaction
>> process works in Streaming fashion, thus even if I decide to generate a KV
>> per KV I see, it still shouldn't be a problem memory wise.
>>
>> Of course I'm trying to improve my code so it will generate much less new
>> KV (by simply altering the existing KVs received from the InternalScanner).
>>
>> Thank you,
>>
>> Asaf
+
Anoop Sam John 2013-02-12, 06:44
+
Mesika, Asaf 2013-02-12, 09:30
+
Mesika, Asaf 2013-02-12, 09:13
+
Anoop Sam John 2013-02-12, 09:22
+
Mesika, Asaf 2013-02-12, 14:41
+
James Taylor 2013-02-12, 15:32
+
Ted Yu 2013-02-12, 16:00
+
Anoop Sam John 2013-02-13, 05:29