Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space

Copy link to this message
Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
Mesika, Asaf 2013-02-12, 05:46

I wrote a RegionObserver which does preCompact.
I activated in pre-production, and then entire cluster dropped dead: One RegionServer after another crashed on OutOfMemoryException: Heap Space.

My preCompact method generates a KeyValue per each set of Column Qualifiers it sees.
When I remove the coprocessor and restart the cluster, cluster remains stable.
I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific table I'm working on) per Region Server.
Running HBase 0.94.3

The crash occur when the major compaction fires up, apparently cluster wide.
My question is this: Where do the KV generated during the compaction process queue up before being written to the disk? Is this buffer configurable?
When I wrote the Region Observer my assumption was the the compaction process works in Streaming fashion, thus even if I decide to generate a KV per KV I see, it still shouldn't be a problem memory wise.

Of course I'm trying to improve my code so it will generate much less new KV (by simply altering the existing KVs received from the InternalScanner).

Thank you,

lars hofhansl 2013-02-12, 05:50
Ted Yu 2013-02-12, 05:53
Mesika, Asaf 2013-02-12, 07:22
Anoop Sam John 2013-02-12, 06:44
Mesika, Asaf 2013-02-12, 09:30
Mesika, Asaf 2013-02-12, 09:13
Anoop Sam John 2013-02-12, 09:22
Mesika, Asaf 2013-02-12, 14:41
James Taylor 2013-02-12, 15:32
Ted Yu 2013-02-12, 16:00
Anoop Sam John 2013-02-13, 05:29