Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - On source code of File Section mechanism in minor compaction


+
yun peng 2013-04-11, 14:52
+
Ted Yu 2013-04-11, 15:00
+
yun peng 2013-04-11, 15:04
+
Ted Yu 2013-04-11, 16:12
Copy link to this message
-
Re: On source code of File Section mechanism in minor compaction
yun peng 2013-04-11, 19:24
Hi Ted, thanks a lot for the reference.

Ted and All, I have additional minor question though. Would a flush in
HBase cause cascaded minor compaction? To be more specific, when a minor
compaction finishes, the newly generated HFile may make the condition of
minor compaction met again, in this case, would HBase trigger another minor
compaction?

For example, say we have following files.

HFile  index  1, 2, 3, 4, 5, 6
HFile size     1, 1, 3, 3, 9, 9

with hbase.hstore.compactionThreshold=3 (i.e., at least 3 files to merge
each time) and ratio is 1, then a flush into such file storage initially
generates a new HFile of size 1 (the file is indexed by number 0):

HFile  index  0, 1, 2, 3, 4, 5, 6
HFile size     1, 1, 1, 3, 3, 9, 9

The compaction condition is met for HFile 0, 1, 2 and a minor compaction is
trigger to merge file 1 and 2 and 0 to a new file of size 3, as follows,

HFile  index      2', 3, 4, 5, 6
HFile size         3, 3, 3, 9, 9

Then the condition for minor compaction met again for HFile 2',3,4, would
this be triggered and executed immediately or to say would compaction be
cascaded in HBase?

Regards,
Yun
On Thu, Apr 11, 2013 at 12:12 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> For 0.94, take a look at these files:
>
> 94-hbase tyu$ ls
> src/main/java/org/apache/hadoop/hbase/regionserver/compactions
> CompactSelection.java CompactionProgress.java CompactionRequest.java
>
> You can see the revision history for these files. I suggest you start with
> recent JIRAs that touched them.
>
> Cheers
>
> On Thu, Apr 11, 2013 at 8:04 AM, yun peng <[EMAIL PROTECTED]> wrote:
>
> > Sorry to not mention the version. I am looking at version 0.94.2 which
> > seems file RatioBasedCompactionPolicy is not yet in.
> > regards,
> > Yun
> >
> >
> > On Thu, Apr 11, 2013 at 11:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Which release are you targeting ?
> > >
> > > In trunk, take a look at RatioBasedCompactionPolicy which is the
> default:
> > >
> > > public class RatioBasedCompactionPolicy extends CompactionPolicy {
> > >
> > > Cheers
> > >
> > > On Thu, Apr 11, 2013 at 7:52 AM, yun peng <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi, All
> > > > I am asking the source code showing the file selection mechanism for
> > > minor
> > > > compaction in HBase. I am aware that HBase minor compaction selects
> > file
> > > > based on size and generation time. But I can not find source file
> that
> > > > realizes this mechanism in HBase. I am interested in the algorithm
> > works
> > > > behind it. Kindly point to me the source code and I appreciate it.
> > > Thanks.
> > > > Regards
> > > > Yun
> > > >
> > >
> >
>