Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Major Compaction in 0.90.6


Copy link to this message
-
Re: Major Compaction in 0.90.6
Jean-Daniel Cryans 2013-08-20, 17:36
On Mon, Aug 19, 2013 at 11:52 PM, Monish r <[EMAIL PROTECTED]> wrote:

> Hi Jean,
>

s/Jean/Jean-Daniel ;)
> Thanks for the explanation.
>
> Just a clarification on the third answer,
>
> In our current cluster ( 0.90.6 ) , i find that irrespective of whether TTL
> is set or not , Major compaction compaction rewrites hfile for the region (
> there is only one hfile for that region )  on every manual major compaction
> trigger.
>
Can you enable DEBUG logs? You'd see why the major compaction is triggered.

>
> log :
>
> 2013-08-19 14:15:29,926 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 1 file(s), new
>
> file=hdfs://x.x.x.x:9000/hbase/NOTIFICATION_HISTORY/b00086bca62ee55796a960002291aca4/n/4754838096619480671
>
> i find a new file is created for every major compaction triggger.
>
> Regards,
> R.Monish
>
>
> On Mon, Aug 19, 2013 at 11:52 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
> >wrote:
>
> > Inline.
> >
> > J-D
> >
> >
> > On Mon, Aug 19, 2013 at 2:48 AM, Monish r <[EMAIL PROTECTED]> wrote:
> >
> > > Hi guys,
> > > I have the following questions in HBASE 0.90.6
> > >
> > > 1. Does hbase use only one compaction thread to handle both major and
> > minor
> > > compaction?
> > >
> >
> > Yes, look at CompactSplitThread
> >
> >
> > >
> > > 2. If hbase uses multiple compaction threads, which configuration
> > parameter
> > > defines the number of compaction threads?
> > >
> >
> > It doesn't in 0.90.6 but CompactSplitThread lists those for 0.92+
> >
> > hbase.regionserver.thread.compaction.large
> > hbase.regionserver.thread.compaction.small
> >
> >
> > >
> > > 3. After hbase.majorcompaction.interval from last major compaction ,if
> > > major compaction is executed on a table already major compacted Does
> > hbase
> > > skip all the table regions from major compaction?
> > >
> >
> > Determining if something is major-compacted is definitely not at the
> > table-level.
> >
> > In 0.90.6, MajorCompactionChecker will ask HRegion.isMajorCompaction() to
> > check if it needs to major compact again, which in turns checks every
> > Store. FWW if you have TTL turned on it will still major compact a major
> > compacted file, HFiles don't have an index of what's deleted or TTL'd and
> > it doesn't do a full read of each files to check.
> >
> >
> > >
> > > Regards,
> > > R.Monish
> > >
> >
>