Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Major Compaction in 0.90.6


Copy link to this message
-
Re: Major Compaction in 0.90.6
On Mon, Aug 19, 2013 at 11:52 PM, Monish r <[EMAIL PROTECTED]> wrote:

> Hi Jean,
>

s/Jean/Jean-Daniel ;)
> Thanks for the explanation.
>
> Just a clarification on the third answer,
>
> In our current cluster ( 0.90.6 ) , i find that irrespective of whether TTL
> is set or not , Major compaction compaction rewrites hfile for the region (
> there is only one hfile for that region )  on every manual major compaction
> trigger.
>
Can you enable DEBUG logs? You'd see why the major compaction is triggered.

>
> log :
>
> 2013-08-19 14:15:29,926 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 1 file(s), new
>
> file=hdfs://x.x.x.x:9000/hbase/NOTIFICATION_HISTORY/b00086bca62ee55796a960002291aca4/n/4754838096619480671
>
> i find a new file is created for every major compaction triggger.
>
> Regards,
> R.Monish
>
>
> On Mon, Aug 19, 2013 at 11:52 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
> >wrote:
>
> > Inline.
> >
> > J-D
> >
> >
> > On Mon, Aug 19, 2013 at 2:48 AM, Monish r <[EMAIL PROTECTED]> wrote:
> >
> > > Hi guys,
> > > I have the following questions in HBASE 0.90.6
> > >
> > > 1. Does hbase use only one compaction thread to handle both major and
> > minor
> > > compaction?
> > >
> >
> > Yes, look at CompactSplitThread
> >
> >
> > >
> > > 2. If hbase uses multiple compaction threads, which configuration
> > parameter
> > > defines the number of compaction threads?
> > >
> >
> > It doesn't in 0.90.6 but CompactSplitThread lists those for 0.92+
> >
> > hbase.regionserver.thread.compaction.large
> > hbase.regionserver.thread.compaction.small
> >
> >
> > >
> > > 3. After hbase.majorcompaction.interval from last major compaction ,if
> > > major compaction is executed on a table already major compacted Does
> > hbase
> > > skip all the table regions from major compaction?
> > >
> >
> > Determining if something is major-compacted is definitely not at the
> > table-level.
> >
> > In 0.90.6, MajorCompactionChecker will ask HRegion.isMajorCompaction() to
> > check if it needs to major compact again, which in turns checks every
> > Store. FWW if you have TTL turned on it will still major compact a major
> > compacted file, HFiles don't have an index of what's deleted or TTL'd and
> > it doesn't do a full read of each files to check.
> >
> >
> > >
> > > Regards,
> > > R.Monish
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB