Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in trunk/0.95?


Copy link to this message
-
Re: [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in trunk/0.95?
+1 on experimenting with 0.

On Mon, Jun 24, 2013 at 9:32 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> I think it depends on how large we expect the initially flushed HFile to
> be (just to state the obvious).
> The current default matches the memstore flushsize, so if we mostly flush
> because of that limit the current default should be good.
>
>
> If we have many column families, where one dominates, we want to decrease
> this to make sure that the smallest files - that are created because we
> need to flush all CFs - first.
> Not sure what a good default would be, or much we could auto configure
> this.
>
>
> On the other hand maybe setting this to a very small amount might be a
> good default after all. The larger files will eventually be collected by
> the ratio based selection, and having this small will immediately pick
> abnormally tiny HFiles for compaction.
>
> A good test might be to set this to 0 (so it's never used for file
> selection) and then see how this effects selection in common workloads.
>
>
> We'll probably not find defaults that are right for every workload.
>
>
> -- Lars
>
> ________________________________
> From: Stack <[EMAIL PROTECTED]>
> To: HBase Dev List <[EMAIL PROTECTED]>
> Sent: Monday, June 24, 2013 8:59 AM
> Subject: Re: [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in
> trunk/0.95?
>
>
> On Thu, Jun 20, 2013 at 3:43 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Jun 20, 2013 at 2:41 PM, Sergey Shelukhin <
> [EMAIL PROTECTED]>wrote:
> >
> >> Part of HBASE-7055 patch that we picked includes CompactionConfiguration
> >> class, which uses a prefix for config values.
> >> See ::getMinCompactSize on that class, it's still used in compaction.
> >>
> >>
> > Thanks Sergey.  Found it.
> >
> > Now, should we do Nicolas's suggestion as a default; i.e. any file < 4MB
> > is always added to compaction set (where currently, IIUC, any file <
> > flushsize is  added to the compaction set)?
> >
> >
> Ping on above question.  Any compactor's have an opinion?
> Thanks,
> St.Ack
>