Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in trunk/0.95?


Copy link to this message
-
Re: [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in trunk/0.95?
+1 on experimenting with 0.

On Mon, Jun 24, 2013 at 9:32 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> I think it depends on how large we expect the initially flushed HFile to
> be (just to state the obvious).
> The current default matches the memstore flushsize, so if we mostly flush
> because of that limit the current default should be good.
>
>
> If we have many column families, where one dominates, we want to decrease
> this to make sure that the smallest files - that are created because we
> need to flush all CFs - first.
> Not sure what a good default would be, or much we could auto configure
> this.
>
>
> On the other hand maybe setting this to a very small amount might be a
> good default after all. The larger files will eventually be collected by
> the ratio based selection, and having this small will immediately pick
> abnormally tiny HFiles for compaction.
>
> A good test might be to set this to 0 (so it's never used for file
> selection) and then see how this effects selection in common workloads.
>
>
> We'll probably not find defaults that are right for every workload.
>
>
> -- Lars
>
> ________________________________
> From: Stack <[EMAIL PROTECTED]>
> To: HBase Dev List <[EMAIL PROTECTED]>
> Sent: Monday, June 24, 2013 8:59 AM
> Subject: Re: [COMPACTIONS] Anyone seen hbase.hstore.compaction.min.size in
> trunk/0.95?
>
>
> On Thu, Jun 20, 2013 at 3:43 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Jun 20, 2013 at 2:41 PM, Sergey Shelukhin <
> [EMAIL PROTECTED]>wrote:
> >
> >> Part of HBASE-7055 patch that we picked includes CompactionConfiguration
> >> class, which uses a prefix for config values.
> >> See ::getMinCompactSize on that class, it's still used in compaction.
> >>
> >>
> > Thanks Sergey.  Found it.
> >
> > Now, should we do Nicolas's suggestion as a default; i.e. any file < 4MB
> > is always added to compaction set (where currently, IIUC, any file <
> > flushsize is  added to the compaction set)?
> >
> >
> Ping on above question.  Any compactor's have an opinion?
> Thanks,
> St.Ack
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB