Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> flushing + compactions after config change


Copy link to this message
-
Re: 答复: flushing + compactions after config change
Viral,
     Basically when you increase the memstore flush size ( well ur aim
there is to reduce flushes and make data sit in memory for longer time) you
need to carefully consider the 2 things
1. What is the max heap and what is the % memory you have allocated max for
all the memstores in a RS. And see how many regions and CFs and whether the
allocated max memstore memory is enough. Basically you should not make
premature flushes because of global heap preassure. As you said this will
create many small HFiles and so compactions
2. Check on the max Hlogs setting. Becuase of too many hlogs also there can
be a forced flush which also can create small HFiles. The default value is
32. The HLOg rolling will happen when one HLog's size reaches 95% size of
ur HDFS block size. (This 95% is also configurable)  If you have increased
the flush size and heap allocated for memstores, still you can get
premature flushes because of this HLog stuff.

Also from 94 onwards there is  compression feature for HLog..  Have a look
at that also.
-Anoop-
On Fri, Jun 28, 2013 at 6:52 AM, Viral Bajaria <[EMAIL PROTECTED]>wrote:

> Thanks for the detailed JVM OPTS. On a side note, I doubt the issue with
> too many hlogs has anything to do with that right ? Because I have enough
> spare heap space and my Memstore's are not at the lowerLimit of the total
> available heap (i.e. 0.35)
>
> -Viral
>
> On Thu, Jun 27, 2013 at 6:09 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
> > Hope that helpful.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB