-Re: MemStoreFlusher: region has too many store files (client timeout)
kzurek 2013-02-05, 14:52
Thanks, but I've already read that article and I was trying to set up
settings accordingly. But any type of help could be useful. Here are logs
(compressed with tar gzip): hbase_logs.tar.gz
Kevin O'dell wrote
> Okay, that is a fair flush size(I have not seen larger than 256MB). Do
> you think you could send me a RS log from yesterday while you were seeing
> this issue? I would just like to thumb through it and make some tuning
> recommendations. I have a pretty good idea of what you need to do, if you
> look at:
> http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html The
> article has some good ideas for write tuning.
> On Tue, Feb 5, 2013 at 3:39 AM, kzurek <
> > wrote:
>> Thanks for the reply, although I should clear some misunderstandings. In
>> general, I do know general behavior and difference between minor and
>> compaction, as well as when minor compaction might become (could be
>> promoted) to major compaction. I just wanted to verify influence of
>> compaction (mostly major) on our cluster performance. Thus, I've created
>> test where I'm putting data to only one region (total 71) by one single
>> threaded client using build in caching mechanism (according to "HBase The
>> Definitive Guide" book) and triggering major compaction by hand
>> (HBaseAdmin). Although, after few tests I've noticed that major
>> (Large Compaction) is being triggered (cache flusher, recursive queue) so
>> left it as it was (not triggering it anymore). That brought me to this
>> situation, where I'm putting data and after a while I'm getting timeouts
>> the client, in meanwhile I see that memstore is being flush which cant
>> create new store file (cause there are to many of them) and which is
>> frequently blocked by compaction process. I hope that this short
>> will bring closer look at the issue. In addition, here are some answers
>> your questions:
>> 1. How often are you flushing?
>> I'm not triggering flushing by hand, but I've noticed that data is being
>> flushed every 4s (275m) or 1m 30s-40s (1.5g).
>> 2. How often are you force flushing from HLog rolls?
>> Default settings are: blocksize=64 MB, rollsize=60.8 MB, enabled=true,
>> optionallogflushinternal=1000ms. It seems that roll is made every hour.
>> 3. What size are your flushes?
>> Depends, from 275m up to 1.5g. I've set my memstore flush size to 256m
>> memstore block multiplier to 6. Should I increase the flush size??
>> 4. What does your region count look like as that can affect your flush
>> Initial split is 37 regions on 6 RegionServers, but at the moment there
>> 71 regions.
>> Kevin O'dell wrote
>> > Kzurek,
>> > Just because you turn off time based major compactions, it does not
>> > that you have turned major compaction off. Compaction can still be
>> > promoted to be Majors. Also, the only real difference between a major
>> > minor compaction is one processes deletes. You should really schedule
>> > least daily major compactions. As for your blocking issue, there are
>> > quite
>> > a few things you would want to look at:
>> > How often are you flushing?
>> > How often are you force flushing from HLog rolls?
>> > What size are your flushes?
>> > What does your region count look like as that can affect your flush
>> > etc
>> > When I see HBase blocking constantly it is usually a sign that you need
>> > do some fine grain tuning.
>> > On Mon, Feb 4, 2013 at 7:45 AM, kzurek <
>> > kzurek@
>> > > wrote:
>> >> I'm facing some issues regarding to major compaction. I've disabled
>> >> compaction and it is not triggered manually, but when I'm loading data
>> >> selected region, I saw that major compaction queue is growing and it
>> >> being triggered ('Large Compaction' in logs) quite often (mainly due
View this message in context: http://apache-hbase.679495.n3.nabble.com/MemStoreFlusher-region-has-too-many-store-files-client-timeout-tp4037887p4037960.html
Sent from the HBase User mailing list archive at Nabble.com.