Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> major compaction and regionserver failure...


Copy link to this message
-
Re: major compaction and regionserver failure...
Andy,

Some detail on the current compaction algorithm.  Major compactions can be
triggered in 3 ways:

1) User requested : e.g. From the shell
2) Size based : compact when file <= sum(smaller_files) *
'hbase.hstore.compaction.ratio'

This is the normal minor compaction logic.  It will upgrade to a major
compaction if all files in the Store meet this criteria.

3) Time delayed : after (now - min(StoreFile.timestamp)) >
"hbase.hregion.majorcompaction" + rand() *
hbase.hregion.majorcompaction.jitter"

That's the config you're currently toggling.  Note that
"hbase.hregion.majorcompaction" is a relative toggle, not an absolute
toggle.  It's the relative time between major compactions.  Note that this
time will reset if a user-requested or size-based compaction is issued.
Also, jitter is enabled by default so that major compaction storms don't
occur.  

There is currently no way to control compaction on an absolute time (e.g.
every Sunday @ midnight).  Although this feature could be added, the
addition of multi-threaded compactions in 0.92 should eliminate the need
for this feature, because the major compaction queue will no longer slow
down the minor compaction queue.
On 12/11/11 11:40 AM, "Andy Sautins" <[EMAIL PROTECTED]> wrote:

>
>  Thanks Mikhail.  That's very helpful information.
>
>   We followed some of the advice that I believe was posted to this list
>in the past to schedule major compactions manually to run during off-peak
>times.  That has gone very well for us, except when failures happen it
>seems we end up not major compacting some regions.  We currently have
>hbase.hregion.majorcompaction set to 0 to disable automated major
>compactions.  I'm wondering if I manually run major compactions every 'x'
>days if I set hbase.hregion.majorcompaction to 'x+m' where m is some
>number of days after the number of scheduled compactions then ideally the
>automated major compaction would just catch any regions that didn't get
>compacted normally.  For example, if I want to major compact every
>Saturday then I could set hbase.hregion.majorcompaction to 9 days
>(777600000 milliseconds ) so if for some reason the weekly Saturday major
>compaction didn't succeed for a subset of the regions it would be caught
>by the automated compaction 2 days later.  I would assume that if the
>manually scheduled major compactions completed successfully the automated
>major compactions wouldn't kick in because not enough time had passed.
>
>    Curious if you think that is a reasonably way to manually schedule
>major compactions but also have a catch-all in case of regionserver
>failures.
>
>    Thanks again for your input.  Very helpful.
>
>-----Original Message-----
>From: Mikhail Bautin [mailto:[EMAIL PROTECTED]]
>Sent: Sunday, December 11, 2011 3:10 AM
>To: [EMAIL PROTECTED]
>Subject: Re: major compaction and regionserver failure...
>
>Hi Andy,
>
>We don't persist manual compaction requests across server restarts, so if
>e.g. a major compaction was requested manually and the regionserver
>crashed, the new regionserver that picks up that region will not know
>about the major compaction request. For automatic compactions, however,
>compaction criteria are the same on the old and the new region server, so
>we would expect the new server to figure out that a compaction is
>necessary and start it again from the beginning if the old regionserver
>died in the middle of a compaction.
>
>Thanks,
>--Mikhail
>
>On Sat, Dec 10, 2011 at 2:25 PM, Andy Sautins
><[EMAIL PROTECTED]>wrote:
>
>>
>>  Mikhail,
>>
>>  Thanks for the response.  So to summarize compaction requests in the
>> compaction queue of the failed regionserver are lost and not picked up
>> when the regions are picked up by another regionserver.  So if we lose
>> a regionserver during a major compaction the regions that had not yet
>> been major compacted on the failed regionserver will not be major
>> compacted as part of that major compaction request, correct?