Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Compaction time


+
samar.opensource 2013-03-04, 06:44
+
Viral Bajaria 2013-03-04, 06:56
+
samar.opensource 2013-03-04, 07:57
Copy link to this message
-
Re: Compaction time
Jean-Marc Spaggiari 2013-03-04, 12:50
So can you simply do something like disabling automatic compactions,
and run small manual one after each light job? That way it might not
impact you heavy job with a big 40 minutes compactions since data will
almost alway be compacted correctly? You can even maybe keep the
automatic compaction to on if you do that since when you heavy job
will run, your data compaction will be almost totally done.

JM

2013/3/4 samar.opensource <[EMAIL PROTECTED]>:
> Hi Viral,
>   The jobs dont run often. may be few times a day(10 ). But the problem is
> there are other application which are running other jobs which may not be as
> heavy but running compaction any time might block those jobs.
> so there is
> heavy jobs running less frequently
> and light jobs running more frequently
>
> Regards,
> Samar
>
> On 04/03/13 12:26 PM, Viral Bajaria wrote:
>>
>> How often do you run those jobs ? Do they run periodically or are they
>> running all the time ?
>>
>> If you have a predictable periodic behavior, you could disable automatic
>> compaction and trigger it manually using a cron job (not the recommended
>> approach, AFAIK). Or you could set the compaction to trigger at a set time
>> of the day when you know your jobs are not running.
>>
>> -Viral
>>
>> On Sun, Mar 3, 2013 at 10:44 PM, samar.opensource <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>     We are running some high load jobs which are mostly writes. During
>>> these jobs, compaction is triggered which takes sometime as longs as
>>> 40mins
>>> to complete. This causes blocking (as others wait for compaction in the
>>> queue). Please suggest how much compaction time is reasonable for
>>> compacting 2Gb store files . And best way to avoid long blocking
>>> compactions.
>>> Using Cloudera hbase vesion 3u3.
>>> Regards,
>>> Samar
>>>
>
+
samar.opensource 2013-03-04, 13:40