Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Compaction problems


Copy link to this message
-
Re: Compaction problems
Nicolas Spiegelberg 2011-04-27, 23:42
Note that we have a compaction recursive enqueue patch on our internal
branches, but we wanted to give it some run time before contributing back
to make sure it was safe.  I'll port that to trunk.

On 4/27/11 9:55 AM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:

>That make sense, would you mind opening a jira?
>
>Thx,
>
>J-D
>
>On Tue, Apr 26, 2011 at 8:52 PM, Schubert Zhang <[EMAIL PROTECTED]> wrote:
>> After more test, a obvious issue/problem is, the complete of a minor
>> compaction does not check if current storefiles need more minor
>>compaction.
>>
>> I think this may be a bug or leak.
>>
>> Try this test:
>>
>> 1. Put many data to a region, then there is 30 storefiles accumulated,
>> because the backend compaction cannot catch up with the fast put.
>> (hbase.hstore.compactionThreshold=8, hbase.hstore.compaction.max=12)
>>
>> 2. Then stop put.
>>
>> 3. Then, these many 30 storefiles are still there for a long time, (no
>> automatic minor compaction)
>>
>> 4. Submit a compaction on this region, then, only 12 files are
>>compaction,
>> now, we have 19 storefiles. The minor compaction stopped.
>>
>> I think, when a minor compaction complete, it should check if the
>>number of
>> storefiles still many, if so, another minor compaction should start
>> continuiously.
>> Schubert Zhang
>> On Wed, Apr 27, 2011 at 9:32 AM, Schubert Zhang <[EMAIL PROTECTED]>
>>wrote:
>>
>>> I have a busy region, and there are 43 StoreFiles (>
>>> compactionThreshold=8) in this region.
>>> Now, I stopped the client and stopped put new data into it. I expect
>>> these StoreFiles to be compacted later.
>>>
>>> But, one day after, these 43 StoreFiles are still there.
>>>
>>> It seems the compaction does not work automatically.
>>> And I checked the code, it seems the compaction would only start after
>>> split....
>>>
>>>
>>>
>>