Hi

Before filling a jira, I  would like to check this with you.
It seems that the major compaction behavior changed between 0.90.x and 0.92
(not taking about all the nice stuff you added there...)

Effectively when i trigger a major compaction (via hbase shell or ui) when
other compactions (minor) are running (because i had a lot of storefiles
and a big compactionQueueSize), the major compaction is not executed. While
in 0.90.X major_compact command was executed immediately after any current
compaction.
Mikael.S

Can you show us some logs that exhibit this behavior?

Thx,

J-D

On Thu, Feb 23, 2012 at 2:58 PM, Mikael Sitruk <[EMAIL PROTECTED]> wrote:
> Hi
>
> Before filling a jira, I  would like to check this with you.
> It seems that the major compaction behavior changed between 0.90.x and 0.92
> (not taking about all the nice stuff you added there...)
>
> Effectively when i trigger a major compaction (via hbase shell or ui) when
> other compactions (minor) are running (because i had a lot of storefiles
> and a big compactionQueueSize), the major compaction is not executed. While
> in 0.90.X major_compact command was executed immediately after any current
> compaction.
>
>
> Mikael.S

I triggered several time major_compaction between 22:30 and the time i
wrote the original email 00:58
>From the log of one of the severs between those time no major compaction
was running a major_compaction ended at 22:23 on the same server but this
is because the number of file was high (after a load phase) and not
manually triggered.
On the system major_compaction are disabled.

>From the logs:
1. 22:23:42 --> 22:23:48 - single major compaction runs
2. 22:23:48 --> 23:44:27 - minor compaction runs
3. 23:44:27 --> 23:44:32 - single major compaction runs
5. 23:44:32 --> 01:13:49 - minor compaction are run
5. 01:13:49 the cycle of major compaction began to run (heavily working
only from 01:40:).
-------------------------------
*below a grep of compaction (ends, with major and minor to illustrate the
above)*

2012-02-23 22:23:48,272 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 1 file(s) in PI of
TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d82840e84eb870a6.
into 7400a662b5924d3baf3ac520b1eb7ad9, size=49.0m; total size for store is
49.0m
2012-02-23 22:25:38,477 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
into 77c43127e38846f78e7ada302d399732, size=540.8m; total size for store is
2.1g
2012-02-23 22:27:41,930 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000003951999,1329399349764.f55cfda6067ad1425ba68d20d82c945b.
into f9c31ac78b3d4feb848b1e8d78975e34, size=532.0m; total size for store is
2.1g
2012-02-23 22:29:43,260 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010971999,1329399349802.15720106584fe32f58287b6174970f31.
into af2472f30a9f4c558323a3268f98183c, size=517.9m; total size for store is
2.6g
2012-02-23 22:30:19,476 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731.
into 3ac255c439ab4c518f4a938d542c9220, size=335.4m; total size for store is
2.1g
2012-02-23 22:31:15,051 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000008501999,1329399349789.fa0ac64d8ac9050f4acf2b0deb6e2dc7.
into b8c733a27d02468a82f8a2c4d84db4ff, size=523.0m; total size for store is
2.1g
2012-02-23 22:32:33,430 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000004991999,1329399349770.46a2a16d484faacfcda36bc52bb218ba.
into 5344911081424de69299128412d41573, size=538.0m; total size for store is
2.2g
2012-02-23 22:33:28,882 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000004081999,1329399349765.a5ca0a6a0a5561af5b6964e458615493.
into d15c1e5b089d46beb2c4d01c796367a7, size=520.7m; total size for store is
2.1g
2012-02-23 22:34:46,714 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000007591999,1329399349784.711b032969e1ae6cce8e16bb9b
...
..
..
2012-02-23 23:44:32,215 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 1 file(s) in PI of
TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206.
into 2c701bd613314198889d8b3d79b82992, size=40.8m; total size for store is
40.8m
2012-02-23 23:45:00,838 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000005251999,1329399349771.50df7d847b5e36be9982b47eb3234ad4.
into 701d49ca010a41169ca495917b4ce3d4, size=1.3g; total size for store is
2.1g
2012-02-23 23:47:12,191 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000002261999,1329399349751.ffb8f2f3d9d5a23aee280359793557ae.
into d2b8336a36b1451d9a7d38c6bf3e79ee, size=1.2g; total size for store is
2.1g
2012-02-23 23:49:28,440 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000011881999,1329399349806.81f2bb0de10974b617521b62b9b85f09.
into 19208d50fe8543a682e3351c60801eb9, size=1.2g; total size for store is
2.1g
2012-02-23 23:52:12,523 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206.
into 639ffb3d8c434363aac0d007b6f78ce9, size=1.5g; total size for store is
2.1g
2012-02-23 23:56:14,058 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000005901999,1329399349775.48aa14a1feea4793ec68bfe076570257.
into 3e2f293a33f14612b8fa1ad570799feb, size=1.6g; total size for store is
2.1g
2012-02-23 23:59:08,211 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
into dd7f92dd786d4e3ea9cb733cb3a6d2f8, size=1.6g; total size for store is
2.1g
...
...
...
...
2012-02-24 01:17:15,611 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 6 file(s) in EVENT of
TC,00000000000012401999,1329399349808.03a3d34ac2c0c7d09619
8032a151120f. into eff7b5bda7bf4653a9f229c1b40e0e4f, size=1.3g; total size
for store is 2.1g
2012-02-24 01:17:51,057 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000000441999,1329399349737.02354040d2650db1d59de0e01b
0edd1d. into ac1d5e5f1ab746229bad223f56534054, size=329.8m; total size for
store is 2.2g
2012-02-24 01:20:40,454 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in EVENT of
TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d828
40e84eb870a6. into ae82b7832eed4bafb9e64ed45574831c, size=978.9m; total
size for store is 2.5g
2012-02-24 01:21:13,635 INFO org.apache.hadoop.hbase.regionserver.St

In 0.92, we can handle multiple compactions but we wait on major compact
if minor compactions are ongoing for the same store.  It should be
enqueued and work after the minor compaction finishes, it's just not
immediately enqueued.  See HBASE-5330 for some discussion on this.  There
should be a log message in the regionserver to verify the the
MAJOR_COMPACT request was received#16;.  The existing contract on major
compaction requests are that they are asynchronous & the request has
in-memory persistence.

On 2/24/12 4:32 AM, "Mikael Sitruk" <[EMAIL PROTECTED]> wrote:

>I triggered several time major_compaction between 22:30 and the time i
>wrote the original email 00:58
>From the log of one of the severs between those time no major compaction
>was running a major_compaction ended at 22:23 on the same server but this
>is because the number of file was high (after a load phase) and not
>manually triggered.
>On the system major_compaction are disabled.
>
>From the logs:
>1. 22:23:42 --> 22:23:48 - single major compaction runs
>2. 22:23:48 --> 23:44:27 - minor compaction runs
>3. 23:44:27 --> 23:44:32 - single major compaction runs
>5. 23:44:32 --> 01:13:49 - minor compaction are run
>5. 01:13:49 the cycle of major compaction began to run (heavily working
>only from 01:40:).
>-------------------------------
>*below a grep of compaction (ends, with major and minor to illustrate the
>above)*
>
>2012-02-23 22:23:48,272 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed major compaction of 1 file(s) in PI of
>TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d82840e84eb870a6.
>into 7400a662b5924d3baf3ac520b1eb7ad9, size=49.0m; total size for store is
>49.0m
>2012-02-23 22:25:38,477 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
>into 77c43127e38846f78e7ada302d399732, size=540.8m; total size for store
>is
>2.1g
>2012-02-23 22:27:41,930 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000003951999,1329399349764.f55cfda6067ad1425ba68d20d82c945b.
>into f9c31ac78b3d4feb848b1e8d78975e34, size=532.0m; total size for store
>is
>2.1g
>2012-02-23 22:29:43,260 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000010971999,1329399349802.15720106584fe32f58287b6174970f31.
>into af2472f30a9f4c558323a3268f98183c, size=517.9m; total size for store
>is
>2.6g
>2012-02-23 22:30:19,476 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731.
>into 3ac255c439ab4c518f4a938d542c9220, size=335.4m; total size for store
>is
>2.1g
>2012-02-23 22:31:15,051 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000008501999,1329399349789.fa0ac64d8ac9050f4acf2b0deb6e2dc7.
>into b8c733a27d02468a82f8a2c4d84db4ff, size=523.0m; total size for store
>is
>2.1g
>2012-02-23 22:32:33,430 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000004991999,1329399349770.46a2a16d484faacfcda36bc52bb218ba.
>into 5344911081424de69299128412d41573, size=538.0m; total size for store
>is
>2.2g
>2012-02-23 22:33:28,882 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000004081999,1329399349765.a5ca0a6a0a5561af5b6964e458615493.
>into d15c1e5b089d46beb2c4d01c796367a7, size=520.7m; total size for store
>is
>2.1g
>2012-02-23 22:34:46,714 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed compaction of 3 file(s) in EVENT of
>TC,00000000000007591999,1329399349784.711b032969e1ae6cce8e16bb9b
>...
>..
>..
>2012-02-23 23:44:32,215 INFO org.apache.hadoop.hbase.regionserver.Store:
>Completed major compaction of 1 file(s) in PI of
>TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206.

Hi Nicolas,

Well
1. I don't see such message, on the second hand I'm not using "debug"
severity in the log (unless this is not a debug message)
2. If major compaction are just entering the compaction queue it seems to
me quite problematic; especially when you want to control the impact of
compaction!
I had already a big queue of compaction request (minor) due to an heavy
load, so it waited at least 2 hours between the time I triggered the major
compaction till it begun the cycle.
Such behavior doesn't seem to be adequate for systems that have restricted
window time frame for internal task, don't you think? (because if you are
in the burst of minor compaction you will may miss your window)
Should the major compaction request remove all the minor compaction queued
and be the first?

Mikael.S

On Fri, Feb 24, 2012 at 6:36 PM, Nicolas Spiegelberg <[EMAIL PROTECTED]>wrote:

> In 0.92, we can handle multiple compactions but we wait on major compact
> if minor compactions are ongoing for the same store.  It should be
> enqueued and work after the minor compaction finishes, it's just not
> immediately enqueued.  See HBASE-5330 for some discussion on this.  There
> should be a log message in the regionserver to verify the the
> MAJOR_COMPACT request was received .  The existing contract on major
> compaction requests are that they are asynchronous & the request has
> in-memory persistence.
>
> On 2/24/12 4:32 AM, "Mikael Sitruk" <[EMAIL PROTECTED]> wrote:
>
> >I triggered several time major_compaction between 22:30 and the time i
> >wrote the original email 00:58
> >From the log of one of the severs between those time no major compaction
> >was running a major_compaction ended at 22:23 on the same server but this
> >is because the number of file was high (after a load phase) and not
> >manually triggered.
> >On the system major_compaction are disabled.
> >
> >From the logs:
> >1. 22:23:42 --> 22:23:48 - single major compaction runs
> >2. 22:23:48 --> 23:44:27 - minor compaction runs
> >3. 23:44:27 --> 23:44:32 - single major compaction runs
> >5. 23:44:32 --> 01:13:49 - minor compaction are run
> >5. 01:13:49 the cycle of major compaction began to run (heavily working
> >only from 01:40:).
> >-------------------------------
> >*below a grep of compaction (ends, with major and minor to illustrate the
> >above)*
> >
> >2012-02-23 22:23:48,272 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed major compaction of 1 file(s) in PI of
> >TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d82840e84eb870a6.
> >into 7400a662b5924d3baf3ac520b1eb7ad9, size=49.0m; total size for store is
> >49.0m
> >2012-02-23 22:25:38,477 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed compaction of 3 file(s) in EVENT of
> >TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
> >into 77c43127e38846f78e7ada302d399732, size=540.8m; total size for store
> >is
> >2.1g
> >2012-02-23 22:27:41,930 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed compaction of 3 file(s) in EVENT of
> >TC,00000000000003951999,1329399349764.f55cfda6067ad1425ba68d20d82c945b.
> >into f9c31ac78b3d4feb848b1e8d78975e34, size=532.0m; total size for store
> >is
> >2.1g
> >2012-02-23 22:29:43,260 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed compaction of 3 file(s) in EVENT of
> >TC,00000000000010971999,1329399349802.15720106584fe32f58287b6174970f31.
> >into af2472f30a9f4c558323a3268f98183c, size=517.9m; total size for store
> >is
> >2.6g
> >2012-02-23 22:30:19,476 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed compaction of 3 file(s) in EVENT of
> >TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731.
> >into 3ac255c439ab4c518f4a938d542c9220, size=335.4m; total size for store
> >is
> >2.1g
> >2012-02-23 22:31:15,051 INFO org.apache.hadoop.hbase.regionserver.Store:
> >Completed compaction of 3 file(s) in EVENT of
> >TC,00000000000008501999,1329399349789.fa0ac64d8ac9050f4acf2b0deb6e2dc7.
Mikael.S