Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - Major Compaction Concerns


+
Mikael Sitruk 2012-01-08, 15:25
+
Mikael Sitruk 2012-01-08, 15:27
+
Ted Yu 2012-01-08, 17:13
+
Mikael Sitruk 2012-01-08, 17:28
Copy link to this message
-
Re: Major Compaction Concerns
yuzhihong@... 2012-01-08, 17:54
About 2b, have you tried getting the major compaction setting from column descriptor ?

For 2a, what you requested would result in new methods of HBaseConfiguration class to be added. Currently the configuration on client class path would be used.

Cheers

On Jan 8, 2012, at 9:28 AM, Mikael Sitruk <[EMAIL PROTECTED]> wrote:

> Ted hi
> First thanks for answering, regarding the JIRA i will fill them
> Second, it seems that i did not explain myself correctly regarding 2.a. -
> As you i do not expect that a configuration set on my client will be
> propagated to the cluster, but i do expect that if i set a configuration on
> a server then doing connection.getConfiguration() from a client i will get
> teh configuration from the cluster.
> Currently the configuration returned is from the client config.
> So the problem is that you have no way to check the configuration of a
> cluster.
> I would expect to have some API to return the cluster config and even
> getting a map <serverInfo, config> so it can be easy to check cluster
> problem using code.
>
> 2.b. I know this code, and i tried to validate it. I set in the server
> config the "hbase.hregion.majorcompaction" to "0", then start the server
> (cluster). Since from the UI or from JMX this parameter is not visible at
> the cluster level, I try to get the value from the client (to see that the
> cluster is using it)
>
> *HTableDescriptor hTableDescriptor > conn.getHTableDescriptor(Bytes.toBytes("my table"));*
>
> *hTableDescriptor.getValue("hbase.hregion.majorcompaction")*
> but i still got 24h (and not the value set in the config)! that was my
> problem from the beginning! ==> Using the config (on the server side) will
> not propagate into the table/column family
>
> Mikael.S
>
> On Sun, Jan 8, 2012 at 7:13 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> I am not expert in major compaction feature.
>> Let me try to answer questions in #2.
>>
>> 2.a
>>> If I set the property via the configuration shouldn’t all the cluster be
>>> aware of?
>>
>> There're multiple clients connecting to one cluster. I wouldn't expect
>> values in the configuration (m_hbConfig) to propagate onto the cluster.
>>
>> 2.b
>> Store.getNextMajorCompactTime() shows that "hbase.hregion.majorcompaction"
>> can be specified per column family:
>>
>> long getNextMajorCompactTime() {
>>   // default = 24hrs
>>   long ret = conf.getLong(HConstants.MAJOR_COMPACTION_PERIOD,
>> 1000*60*60*24);
>>   if (family.getValue(HConstants.MAJOR_COMPACTION_PERIOD) != null) {
>>
>> 2.d
>>> d. I tried also to setup the parameter via hbase shell but setting such
>>> properties is not supported. (do you plan to add such support via the
>>> shell?)
>>
>> This is a good idea. Please open a JIRA.
>>
>> For #5, HBASE-3965 is an improvement and doesn't have a patch yet.
>>
>> Allow me to quote Alan Kay: 'The best way to predict the future is to
>> invent it.'
>>
>> Once we have a patch, we can always backport it to 0.92 after some people
>> have verified the improvement.
>>
>>> 6.       In case a compaction (major) is running it seems there is no way
>>> to stop-it. Do you plan to add such feature?
>>
>> Again, logging a JIRA would provide a good starting point for discussion.
>>
>> Thanks for the verification work and suggestions, Mikael.
>>
>> On Sun, Jan 8, 2012 at 7:27 AM, Mikael Sitruk <[EMAIL PROTECTED]
>>> wrote:
>>
>>> I forgot to mention, I'm using HBase 0.90.1
>>>
>>> Regards,
>>> Mikael.S
>>>
>>> On Sun, Jan 8, 2012 at 5:25 PM, Mikael Sitruk <[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> I have some concern regarding major compactions below...
>>>>
>>>>
>>>>   1. According to best practices from the mailing list and from the
>>>>   book, automatic major compaction should be disabled. This can be
>> done
>>> by
>>>>   setting the property ‘hbase.hregion.majorcompaction’ to �
�’.
>>> Neverhteless
>>>>   even after having doing this I STILL see “major compaction” messages
�*
+
Mikael Sitruk 2012-01-08, 20:04
+
Ted Yu 2012-01-08, 20:22
+
Mikael Sitruk 2012-01-08, 20:55
+
Ted Yu 2012-01-08, 21:07
+
Mikael Sitruk 2012-01-08, 23:05
+
Ted Yu 2012-01-09, 00:19
+
Nicolas Spiegelberg 2012-01-09, 18:55
+
Ted Yu 2012-01-09, 19:37
+
Nicolas Spiegelberg 2012-01-09, 19:42
+
Mikael Sitruk 2012-01-09, 23:03
+
Nicolas Spiegelberg 2012-01-12, 21:44
+
Ted Yu 2012-01-12, 22:09
+
lars hofhansl 2012-01-12, 22:20
+
Mikael Sitruk 2012-01-13, 23:23
+
Nicolas Spiegelberg 2012-01-14, 02:06
+
Mikael Sitruk 2012-01-14, 21:30
+
Doug Meil 2012-01-16, 22:39
+
Mikael Sitruk 2012-02-19, 22:05
+
Nicolas Spiegelberg 2012-02-20, 03:47