Hi Laxman,

Personally I have not seen many variations - if at all - based on cluster size, apart from the heap settings. What are you proposing should be different for each profile?

Lars

On Jul 4, 2012, at 9:15, Laxman <[EMAIL PROTECTED]> wrote:

> Hi Devs,
>
> How about introducing multiple GC tuning profiles (like STANDALONE, SMALL,
> MEDIUM, LARGE ...) in HBase itself?
> Profiles can be categorized based on cluster size, usage, resources
> available (Memory, CPU).
>
> Why tuning profiles?
> -> Its tedious & error-prone to find, edit some parameter in script file.
> -> While editing manually and I'm sure we miss one or the other parameters
> in some cluster machine.
> -> These profiles boosts the user's confidence rather tune himself by trial
> and error for each parameter.
>
>
> I know I will get the following immediate answers. :-)
> -> it's the responsibility of user to tune GC
> -> it depends on usage
> -> it depends on environment (machine resources available)
> -> we do provide recommendations in hbase book
> -> there are so many resources over web about tuning. Why don't you refer to
> that?
> -> so on...
>
> I humbly accept all the above reasons but still I wanted to put forward the
> idea of tuning profiles and get your opinion.
> --
> Regards,
> Laxman
>
>

> Personally I have not seen many variations - if at all - based on
> cluster size, apart from the heap settings. What are you proposing
> should be different for each profile?

Not necessarily. We may have one profile only if we don't have many
variations. Few I can see [Xms, Xmx, NewSize, MaxNewSize]
IMHO, user can take advantage of community recommendations (GC collection
algorithm - HBase documentation, DirectMemorySize - HDFS-3600).

Also, if possible we can extend this profiles concept to HBase
configurations as well.
--
Regards,
Laxman

Hi Devs,

How about introducing multiple GC tuning profiles (like STANDALONE, SMALL,
MEDIUM, LARGE ...) in HBase itself?
Profiles can be categorized based on cluster size, usage, resources
available (Memory, CPU).

Why tuning profiles?
-> Its tedious & error-prone to find, edit some parameter in script file.
-> While editing manually and I'm sure we miss one or the other parameters
in some cluster machine.
-> These profiles boosts the user's confidence rather tune himself by trial
and error for each parameter.
I know I will get the following immediate answers. :-)
-> it's the responsibility of user to tune GC
-> it depends on usage
-> it depends on environment (machine resources available)
-> we do provide recommendations in hbase book
-> there are so many resources over web about tuning. Why don't you refer to
that?
-> so on...

I humbly accept all the above reasons but still I wanted to put forward the
idea of tuning profiles and get your opinion.
--
Regards,
Laxman

I would like to second Laxman's proposal. Currently, some of the
"default" hbase configuration is targeted towards newbies so as to
avoid getting basic questions in the mailing list ... which is ok. I
think we at least need something like an hbase-prod.xml that lists
more realistic values for a production like environment - or perhaps
provide value ranges that are appropriate for a production-like
environment.

For instance, default zookeeper.session.timeout of 3 minutes is known
to be too long ... so, what is a more realistic one for production ...
and so forth.
The profiles approach is even better if we see a lot of variations
depending on cluster / available heap size / etc.

I don't think anyone should expect blindly using them - but at least
it gives a much better starting point for tuning the cluster for folks
taking this into production.
--Suraj

On Wed, Jul 4, 2012 at 9:55 PM, Laxman <[EMAIL PROTECTED]> wrote:
>> Personally I have not seen many variations - if at all - based on
>> cluster size, apart from the heap settings. What are you proposing
>> should be different for each profile?
>
> Not necessarily. We may have one profile only if we don't have many
> variations. Few I can see [Xms, Xmx, NewSize, MaxNewSize]
> IMHO, user can take advantage of community recommendations (GC collection
> algorithm - HBase documentation, DirectMemorySize - HDFS-3600).
>
> Also, if possible we can extend this profiles concept to HBase
> configurations as well.
> --
> Regards,
> Laxman
>
>