Hello,

What are the main metrics/signals people use/look at to say "OK, I really
need to do a major compaction now", other than "hm, performance is not
good, let's try major compaction tonight"?

Thanks,
Otis
--
HBASE Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

Otis,

I think it depends on your use case.  Some use cases are extremely
dependent on locality and latency, so they will major compact any region
the balancer has moved.  Some people follow the HFile count and will
trigger a major when there are x amount of files.  Others will just follow
best practices and do it daily.  Some do very little writes and only
compact once a week.

On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> What are the main metrics/signals people use/look at to say "OK, I really
> need to do a major compaction now", other than "hm, performance is not
> good, let's try major compaction tonight"?
>
> Thanks,
> Otis
> --
> HBASE Performance Monitoring - http://sematext.com/spm/index.html
> Search Analytics - http://sematext.com/search-analytics/index.html
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

bq. they will major compact any region the balancer has moved.

Interesting. Do they achieve this through scripting ?

I think HBase can perform some follow-on action after region movement.

Cheers

On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Otis,
>
> I think it depends on your use case.  Some use cases are extremely
> dependent on locality and latency, so they will major compact any region
> the balancer has moved.  Some people follow the HFile count and will
> trigger a major when there are x amount of files.  Others will just follow
> best practices and do it daily.  Some do very little writes and only
> compact once a week.
>
> On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > What are the main metrics/signals people use/look at to say "OK, I really
> > need to do a major compaction now", other than "hm, performance is not
> > good, let's try major compaction tonight"?
> >
> > Thanks,
> > Otis
> > --
> > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > Search Analytics - http://sematext.com/search-analytics/index.html
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Ted,

  Yes, it is all done through scripting.

On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> bq. they will major compact any region the balancer has moved.
>
> Interesting. Do they achieve this through scripting ?
>
> I think HBase can perform some follow-on action after region movement.
>
> Cheers
>
> On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <[EMAIL PROTECTED]
> >wrote:
>
> > Otis,
> >
> > I think it depends on your use case.  Some use cases are extremely
> > dependent on locality and latency, so they will major compact any region
> > the balancer has moved.  Some people follow the HFile count and will
> > trigger a major when there are x amount of files.  Others will just
> follow
> > best practices and do it daily.  Some do very little writes and only
> > compact once a week.
> >
> > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > What are the main metrics/signals people use/look at to say "OK, I
> really
> > > need to do a major compaction now", other than "hm, performance is not
> > > good, let's try major compaction tonight"?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > Search Analytics - http://sematext.com/search-analytics/index.html
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

Thanks confirmation, Kevin.

I logged HBASE-7340 so that user doesn't have to script for compaction.

On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Ted,
>
>   Yes, it is all done through scripting.
>
> On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > bq. they will major compact any region the balancer has moved.
> >
> > Interesting. Do they achieve this through scripting ?
> >
> > I think HBase can perform some follow-on action after region movement.
> >
> > Cheers
> >
> > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Otis,
> > >
> > > I think it depends on your use case.  Some use cases are extremely
> > > dependent on locality and latency, so they will major compact any
> region
> > > the balancer has moved.  Some people follow the HFile count and will
> > > trigger a major when there are x amount of files.  Others will just
> > follow
> > > best practices and do it daily.  Some do very little writes and only
> > > compact once a week.
> > >
> > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Hello,
> > > >
> > > > What are the main metrics/signals people use/look at to say "OK, I
> > really
> > > > need to do a major compaction now", other than "hm, performance is
> not
> > > > good, let's try major compaction tonight"?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > > Search Analytics - http://sematext.com/search-analytics/index.html
> > > >
> > >
> > >
> > >
> > > --
> > > Kevin O'Dell
> > > Customer Operations Engineer, Cloudera
> > >
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Ted,

  That is awesome.  Thank you!

On Wed, Dec 12, 2012 at 3:36 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks confirmation, Kevin.
>
> I logged HBASE-7340 so that user doesn't have to script for compaction.
>
> On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <[EMAIL PROTECTED]
> >wrote:
>
> > Ted,
> >
> >   Yes, it is all done through scripting.
> >
> > On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > bq. they will major compact any region the balancer has moved.
> > >
> > > Interesting. Do they achieve this through scripting ?
> > >
> > > I think HBase can perform some follow-on action after region movement.
> > >
> > > Cheers
> > >
> > > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Otis,
> > > >
> > > > I think it depends on your use case.  Some use cases are extremely
> > > > dependent on locality and latency, so they will major compact any
> > region
> > > > the balancer has moved.  Some people follow the HFile count and will
> > > > trigger a major when there are x amount of files.  Others will just
> > > follow
> > > > best practices and do it daily.  Some do very little writes and only
> > > > compact once a week.
> > > >
> > > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > What are the main metrics/signals people use/look at to say "OK, I
> > > really
> > > > > need to do a major compaction now", other than "hm, performance is
> > not
> > > > > good, let's try major compaction tonight"?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > > > Search Analytics - http://sematext.com/search-analytics/index.html
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kevin O'Dell
> > > > Customer Operations Engineer, Cloudera
> > > >
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

In RegionObserver, we already have the following hook:

  /**
   * Called after the region is reported as open to the master.
   * @param c the environment provided by the region server
   */
  void postOpen(final ObserverContext<RegionCoprocessorEnvironment> c);

Auto-compaction logic can be triggered through the above hook.

Cheers

On Wed, Dec 12, 2012 at 12:36 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks confirmation, Kevin.
>
> I logged HBASE-7340 so that user doesn't have to script for compaction.
>
>
> On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:
>
>> Ted,
>>
>>   Yes, it is all done through scripting.
>>
>> On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>> > bq. they will major compact any region the balancer has moved.
>> >
>> > Interesting. Do they achieve this through scripting ?
>> >
>> > I think HBase can perform some follow-on action after region movement.
>> >
>> > Cheers
>> >
>> > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <
>> [EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > Otis,
>> > >
>> > > I think it depends on your use case.  Some use cases are extremely
>> > > dependent on locality and latency, so they will major compact any
>> region
>> > > the balancer has moved.  Some people follow the HFile count and will
>> > > trigger a major when there are x amount of files.  Others will just
>> > follow
>> > > best practices and do it daily.  Some do very little writes and only
>> > > compact once a week.
>> > >
>> > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > What are the main metrics/signals people use/look at to say "OK, I
>> > really
>> > > > need to do a major compaction now", other than "hm, performance is
>> not
>> > > > good, let's try major compaction tonight"?
>> > > >
>> > > > Thanks,
>> > > > Otis
>> > > > --
>> > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
>> > > > Search Analytics - http://sematext.com/search-analytics/index.html
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kevin O'Dell
>> > > Customer Operations Engineer, Cloudera
>> > >
>> >
>>
>>
>>
>> --
>> Kevin O'Dell
>> Customer Operations Engineer, Cloudera
>>
>
>