Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Simple stastics per region


Copy link to this message
-
Re: Simple stastics per region
On Fri, Feb 22, 2013 at 10:40 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> This topic comes up now and then (see recent discussion about translating
> multi Gets into Scan+Filter).
>
> It's not that hard to keep statistics as part of compactions.
> I envision two knobs:
> 1. Max number of distinct values to track directly. If a column has less
> this # of values, keep track of their occurrences explicitly.
> 2. Number of (equal width) histogram partitions to maintain.
>
> Statistics would be kept per store (i.e. per region per column family) and
> stored into an HBase table (one row per store).Initially we could just
> support major compactions that atomically insert a new version of that
> statistics for the store.
>
>
Sounds great.

In .META. add columns for each each cf on each region row?  Or another
table?

What kind of stats would you keep?  Would they be useful for operators?  Or
just for stuff like say Phoenix making decisions?

> An simple implementation (not knowing ahead of time how many values it
> will see during the compaction) could start by keeping track of individual
> values for columns. If it gets past the max # of distinct values to track,
> start with equal width histograms (using the distinct values picket up so
> far to estimate an initial partition width).
> If the number of partition gets larger than what was configured it would
> increase the width and merge the previous counts into the new width (which
> means the new partition width must be a multiple of the previous size).
> There's probably a lot of other fanciness that could be used here (haven't
> spent a lot of time thinking about details).
>
>
> Is this something that should be in core HBase or rather be implemented as
> coprocessor?
>
I think it could go in core if it generated pretty pictures.

St.Ack