Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Re: HBase Developer's Pow-wow.


+
N Keywal 2012-08-30, 07:38
+
Stack 2012-08-30, 04:25
+
Ramkrishna.S.Vasudevan 2012-08-30, 04:35
+
Stack 2012-08-30, 04:56
+
Jonathan Hsieh 2012-08-29, 18:30
+
Ted Yu 2012-08-30, 17:20
+
Devaraj Das 2012-08-29, 20:06
+
Stack 2012-08-29, 20:32
+
Ramkrishna.S.Vasudevan 2012-08-30, 04:21
+
Devaraj Das 2012-08-29, 21:43
+
Jonathan Hsieh 2012-08-29, 23:12
+
Devaraj Das 2012-08-30, 06:12
+
Ramkrishna.S.Vasudevan 2012-08-30, 07:05
+
Andrew Purtell 2012-08-30, 06:58
+
Jimmy Xiang 2012-08-29, 20:11
+
Andrew Purtell 2012-08-29, 20:15
+
Lars George 2012-08-30, 22:04
+
Devaraj Das 2012-08-30, 22:36
+
Stack 2012-08-30, 22:42
+
Stack 2012-08-31, 22:59
+
Stack 2012-09-03, 15:40
+
Ramkrishna.S.Vasudevan 2012-09-05, 04:18
+
Stack 2012-09-09, 22:08
+
Jesse Yates 2012-09-09, 22:11
+
lars hofhansl 2012-09-10, 22:46
+
Stack 2012-09-09, 22:21
+
Jesse Yates 2012-09-09, 22:25
+
Stack 2012-09-09, 22:44
+
Jacques 2012-09-10, 03:03
+
Jacques 2012-09-10, 07:03
+
Andrew Purtell 2012-09-10, 18:09
+
Matt Corgan 2012-09-10, 19:13
+
Jacques 2012-09-10, 23:40
+
Matt Corgan 2012-09-11, 01:20
Copy link to this message
-
Re: HBase Developer's Pow-wow.
On Mon, Sep 10, 2012 at 6:20 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> ... snipping lots of helpful use cases...
It seems like portions of what you discussed would probably be nominally
impacted by indexes while other would be very impacted.  Also seems like
compound-qualifier indexing would potentially be of interest to you...
(although I'm not sure how much it would buy you). Are you going to be at
the powwow tomorrow?
> Seems like there are 3 categories of sparseness:
> 1) sparse indexes (like ipAddress) where a per-table approach is more
> efficient for reads
> 2) dense indexes (like eventType) where there are likely values of every
> index key on each region
> 3) very dense indexes (like male/female) where you should just be doing a
> table scan anyway
>
Yes.  I probably shouldn't have used the male/female example since you're
right that a table scan is probably the best the option in that case.  For
category one, I was imagining a situation of more extreme sparseness such
as one target row in a large number of regions.  This is the place where
the all region checking of region-based approach is the most egregious.
 I'd probably put anything that was in at a small percentages of regions as
the second case. (I also wonder if, in the single row scenario, a judicious
use of bloomfilters might provide satisfactory performance even if you do
need to hit all regions-- one of the things we've used as a guiding
principle for our search stuff is that if you're trying to hit realtime,
you can actually eat the most latency on the smallest scan since you have
so little data to move around...depends on allowable memory usage I
suppose.)
> Why is the per-region
> approach more beneficial than the per-table?  Is it because it's easier to
> plug into hbase's existing per-region MapReduce splitter?
>

Part of it has to do with a bunch of non-HBase work I've been doing over
the past few years.  That's why I really hope people share as many use
cases as possible... so that the conclusions that come out of our work are
representative of everyone's needs (as much as possible).  What makes me
lean towards region-level for a lot of use cases are the following:  (I
hadn't even really thought about the existing MR splitter.)
- How to maintain consistency (maybe this is unimportant?)
- How to avoid network bottleneck as the cluster expands (in the case of a
per-table approach, you're going to have pass primary keys around
constantly except in the case that the only value you want is the indexed
value and you saved that entire value in the index table.)
- How to maximize scale.  (In the per table case, a particular set of
indexed values will probably be colocated among a fraction of all nodes.
 Any kind of parallel/MR job will then be constrained by these nodes.)
- How to minimize long term storage cost of indexes.  (If we have
region-level relationships, we can get more tightly coupled over time and
use more efficient compact approaches like the store file position approach
I tossed out in one of my other emails.)

I spent some time in the Cassandra community doing a review of various
indexing use cases.  I should go take another look to see what they do and
how it works for them...
>> Thanks for starting the important discussion.

Lots to talk about. Lots to potentially do.  It will be interesting to see
who has time to put against this as that will probably substantially
constrain all of our great ideas :)

Jacques
+
Andrew Purtell 2012-09-11, 04:22
+
Ramkrishna.S.Vasudevan 2012-09-11, 04:47
+
Ted Yu 2012-09-10, 17:51
+
Jacques 2012-09-10, 20:45
+
Stack 2012-09-10, 04:41
+
Andrew Purtell 2012-09-10, 17:58
+
Jacques 2012-09-10, 20:50
+
Ted Yu 2012-08-29, 18:40
+
Devaraj Das 2012-09-11, 00:21
+
Matt Corgan 2012-09-11, 05:59
+
Stack 2012-09-05, 04:36