Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - Re: A general question on maxVersion handling when we have Secondary index tables


+
Jonathan Hsieh 2012-08-29, 13:47
+
Ted Yu 2012-08-29, 14:15
+
Ramkrishna.S.Vasudevan 2012-08-29, 15:12
+
Jonathan Hsieh 2012-08-29, 16:11
+
Ted Yu 2012-08-29, 16:19
Copy link to this message
-
Re: A general question on maxVersion handling when we have Secondary index tables
Jesse Yates 2012-08-29, 17:03
Client library style stuff is _nice_ but one of the things everyone asks of
database is that we provide an index (cassandra has it, riak has it, mysql
has it...hbase doesn't? Yes, different systems,etc.,etc., but the point is
we could do it). Further, if we build it as a part of hbase, we can make it
faster... though don't ask me the _how_ on that yet ;)

Talking with Lars, we could provide a lot of the indexing infrastructure,
but leave the actual indexing (convert row|cf|cq|ts|value to an index value
and vice-versa) to a client library gives us a lot of the flexibility that
people would need. And I take it that most people already have some form of
indexing already (be it consistent or not), so we can do it 'the right way'
in terms of queries, etc. and provide pluggable infrastructure (with a
decent default) so people can roll in their own implementations.

That said, I think we can do secondary indexing without too many changes to
HBase (region co-location/pinning that Ted suggests would just be sweet
overall)arguing for a client library. However, if we decide this is one of
the things we want to support going forward as a project, then it makes
more sense to do it as part of HBase, rather than pointing people to some
guy/gal's website with the information (which may or may not be up to date)
for how munge indexing in. Instead, it would be so much nicer to just flip
a couple switches, maybe plug in a couple of classes and have indexing
_just work_.

Just my $0.02

-Jesse
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, Aug 29, 2012 at 9:19 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> For the secondary index based on state portion of address example, I wonder
> if we can achieve comparable performance using scan with proper filter.
>
> Cheers
>
> On Wed, Aug 29, 2012 at 9:11 AM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:
>
> > Ted,
> >
> > Ram's summarizes the concern succinctly -- to answer the specific
> question
> > it isn't for versions -- it is for the case where a secondary index can
> > point to many many primary rows.  (let's say we have a rowkey userid and
> we
> > want to have a 2ndary index based on the state portion of there address
> >  --- we'll end up pointing to many many primary rows).
> >
> > Jon.
> >
> >
> >
> > On Wed, Aug 29, 2012 at 7:15 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Thanks for the detailed response, Jon.
> > >
> > > bq. it would mean that a query based on secondary index would
> > > potentially have to hit every region server that has a region in the
> > > primary table.
> > >
> > > Can you elaborate on the above a little bit ?
> > > Is this because secondary index would point us to more than one region
> in
> > > the data table because several versions are saved for the same row ?
> > >
> > > My thinking was to ease management of simultaneous (data and index)
> > region
> > > split through region colocation.
> > >
> > > Cheers
> > >
> > > On Wed, Aug 29, 2012 at 6:47 AM, Jonathan Hsieh <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I'm more of a fan of having secondary indexes added as an external
> > > feature
> > > > (coproc or new client library on top of our current client library)
> and
> > > > focusing on only adding apis necessary to make 2ndary indexes
> possible
> > > and
> > > > correct on/in HBase.  There are many different use patterns and
> > > > requirements and one style of secondary index will not be good for
> > > > everything.  Do we only care about this working well for highly
> > > selectivity
> > > > keys?  What are possible indexes (col name, value, value prefix,
> > > everything
> > > > our filters support?)  Do we care more about writes or reads, ACID
> > > > correctness or speed, etc?  Also, there are several questions about
> how
> > > we
> > > > handle other features in conjunction with 2ndary indexes:
> replication,
> > > bulk
> > > > load, snapshots, to name a few.
> > > >
> > > > Maybe it makes sense to spend some time defining what we want to
+
Ted Yu 2012-08-29, 17:07
+
Jonathan Hsieh 2012-08-29, 18:18
+
Ramkrishna.S.Vasudevan 2012-08-30, 04:34
+
Jonathan Hsieh 2012-08-29, 17:46
+
Ramkrishna.S.Vasudevan 2012-08-30, 04:18
+
Jesse Yates 2012-08-28, 07:59
+
Ramkrishna.S.Vasudevan 2012-08-28, 08:51
+
Wei Tan 2012-08-28, 15:52
+
Ramkrishna.S.Vasudevan 2012-08-29, 04:18
+
Ted Yu 2012-08-28, 16:03
+
Stack 2012-08-29, 22:32
+
Jesse Yates 2012-08-28, 17:03
+
Ted Yu 2012-08-28, 17:34
+
Ramkrishna.S.Vasudevan 2012-08-28, 07:24