Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - secondary index feature

Copy link to this message
Re: secondary index feature
lars hofhansl 2013-12-22, 22:37
The devil is often in the details. On the surface it looks simple.

How specifically are the stale indexes ignored? Are the guaranteed to be no races?
Is deletion handled correctly?Does it work with multiple versions?
What happens when the client dies 1/2 way through an update?
It's easy to do eventually consistent indexes. Truly consistent indexes without transactions are tricky.
Also, scanning an index and then doing point-gets against a main table is slow (unless the index is very selective. The Phoenix team measured that there is only an advantage if the index filters out 98-99% of the data).
So then one would revert to covered indexes and suddenly is not so easy
to detect stale index entries.

I blogged about these issues here:

Phoenix has a (pretty involved) solution now that works around the fact that HBase has no transactions.
-- Lars

 From: Henning Blohm <[EMAIL PROTECTED]>
Sent: Sunday, December 22, 2013 2:11 AM
Subject: secondary index feature

Lately we have added a secondary index feature to a persistence tier
over HBASE. Essentially we implemented what is described as "Dual-Write
Secondary Index" in http://hbase.apache.org/book/secondary.indexes.html.
I.e. while updating an entity, actually before writing the actual
update, indexes are updated. Lookup via the index ignores stale entries.
A recurring rebuild and clean out of stale entries takes care the
indexes are trimmed and accurate.

None of this was terribly complex to implement. In fact, it seemed like
something you could do generically, maybe not on the HBASE level itself,
but as a toolbox / utility style library.

Is anybody on the list aware of anything useful already existing in that

Henning Blohm

*ZFabrik Software KG*

T:     +49 6227 3984255
F:     +49 6227 3984254
M:     +49 1781891820

Lammstrasse 2 69190 Walldorf

Linkedin <http://www.linkedin.com/pub/henning-blohm/0/7b5/628>
ZFabrik <http://www.zfabrik.de>
Blog <http://www.z2-environment.net/blog>
Z2-Environment <http://www.z2-environment.eu>
Z2 Wiki <http://redmine.z2-environment.net>