Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - hbase table as a queue.


Copy link to this message
-
hbase table as a queue.
Jack Levin 2011-07-16, 16:38
Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user.  In simple terms, suppose you
had a data source table and a queue table.  The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows.  (so Queue{Row,
Value} -> Perma-Set {Row, Value}).  Or Q-Value -> P-Row.   Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue.  Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.

Has anyone done something similar before?  Any gotchas we should be away of?

Thanks.

-Jack