Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase table as a queue.


Copy link to this message
-
hbase table as a queue.
Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user.  In simple terms, suppose you
had a data source table and a queue table.  The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows.  (so Queue{Row,
Value} -> Perma-Set {Row, Value}).  Or Q-Value -> P-Row.   Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue.  Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.

Has anyone done something similar before?  Any gotchas we should be away of?

Thanks.

-Jack
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB