Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase table as a queue.


Copy link to this message
-
Re: hbase table as a queue.
One thing I need to point out, is that we do not need the Queue items
to be worked on in order, so there is no traditional head and tail of
the Queue.  The Queue table is simply a set of work orders that can be
fetched randomly or by applying a scan for a particular set of Rows
that can even come out from the middle of the table.

-Jack

On Sat, Jul 16, 2011 at 9:38 AM, Jack Levin <[EMAIL PROTECTED]> wrote:
> Hello, we are thinking about using Hbase table as a simple queue which
> will dispatch the work for a mapreduce job, as well as real time
> fetching of data to present to end user.  In simple terms, suppose you
> had a data source table and a queue table.  The queue table has a
> smaller set of Rows that point to Values which in turn point to
> Perma-set table, which has large collection of Rows.  (so Queue{Row,
> Value} -> Perma-Set {Row, Value}).  Or Q-Value -> P-Row.   Our Goal is
> to look up which Rows to retrieve from the Perma-Set table by looking
> through the Queue.  Once the lookup into the Queue is done, the Row
> from the Queue must be deleted to avoid the same process of Perma-Set
> lookup be done twice; We expect many concurrent lookups to happen, so
> I assume the first thing we need to do is to have a client that does
> the work is acquire a lock on the Queue Row, process the work, then
> Remove the Queue Row.
>
> Has anyone done something similar before?  Any gotchas we should be away of?
>
> Thanks.
>
> -Jack
>