Seems like it should be possible to implement a queue service like
Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
once a client picks them, but pop back into the queue after a timeout
unless the client doesn't explicitly delete them.

Anyone done that?

--
-- Mike

Mike Schilli
[EMAIL PROTECTED]

[1] http://aws.amazon.com/sqs

I proposed implementing an SQS intf a long while back. However at this
point you'd be better off using BK or Hedwig to implement it.

Patrick

On Tue, Dec 6, 2011 at 11:16 AM, Mike Schilli <[EMAIL PROTECTED]> wrote:
> Seems like it should be possible to implement a queue service like
> Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
> once a client picks them, but pop back into the queue after a timeout
> unless the client doesn't explicitly delete them.
>
> Anyone done that?
>
> --
> -- Mike
>
> Mike Schilli
> [EMAIL PROTECTED]
>
> [1] http://aws.amazon.com/sqs

Curator has both a Queue and a Priority Queue as described on the
ZooKeeper recipes page:

https://github.com/Netflix/curator/wiki/Recipes
-JZ

On 12/6/11 11:16 AM, "Mike Schilli" <[EMAIL PROTECTED]> wrote:

>Seems like it should be possible to implement a queue service like
>Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
>once a client picks them, but pop back into the queue after a timeout
>unless the client doesn't explicitly delete them.
>
>Anyone done that?
>
>--
>-- Mike
>
>Mike Schilli
>[EMAIL PROTECTED]
>
>[1] http://aws.amazon.com/sqs
>

Mike,

Just curious, why wouldn't you want to use a messaging queue like Kafka (
http://incubator.apache.org/kafka/) or ActiveMQ ?

Thanks,
Neha

On Tue, Dec 6, 2011 at 11:21 AM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> Curator has both a Queue and a Priority Queue as described on the
> ZooKeeper recipes page:
>
> https://github.com/Netflix/curator/wiki/Recipes
>
>
> -JZ
>
> On 12/6/11 11:16 AM, "Mike Schilli" <[EMAIL PROTECTED]> wrote:
>
> >Seems like it should be possible to implement a queue service like
> >Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
> >once a client picks them, but pop back into the queue after a timeout
> >unless the client doesn't explicitly delete them.
> >
> >Anyone done that?
> >
> >--
> >-- Mike
> >
> >Mike Schilli
> >[EMAIL PROTECTED]
> >
> >[1] http://aws.amazon.com/sqs
> >
>
>

ZK definitely provides the raw material for something like SQS.

Generally, however, a queue *in* ZK isn't quite as good an idea as a queue
that is *coordinated* by ZK, mostly for performance reasons.  Thus Pat's
suggestion to look at Bookkeeper is not a bad one at all.  A direct clone
of SQS that pushes state to ZK for fail-over purposes would be very easy to
build.
On Tue, Dec 6, 2011 at 11:16 AM, Mike Schilli <[EMAIL PROTECTED]> wrote:

> Seems like it should be possible to implement a queue service like
> Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
> once a client picks them, but pop back into the queue after a timeout
> unless the client doesn't explicitly delete them.
>
> Anyone done that?
>
> --
> -- Mike
>
> Mike Schilli
> [EMAIL PROTECTED]
>
> [1] http://aws.amazon.com/sqs
>

This is exactly what Hedwig does.
John as Neha mentioned,
 You might want to look at Kafka/Hedwig/ActiveMQ.

mahadev

On Tue, Dec 6, 2011 at 11:41 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> ZK definitely provides the raw material for something like SQS.
>
> Generally, however, a queue *in* ZK isn't quite as good an idea as a queue
> that is *coordinated* by ZK, mostly for performance reasons.  Thus Pat's
> suggestion to look at Bookkeeper is not a bad one at all.  A direct clone
> of SQS that pushes state to ZK for fail-over purposes would be very easy to
> build.
>
>
> On Tue, Dec 6, 2011 at 11:16 AM, Mike Schilli <[EMAIL PROTECTED]> wrote:
>
>> Seems like it should be possible to implement a queue service like
>> Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
>> once a client picks them, but pop back into the queue after a timeout
>> unless the client doesn't explicitly delete them.
>>
>> Anyone done that?
>>
>> --
>> -- Mike
>>
>> Mike Schilli
>> [EMAIL PROTECTED]
>>
>> [1] http://aws.amazon.com/sqs
>>

Sorry not John, Mike :)...

mahadev

PS: not getting enough sleep! :)

On Tue, Dec 6, 2011 at 12:44 PM, Mahadev Konar <[EMAIL PROTECTED]> wrote:
> This is exactly what Hedwig does.
> John as Neha mentioned,
>  You might want to look at Kafka/Hedwig/ActiveMQ.
>
> mahadev
>
> On Tue, Dec 6, 2011 at 11:41 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>> ZK definitely provides the raw material for something like SQS.
>>
>> Generally, however, a queue *in* ZK isn't quite as good an idea as a queue
>> that is *coordinated* by ZK, mostly for performance reasons.  Thus Pat's
>> suggestion to look at Bookkeeper is not a bad one at all.  A direct clone
>> of SQS that pushes state to ZK for fail-over purposes would be very easy to
>> build.
>>
>>
>> On Tue, Dec 6, 2011 at 11:16 AM, Mike Schilli <[EMAIL PROTECTED]> wrote:
>>
>>> Seems like it should be possible to implement a queue service like
>>> Amazon's SQS [1] with ZooKeeper, with queue items becoming 'invisible'
>>> once a client picks them, but pop back into the queue after a timeout
>>> unless the client doesn't explicitly delete them.
>>>
>>> Anyone done that?
>>>
>>> --
>>> -- Mike
>>>
>>> Mike Schilli
>>> [EMAIL PROTECTED]
>>>
>>> [1] http://aws.amazon.com/sqs
>>>