Hello folks,

    I got to the Curator wiki page to see something about the distributed
queue recipe and got surprised with the big  warning telling to not use
zookeeper as a distributed queue controller, the reason page described some
cases that I passed here... is this *TRUE*, if so, what option do we have.

The message:
*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues. Please
see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
details. Links

https://github.com/Netflix/curator/wiki/Distributed-Queue
https://github.com/Netflix/curator/wiki/Tech-Note-4

--
Att.
Diego de Oliveira
System Architect
[EMAIL PROTECTED]
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference

That is generally accurate. ZooKeeper isn't written to be a distributed
queue in the high-throughput messaging kind of sense. If you need something
truly lightweight and simple to serve as a distributed queue for a few
messages, it might work. But for anything that could see significant
volume, there's better options out there.
ZooKeeper is used to implement several systems that will serve this need.
Depending on what you're looking for, you might check out:
Kafka: http://incubator.apache.org/kafka/
Kestrel: https://github.com/robey/kestrel

Or many of the brokered messaging servers like ActiveMQ, etc.

C
On Tue, Dec 4, 2012 at 9:42 PM, Diego Oliveira <[EMAIL PROTECTED]> wrote:

> Hello folks,
>
>     I got to the Curator wiki page to see something about the distributed
> queue recipe and got surprised with the big  warning telling to not use
> zookeeper as a distributed queue controller, the reason page described some
> cases that I passed here... is this *TRUE*, if so, what option do we have.
>
> The message:
> *IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues. Please
> see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
> details. Links
>
> https://github.com/Netflix/curator/wiki/Distributed-Queue
> https://github.com/Netflix/curator/wiki/Tech-Note-4
>
> --
> Att.
> Diego de Oliveira
> System Architect
> [EMAIL PROTECTED]
> www.diegooliveira.com
> Never argue with a fool -- people might not be able to tell the difference
>

Is there a number pointing what lightweight mean for zookeeper (like
messages/seconds or message/seconds/size)?

--
Diego

On Wed, Dec 5, 2012 at 1:36 AM, Camille Fournier <[EMAIL PROTECTED]> wrote:

> That is generally accurate. ZooKeeper isn't written to be a distributed
> queue in the high-throughput messaging kind of sense. If you need something
> truly lightweight and simple to serve as a distributed queue for a few
> messages, it might work. But for anything that could see significant
> volume, there's better options out there.
> ZooKeeper is used to implement several systems that will serve this need.
> Depending on what you're looking for, you might check out:
> Kafka: http://incubator.apache.org/kafka/
> Kestrel: https://github.com/robey/kestrel
>
> Or many of the brokered messaging servers like ActiveMQ, etc.
>
> C
>
>
> On Tue, Dec 4, 2012 at 9:42 PM, Diego Oliveira <[EMAIL PROTECTED]> wrote:
>
> > Hello folks,
> >
> >     I got to the Curator wiki page to see something about the distributed
> > queue recipe and got surprised with the big  warning telling to not use
> > zookeeper as a distributed queue controller, the reason page described
> some
> > cases that I passed here... is this *TRUE*, if so, what option do we
> have.
> >
> > The message:
> > *IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
> Please
> > see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4>
> for
> > details. Links
> >
> > https://github.com/Netflix/curator/wiki/Distributed-Queue
> > https://github.com/Netflix/curator/wiki/Tech-Note-4
> >
> > --
> > Att.
> > Diego de Oliveira
> > System Architect
> > [EMAIL PROTECTED]
> > www.diegooliveira.com
> > Never argue with a fool -- people might not be able to tell the
> difference
> >
>

--
Att.
Diego de Oliveira
System Architect
[EMAIL PROTECTED]
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference

InterestingŠ. I had long ago requested clarification of the zookeeper
limits.  This is in keeping with my expectations.  Good for relatively
small numbers of children.

For example, you would be safe trying to coordinate 1000 machines but not
safe trying to coordinate the locations of 100,000+ shards over 1000
machines.

Bryan

On 12/4/12 9:42 PM, "Diego Oliveira" <[EMAIL PROTECTED]> wrote:

>Hello folks,
>
>    I got to the Curator wiki page to see something about the distributed
>queue recipe and got surprised with the big  warning telling to not use
>zookeeper as a distributed queue controller, the reason page described
>some
>cases that I passed here... is this *TRUE*, if so, what option do we have.
>
>The message:
>*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
>Please
>see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
>details. Links
>
>https://github.com/Netflix/curator/wiki/Distributed-Queue
>https://github.com/Netflix/curator/wiki/Tech-Note-4
>
>--
>Att.
>Diego de Oliveira
>System Architect
>[EMAIL PROTECTED]
>www.diegooliveira.com
>Never argue with a fool -- people might not be able to tell the difference

I took a look last night and it sounds like the big problem is the
serialization mechanism. The transportation layer has a limit of 1M in
size, the getChildren method pass all the children name in that chunk, so
if you have a lot of children then you are in trouble.
On Wed, Dec 5, 2012 at 7:09 AM, Bryan Thompson <[EMAIL PROTECTED]> wrote:

> InterestingŠ. I had long ago requested clarification of the zookeeper
> limits.  This is in keeping with my expectations.  Good for relatively
> small numbers of children.
>
> For example, you would be safe trying to coordinate 1000 machines but not
> safe trying to coordinate the locations of 100,000+ shards over 1000
> machines.
>
> Bryan
>
> On 12/4/12 9:42 PM, "Diego Oliveira" <[EMAIL PROTECTED]> wrote:
>
> >Hello folks,
> >
> >    I got to the Curator wiki page to see something about the distributed
> >queue recipe and got surprised with the big  warning telling to not use
> >zookeeper as a distributed queue controller, the reason page described
> >some
> >cases that I passed here... is this *TRUE*, if so, what option do we have.
> >
> >The message:
> >*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
> >Please
> >see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
> >details. Links
> >
> >https://github.com/Netflix/curator/wiki/Distributed-Queue
> >https://github.com/Netflix/curator/wiki/Tech-Note-4
> >
> >--
> >Att.
> >Diego de Oliveira
> >System Architect
> >[EMAIL PROTECTED]
> >www.diegooliveira.com
> >Never argue with a fool -- people might not be able to tell the difference
>
>
--
Att.
Diego de Oliveira
System Architect
[EMAIL PROTECTED]
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference