Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Use cases for ZooKeeper


Copy link to this message
-
Re: Use cases for ZooKeeper
OK - so this is two options for doing the same thing. You use a Leader
Election algorithm to make sure that only one node in the cluster is
operating on a work unit. Curator has an implementation (it's really just
a distributed lock with a slightly different API).

-JZ

On 1/5/12 12:04 AM, "Josh Stone" <[EMAIL PROTECTED]> wrote:

>Thanks for the response. Comments below:
>
>On Wed, Jan 4, 2012 at 10:46 PM, Jordan Zimmerman
><[EMAIL PROTECTED]>wrote:
>
>> Hi Josh,
>>
>> >Second use case: Distributed locking
>> This is one of the most common uses of ZooKeeper. There are many
>> implementations - one included with the ZK distro. Also, there is
>>Curator:
>> https://github.com/Netflix/curator
>>
>> >First use case: Distributing work to a cluster of nodes
>> This sounds feasible. If you give more details I and others on this list
>> can help more.
>>
>
>Sure. I basically want to handle race conditions where two commands that
>operate on the same data are received by my cluster of znodes,
>concurrently. One approach is to lock on the data that is effected by the
>command (distributed lock). Another approach is make sure that all of the
>commands that operate on any set of data are routed to the same node,
>where
>they can be processed serially using local synchronization. Consistent
>hashing is an algorithm that can be used to select a node to handle a
>message (where the inputs are the key to hash and the number of nodes in
>the cluster).
>
>There are various implementations for this floating around. I'm just
>interesting to know how this is working for anyone else.
>
>Josh
>
>
>>
>> -JZ
>>
>> ________________________________________
>> From: Josh Stone [[EMAIL PROTECTED]]
>> Sent: Wednesday, January 04, 2012 8:09 PM
>> To: [EMAIL PROTECTED]
>> Subject: Use cases for ZooKeeper
>>
>> I have a few use cases that I'm wondering if ZooKeeper would be suitable
>> for and would appreciate some feedback.
>>
>> First use case: Distributing work to a cluster of nodes using consistent
>> hashing to ensure that messages of some type are consistently handled by
>> the same node. I haven't been able to find any info about ZooKeeper +
>> consistent hashing. Is anyone using it for this? A concern here would be
>> how to redistribute work as nodes come and go from the cluster.
>>
>> Second use case: Distributed locking. I noticed that there's a recipe
>>for
>> this on the ZooKeeper wiki. Is anyone doing this? Any issues? One
>>concern
>> would be how to handle orphaned locks if a node that obtained a lock
>>goes
>> down.
>>
>> Third use case: Fault tolerance. If we utilized ZooKeeper to distribute
>> messages to workers, can it be made to handle a node going down by
>> re-distributing the work to another node (perhaps messages that are not
>> ack'ed within a timeout are resent)?
>>
>> Cheers,
>> Josh
>>