Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - Is ZooKeeper the right tool for our needs?


Copy link to this message
-
Re: Is ZooKeeper the right tool for our needs?
Patrick Hunt 2012-06-04, 22:33
This paper from Usenix might shed some light as well, esp around how
things work and why we do what we do (i.e. majority quorums):
http://static.usenix.org/event/usenix10/tech/full_papers/Hunt.pdf

Good Luck,

Patrick

On Mon, Jun 4, 2012 at 3:23 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
>> However, the
>> ZooKeeper client connects to a ZooKeeper server at random. This is another
>> indication that ZooKeeper might be the wrong tool to use.
>
> You seed the client with the list of servers to connect to. If you
> only seed it with one server, it will only connect to one server even
> if there are others in the quorum, so this should not be an issue.
>
> As for your questions, you're definitely pushing the envelope on the
> standard deployment model but ZK is probably the right tool for you to
> use if you care about correctness in your distributed locking. Using
> ZK for distributed locking is one of its key uses. Embedding it inside
> an application is not common but not unheard of. The danger with
> embedding ZK is that you now have another system that can cause ZK to
> perform poorly due to GC, cpu contention, etc. Is there some reason
> you're concerned with running ZK as a separate VM? It won't be highly
> available, but it sounds like running a standalone ZK on a separate
> server than your two system servers might be the best choice for you
> right now. I would probably recommend this over trying to run an
> embedded quorum.
>
> C
>
> On Mon, Jun 4, 2012 at 5:44 PM, David Nickerson <
> [EMAIL PROTECTED]> wrote:
>
>> I work for a small software company and I have been tasked with researching
>> a distributed lock manager for us to use. I would appreciate any feedback
>> on whether or not ZooKeeper is the right choice.
>>
>> Our software solution utilizes multiple servers, but only the Java
>> application called "Administrator" needs to acquire locks. The server that
>> Administrator runs on is called the System Server, which presents a web
>> interface to potentially thousands of users, and is highly multi-threaded.
>>
>> So far, we have used only one System Server, and we have needed
>> locks only among threads in one application. However, we are looking to
>> scale the number of System Servers, and therefore need a distributed lock
>> manager to be shared among them.
>>
>> We don't want to set up another server just for ZooKeeper, and we don't
>> want the System Server to run another instance of the JVM just for
>> ZooKeeper, so we have decided to embed the ZooKeeper server in our
>> Administrator application.
>>
>> If the System Server goes down for whatever reason, we no longer have a
>> need for locks, so it's not a problem if ZooKeeper goes down too. However,
>> if we have two System Servers, we would like to be able to fail over to the
>> second server. However, ZooKeeper requires that a majority of the ZooKeeper
>> servers be up in order to run. This requirement seems strange and
>> worrisome. For the time being, we can just run an additional ZooKeeper
>> server on another server to give ZooKeeper the redundancy it needs.
>>
>> Another issue is that we would like to have fast locks and reduced network
>> traffic by being able to read from the local ZooKeeper server. However, the
>> ZooKeeper client connects to a ZooKeeper server at random. This is another
>> indication that ZooKeeper might be the wrong tool to use.
>>
>> An important feature that we need is deadlock detection. I had to write
>> this myself, but I was able to do so using the interface that ZooKeeper
>> provides.
>>
>> We also need to be able to administrate the ZooKeeper server from our
>> program. For example, if the ZooKeeper server has an error or we need to
>> shut down the System Server, then the program may need to shutdown the
>> ZooKeeper server or restart it. Some code to do this might be:
>>    // To start the server
>>    QuorumPeerMain main;
>>    try {
>>      QuorumPeerConfig config = new QuorumPeerConfig();