https://github.com/Netflix/curator
What is Curator?
Curator n: a keeper or custodian of a museum or other collection - A
ZooKeeper Keeper.

Curator is a set of Java libraries that make using Apache ZooKeeper much
easier. While ZooKeeper comes bundled with a Java client, using the client
is non-trivial and error prone.

Components
Client - A replacement for the bundled ZooKeeper class that takes care of
some low-level housekeeping and provides some useful utilities

Framework - The Curator Framework is a high-level API that greatly
simplifies using ZooKeeper. It adds many features that build on ZooKeeper
and handles the complexity of managing connections to the ZooKeeper
cluster and retrying operations.

Recipes - Implementations of some of the common ZooKeeper "recipes". The
implementations are built on top of the Curator Framework

Utilities - Various utilities that are useful when using ZooKeeper.

Nice.  Good to see the Apache License.
mahadev

On Mon, Oct 10, 2011 at 2:22 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> https://github.com/Netflix/curator
>
>
> What is Curator?
> Curator n: a keeper or custodian of a museum or other collection - A
> ZooKeeper Keeper.
>
> Curator is a set of Java libraries that make using Apache ZooKeeper much
> easier. While ZooKeeper comes bundled with a Java client, using the client
> is non-trivial and error prone.
>
> Components
> Client - A replacement for the bundled ZooKeeper class that takes care of
> some low-level housekeeping and provides some useful utilities
>
> Framework - The Curator Framework is a high-level API that greatly
> simplifies using ZooKeeper. It adds many features that build on ZooKeeper
> and handles the complexity of managing connections to the ZooKeeper
> cluster and retrying operations.
>
> Recipes - Implementations of some of the common ZooKeeper "recipes". The
> implementations are built on top of the Curator Framework
>
> Utilities - Various utilities that are useful when using ZooKeeper.
>
>
>

We'd appreciate any/all feedback, BTW.

-JZ

On 10/11/11 1:04 PM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:

>Nice.  Good to see the Apache License.
>
>
>mahadev
>
>On Mon, Oct 10, 2011 at 2:22 PM, Jordan Zimmerman
><[EMAIL PROTECTED]>wrote:
>
>> https://github.com/Netflix/curator
>>
>>
>> What is Curator?
>> Curator n: a keeper or custodian of a museum or other collection - A
>> ZooKeeper Keeper.
>>
>> Curator is a set of Java libraries that make using Apache ZooKeeper much
>> easier. While ZooKeeper comes bundled with a Java client, using the
>>client
>> is non-trivial and error prone.
>>
>> Components
>> Client - A replacement for the bundled ZooKeeper class that takes care
>>of
>> some low-level housekeeping and provides some useful utilities
>>
>> Framework - The Curator Framework is a high-level API that greatly
>> simplifies using ZooKeeper. It adds many features that build on
>>ZooKeeper
>> and handles the complexity of managing connections to the ZooKeeper
>> cluster and retrying operations.
>>
>> Recipes - Implementations of some of the common ZooKeeper "recipes". The
>> implementations are built on top of the Curator Framework
>>
>> Utilities - Various utilities that are useful when using ZooKeeper.
>>
>>
>>

You asked!

I started in on reading the code last night.  I don't know if I will have
much more time to go deeper and I apologize for any superficiality of these
reactions.

My first blush is that the the builder style API is not a big win (styles
will vary of course).

I like that you have common recipes, but any time anybody starts talking
global locks in a cluster, I get hives and start looking for the fundamental
error in the interaction model.  Locks are typically a total disaster for
system reliability, throughput and probability of correct coding.

Insofar as specifics are concerned, I looked at the atomic update stuff and
don't like the fast fallback to a lock.  I think that limited randomized
exponential backoff is likely to be a more scalable solution if only because
the lock is hard to integrate into the protocol without, well, putting a
lock into the protocol (see above for my opinion on that).  Incorporating
priority of the update into the backoff time constant is one way to avoid
starvation.

What I prefer to see in this context is a method that takes a closure that
does the desired mutation and which encapsulates the necessary read, modify,
write, retry logic.

On Tue, Oct 11, 2011 at 8:18 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> We'd appreciate any/all feedback, BTW.
>
> -JZ
>
> On 10/11/11 1:04 PM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:
>
> >Nice.  Good to see the Apache License.
> >
> >
> >mahadev
> >
> >On Mon, Oct 10, 2011 at 2:22 PM, Jordan Zimmerman
> ><[EMAIL PROTECTED]>wrote:
> >
> >> https://github.com/Netflix/curator
> >>
> >>
> >> What is Curator?
> >> Curator n: a keeper or custodian of a museum or other collection - A
> >> ZooKeeper Keeper.
> >>
> >> Curator is a set of Java libraries that make using Apache ZooKeeper much
> >> easier. While ZooKeeper comes bundled with a Java client, using the
> >>client
> >> is non-trivial and error prone.
> >>
> >> Components
> >> Client - A replacement for the bundled ZooKeeper class that takes care
> >>of
> >> some low-level housekeeping and provides some useful utilities
> >>
> >> Framework - The Curator Framework is a high-level API that greatly
> >> simplifies using ZooKeeper. It adds many features that build on
> >>ZooKeeper
> >> and handles the complexity of managing connections to the ZooKeeper
> >> cluster and retrying operations.
> >>
> >> Recipes - Implementations of some of the common ZooKeeper "recipes". The
> >> implementations are built on top of the Curator Framework
> >>
> >> Utilities - Various utilities that are useful when using ZooKeeper.
> >>
> >>
> >>
>
>

That's a criticism of ZooKeeper, not Curator.

On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:

>any time anybody starts talking
>global locks in a cluster, I get hives and start looking for the
>fundamental
>error in the interaction model.

The lock is optional. An exponential (with randomization) retry is
available.

On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:

>I think that limited randomized
>exponential backoff is likely to be a more scalable solution

The major raison d'être for Curator is to make adding "recipes" much
easier. Using the Curator Framework APIs you can easily build new
usages/recipes and not worry about connection management.

-JZ

On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:

>What I prefer to see in this context is a method that takes a closure that
>does the desired mutation and which encapsulates the necessary read,
>modify,
>write, retry logic.

It actually is a criticism of any design that uses locks at scale larger
than about 10 nodes.

On Tue, Oct 11, 2011 at 8:50 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> That's a criticism of ZooKeeper, not Curator.
>
> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
> >any time anybody starts talking
> >global locks in a cluster, I get hives and start looking for the
> >fundamental
> >error in the interaction model.
>
>

Ahh... I am so glad that I mentioned my review was cursory.  It looked like
the default was to lock after a single failure.

On Tue, Oct 11, 2011 at 8:51 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> The lock is optional. An exponential (with randomization) retry is
> available.
>
> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
> >I think that limited randomized
> >exponential backoff is likely to be a more scalable solution
>
>

My first reaction to the code, also at a high-level browse, was that it is
way over-complicated. Might just be the builder style, which I also don't
care for. I had to go several levels deep through interfaces and impls to
see the code that does anything of meaning, which makes for a high learning
curve and a debugging headache.

So, it's not clear to me why anyone would use this over zkClient, except
that you have provided a pluggable retry style, which is nice. Any other
major benefits I'm missing from a scan?

C
On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]> wrote:

> The major raison d'être for Curator is to make adding "recipes" much
> easier. Using the Curator Framework APIs you can easily build new
> usages/recipes and not worry about connection management.
>
> -JZ
>
> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
> >What I prefer to see in this context is a method that takes a closure that
> >does the desired mutation and which encapsulates the necessary read,
> >modify,
> >write, retry logic.
>
>

It amazes me that people comment on this or that library that it's too complicated but then don't blanch to use things such as Maven, Guice, Spring, Hibernate, etc.

Issues with zkClient:
* Very poor documentation (is there any?)
* Far too coupled for my taste. zkClient covers far too much scope.
* Error handling is very weak (everything becomes RuntimeException)
* Hard coded retry mechanism
* Some bad anti-patterns
** ZkClient constructor leaks "this" ref
** lots of synchronized/lock blocks

Curator is much more modular. You could use just the Client package (which isn't complicated at all). The recipes are completely de-coupled making them more reusable/extendable. I believe a new ZK user would have a much easier time with Curator than zkClient. My experience with users at Netflix confirms this. Our users can concentrate on the recipes (which is what they're really interested in) and not worry about the details of managing ZK interaction.

-JZ
________________________________________
From: Camille Fournier [[EMAIL PROTECTED]]
Sent: Tuesday, October 11, 2011 4:00 PM
To: [EMAIL PROTECTED]
Subject: Re: ANN: Curator - Netflix's ZooKeeper library

My first reaction to the code, also at a high-level browse, was that it is
way over-complicated. Might just be the builder style, which I also don't
care for. I had to go several levels deep through interfaces and impls to
see the code that does anything of meaning, which makes for a high learning
curve and a debugging headache.

So, it's not clear to me why anyone would use this over zkClient, except
that you have provided a pluggable retry style, which is nice. Any other
major benefits I'm missing from a scan?

C
On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]> wrote:

> The major raison d'être for Curator is to make adding "recipes" much
> easier. Using the Curator Framework APIs you can easily build new
> usages/recipes and not worry about connection management.
>
> -JZ
>
> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
> >What I prefer to see in this context is a method that takes a closure that
> >does the desired mutation and which encapsulates the necessary read,
> >modify,
> >write, retry logic.
>
>

Its not written in Scala :( otherwise the contribution is great in having recipes for more quick start ready, try to get more new users, to build right away but I like the lower level access at times (myself as a zk dev user) I need to find time to release the twitter Scala client I have been using and have grown from their wrapper.

Nothing bad, some good stuff.  First post to this list for me.  Thanks to all an everyone for making a kickass server I use everyday.  The client Netflix released would have helped my initial learning curve but I would have ended up where I am now the same.

/*
Joe Stein
http://www.medialets.com
Twitter: @allthingshadoop
*/

On Oct 11, 2011, at 7:00 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:

> My first reaction to the code, also at a high-level browse, was that it is
> way over-complicated. Might just be the builder style, which I also don't
> care for. I had to go several levels deep through interfaces and impls to
> see the code that does anything of meaning, which makes for a high learning
> curve and a debugging headache.
>
> So, it's not clear to me why anyone would use this over zkClient, except
> that you have provided a pluggable retry style, which is nice. Any other
> major benefits I'm missing from a scan?
>
> C
> On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]> wrote:
>
>> The major raison d'être for Curator is to make adding "recipes" much
>> easier. Using the Curator Framework APIs you can easily build new
>> usages/recipes and not worry about connection management.
>>
>> -JZ
>>
>> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>>
>>> What I prefer to see in this context is a method that takes a closure that
>>> does the desired mutation and which encapsulates the necessary read,
>>> modify,
>>> write, retry logic.
>>
>>

You asked for feedback, I gave it. You know nothing about what I think
of maven, guice, spring, or hibernate, so why bring them up?

There's a middle ground between Curator and zkClient, and I personally
would love to see that middle ground reached. Anyway, I've never made
a decision to use a library or not based on the conditions you listed
below (unless there were actual usability issues related to them,
which for me there weren't in zkClient), so I'm pretty sure I am not
your target audience. But I know there are many here that would be
turned off by those reasons and it is great to give them a different
choice.

C

On Tue, Oct 11, 2011 at 7:34 PM, Jordan Zimmerman
<[EMAIL PROTECTED]> wrote:
> It amazes me that people comment on this or that library that it's too complicated but then don't blanch to use things such as Maven, Guice, Spring, Hibernate, etc.
>
> Issues with zkClient:
> * Very poor documentation (is there any?)
> * Far too coupled for my taste. zkClient covers far too much scope.
> * Error handling is very weak (everything becomes RuntimeException)
> * Hard coded retry mechanism
> * Some bad anti-patterns
> ** ZkClient constructor leaks "this" ref
> ** lots of synchronized/lock blocks
>
> Curator is much more modular. You could use just the Client package (which isn't complicated at all). The recipes are completely de-coupled making them more reusable/extendable. I believe a new ZK user would have a much easier time with Curator than zkClient. My experience with users at Netflix confirms this. Our users can concentrate on the recipes (which is what they're really interested in) and not worry about the details of managing ZK interaction.
>
> -JZ
> ________________________________________
> From: Camille Fournier [[EMAIL PROTECTED]]
> Sent: Tuesday, October 11, 2011 4:00 PM
> To: [EMAIL PROTECTED]
> Subject: Re: ANN: Curator - Netflix's ZooKeeper library
>
> My first reaction to the code, also at a high-level browse, was that it is
> way over-complicated. Might just be the builder style, which I also don't
> care for. I had to go several levels deep through interfaces and impls to
> see the code that does anything of meaning, which makes for a high learning
> curve and a debugging headache.
>
> So, it's not clear to me why anyone would use this over zkClient, except
> that you have provided a pluggable retry style, which is nice. Any other
> major benefits I'm missing from a scan?
>
> C
> On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]> wrote:
>
>> The major raison d'être for Curator is to make adding "recipes" much
>> easier. Using the Curator Framework APIs you can easily build new
>> usages/recipes and not worry about connection management.
>>
>> -JZ
>>
>> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>>
>> >What I prefer to see in this context is a method that takes a closure that
>> >does the desired mutation and which encapsulates the necessary read,
>> >modify,
>> >write, retry logic.
>>
>>
>

Sorry. I didn't mean to come across as prickly.

===================Jordan Zimmerman

On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]> wrote:

> You asked for feedback, I gave it.

Don't worry about being prickly.  Camille and I can beat you any day on that
account.

Just be good hearted and serious about making things good while you are
being prickly.

On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> Sorry. I didn't mean to come across as prickly.
>
> ===================> Jordan Zimmerman
>
> On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]>
> wrote:
>
> > You asked for feedback, I gave it.
>

Good <3 ed. :)

===================Jordan Zimmerman

On Oct 11, 2011, at 7:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:

> Don't worry about being prickly.  Camille and I can beat you any day on that
> account.
>
> Just be good hearted and serious about making things good while you are
> being prickly.
>
> On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:
>
>> Sorry. I didn't mean to come across as prickly.
>>
>> ===================>> Jordan Zimmerman
>>
>> On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]>
>> wrote:
>>
>>> You asked for feedback, I gave it.
>>

It might be possible to put a Scala wrapper around the recipe classes. I'll see what I can do - or if you want to contribute that ;)

===================Jordan Zimmerman

On Oct 11, 2011, at 4:40 PM, "Joe Stein" <[EMAIL PROTECTED]> wrote:

> Its not written in Scala :( otherwise the contribution is great in having recipes for more quick start ready, try to get more new users, to build right away but I like the lower level access at times (myself as a zk dev user) I need to find time to release the twitter Scala client I have been using and have grown from their wrapper.
>
> Nothing bad, some good stuff.  First post to this list for me.  Thanks to all an everyone for making a kickass server I use everyday.  The client Netflix released would have helped my initial learning curve but I would have ended up where I am now the same.
>
> /*
> Joe Stein
> http://www.medialets.com
> Twitter: @allthingshadoop
> */
>
> On Oct 11, 2011, at 7:00 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
>
>> My first reaction to the code, also at a high-level browse, was that it is
>> way over-complicated. Might just be the builder style, which I also don't
>> care for. I had to go several levels deep through interfaces and impls to
>> see the code that does anything of meaning, which makes for a high learning
>> curve and a debugging headache.
>>
>> So, it's not clear to me why anyone would use this over zkClient, except
>> that you have provided a pluggable retry style, which is nice. Any other
>> major benefits I'm missing from a scan?
>>
>> C
>> On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]> wrote:
>>
>>> The major raison d'être for Curator is to make adding "recipes" much
>>> easier. Using the Curator Framework APIs you can easily build new
>>> usages/recipes and not worry about connection management.
>>>
>>> -JZ
>>>
>>> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>>>
>>>> What I prefer to see in this context is a method that takes a closure that
>>>> does the desired mutation and which encapsulates the necessary read,
>>>> modify,
>>>> write, retry logic.
>>>
>>>
>

I might have some time early next week but the Scala ZK lib I use is built
up from https://github.com/twitter/scala-zookeeper-client and I have been
meaning to take some of the additions I have made implementing that (mostly
new tests and some more boiler plate too) but now maybe I can go through
some of your recipes and come up with something useful for Scala folks (like
I did recently for Cassandra users for Hector, in my humble opinion).

Most likely/realistically I can steal an iteration away from my current
workload the first week of November now that I am afforded time to work on
open source.

On Tue, Oct 11, 2011 at 10:55 PM, Jordan Zimmerman
<[EMAIL PROTECTED]>wrote:

> It might be possible to put a Scala wrapper around the recipe classes. I'll
> see what I can do - or if you want to contribute that ;)
>
> ===================> Jordan Zimmerman
>
> On Oct 11, 2011, at 4:40 PM, "Joe Stein" <[EMAIL PROTECTED]>
> wrote:
>
> > Its not written in Scala :( otherwise the contribution is great in having
> recipes for more quick start ready, try to get more new users, to build
> right away but I like the lower level access at times (myself as a zk dev
> user) I need to find time to release the twitter Scala client I have been
> using and have grown from their wrapper.
> >
> > Nothing bad, some good stuff.  First post to this list for me.  Thanks to
> all an everyone for making a kickass server I use everyday.  The client
> Netflix released would have helped my initial learning curve but I would
> have ended up where I am now the same.
> >
> > /*
> > Joe Stein
> > http://www.medialets.com
> > Twitter: @allthingshadoop
> > */
> >
> > On Oct 11, 2011, at 7:00 PM, Camille Fournier <[EMAIL PROTECTED]>
> wrote:
> >
> >> My first reaction to the code, also at a high-level browse, was that it
> is
> >> way over-complicated. Might just be the builder style, which I also
> don't
> >> care for. I had to go several levels deep through interfaces and impls
> to
> >> see the code that does anything of meaning, which makes for a high
> learning
> >> curve and a debugging headache.
> >>
> >> So, it's not clear to me why anyone would use this over zkClient, except
> >> that you have provided a pluggable retry style, which is nice. Any other
> >> major benefits I'm missing from a scan?
> >>
> >> C
> >> On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> The major raison d'être for Curator is to make adding "recipes" much
> >>> easier. Using the Curator Framework APIs you can easily build new
> >>> usages/recipes and not worry about connection management.
> >>>
> >>> -JZ
> >>>
> >>> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> What I prefer to see in this context is a method that takes a closure
> that
> >>>> does the desired mutation and which encapsulates the necessary read,
> >>>> modify,
> >>>> write, retry logic.
> >>>
> >>>
> >
>

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop
*/

Oh here's one thing you may find useful. I did find your TestingServer
nice and tried it out today for some code I was hacking. I hit a bug
though in my test teardowns, and solved it by adding this to stop
(courtesy ClientBase)

ZKDatabase zkDb = server.getZKDatabase();
     try {
            zkDb.close();
        } catch (IOException ie) {
           System.err.println("Error closing logs " + ie);
        }

In case you hit the same error of being unable to delete the directories.

C

On Tue, Oct 11, 2011 at 10:53 PM, Jordan Zimmerman
<[EMAIL PROTECTED]> wrote:
> Good <3 ed. :)
>
> ===================> Jordan Zimmerman
>
> On Oct 11, 2011, at 7:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
>> Don't worry about being prickly.  Camille and I can beat you any day on that
>> account.
>>
>> Just be good hearted and serious about making things good while you are
>> being prickly.
>>
>> On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:
>>
>>> Sorry. I didn't mean to come across as prickly.
>>>
>>> ===================>>> Jordan Zimmerman
>>>
>>> On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> You asked for feedback, I gave it.
>>>
>

Nice! I'll update the code.

On 10/12/11 5:12 PM, "Camille Fournier" <[EMAIL PROTECTED]> wrote:

>Oh here's one thing you may find useful. I did find your TestingServer
>nice and tried it out today for some code I was hacking. I hit a bug
>though in my test teardowns, and solved it by adding this to stop
>(courtesy ClientBase)
>
> ZKDatabase zkDb = server.getZKDatabase();
>     try {
>            zkDb.close();
>        } catch (IOException ie) {
>           System.err.println("Error closing logs " + ie);
>        }
>
>In case you hit the same error of being unable to delete the directories.
>
>C
>
>On Tue, Oct 11, 2011 at 10:53 PM, Jordan Zimmerman
><[EMAIL PROTECTED]> wrote:
>> Good <3 ed. :)
>>
>> ===================>> Jordan Zimmerman
>>
>> On Oct 11, 2011, at 7:46 PM, "Ted Dunning" <[EMAIL PROTECTED]>
>>wrote:
>>
>>> Don't worry about being prickly.  Camille and I can beat you any day
>>>on that
>>> account.
>>>
>>> Just be good hearted and serious about making things good while you are
>>> being prickly.
>>>
>>> On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman
>>><[EMAIL PROTECTED]>wrote:
>>>
>>>> Sorry. I didn't mean to come across as prickly.
>>>>
>>>> ===================>>>> Jordan Zimmerman
>>>>
>>>> On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> You asked for feedback, I gave it.
>>>>
>>
>

Jordan,

I wanted to chime in and say that we ended up writing a similar library for
our application.  For example we have a BreakableLock that implements
java.util.concurrent.Lock but requires you to provide a BrokenLockListener
implementation. Naming aside it looks pretty similar to your implementaion
with the ClientClosingListener.  We also have similar code for dealing with
path creation, retry etc.   In order to deal with watches we defined a
Reconnectable interface that users can implement.  These reconnectables
are added to a client and called on session expiration or other catastrophic
client failures.  This gives users a clean way to reestablish their watches
and reset internal state while centralizing all of the error handling.

When we get around to refactoring or expanding our code base we'll
definitely take a look at using Curator.  If nothing else it makes us feel
good that someone else solved a lot of the same problems in a similar
fashion.

- will

On Wed, Oct 12, 2011 at 8:17 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> Nice! I'll update the code.
>
> On 10/12/11 5:12 PM, "Camille Fournier" <[EMAIL PROTECTED]> wrote:
>
> >Oh here's one thing you may find useful. I did find your TestingServer
> >nice and tried it out today for some code I was hacking. I hit a bug
> >though in my test teardowns, and solved it by adding this to stop
> >(courtesy ClientBase)
> >
> >       ZKDatabase zkDb = server.getZKDatabase();
> >       try {
> >            zkDb.close();
> >        } catch (IOException ie) {
> >           System.err.println("Error closing logs " + ie);
> >        }
> >
> >In case you hit the same error of being unable to delete the directories.
> >
> >C
> >
> >On Tue, Oct 11, 2011 at 10:53 PM, Jordan Zimmerman
> ><[EMAIL PROTECTED]> wrote:
> >> Good <3 ed. :)
> >>
> >> ===================> >> Jordan Zimmerman
> >>
> >> On Oct 11, 2011, at 7:46 PM, "Ted Dunning" <[EMAIL PROTECTED]>
> >>wrote:
> >>
> >>> Don't worry about being prickly.  Camille and I can beat you any day
> >>>on that
> >>> account.
> >>>
> >>> Just be good hearted and serious about making things good while you are
> >>> being prickly.
> >>>
> >>> On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman
> >>><[EMAIL PROTECTED]>wrote:
> >>>
> >>>> Sorry. I didn't mean to come across as prickly.
> >>>>
> >>>> ===================> >>>> Jordan Zimmerman
> >>>>
> >>>> On Oct 11, 2011, at 4:52 PM, "Camille Fournier" <[EMAIL PROTECTED]>
> >>>> wrote:
> >>>>
> >>>>> You asked for feedback, I gave it.
> >>>>
> >>
> >
>
>

Thanks :)

If you have the time, I'd love a code review. In particular, the
ConnectionState class.

-JZ

On 10/13/11 7:29 AM, "Will Johnson" <[EMAIL PROTECTED]> wrote:

>Jordan,
>
>I wanted to chime in and say that we ended up writing a similar library
>for
>our application.  For example we have a BreakableLock that implements
>java.util.concurrent.Lock but requires you to provide a BrokenLockListener
>implementation. Naming aside it looks pretty similar to your implementaion
>with the ClientClosingListener.  We also have similar code for dealing
>with
>path creation, retry etc.   In order to deal with watches we defined a
>Reconnectable interface that users can implement.  These reconnectables
>are added to a client and called on session expiration or other
>catastrophic
>client failures.  This gives users a clean way to reestablish their
>watches
>and reset internal state while centralizing all of the error handling.
>
>When we get around to refactoring or expanding our code base we'll
>definitely take a look at using Curator.  If nothing else it makes us feel
>good that someone else solved a lot of the same problems in a similar
>fashion.
>
>- will
>
>
>
>On Wed, Oct 12, 2011 at 8:17 PM, Jordan Zimmerman
><[EMAIL PROTECTED]>wrote:
>
>> Nice! I'll update the code.
>>
>> On 10/12/11 5:12 PM, "Camille Fournier" <[EMAIL PROTECTED]> wrote:
>>
>> >Oh here's one thing you may find useful. I did find your TestingServer
>> >nice and tried it out today for some code I was hacking. I hit a bug
>> >though in my test teardowns, and solved it by adding this to stop
>> >(courtesy ClientBase)
>> >
>> >       ZKDatabase zkDb = server.getZKDatabase();
>> >       try {
>> >            zkDb.close();
>> >        } catch (IOException ie) {
>> >           System.err.println("Error closing logs " + ie);
>> >        }
>> >
>> >In case you hit the same error of being unable to delete the
>>directories.
>> >
>> >C
>> >
>> >On Tue, Oct 11, 2011 at 10:53 PM, Jordan Zimmerman
>> ><[EMAIL PROTECTED]> wrote:
>> >> Good <3 ed. :)
>> >>
>> >> ===================>> >> Jordan Zimmerman
>> >>
>> >> On Oct 11, 2011, at 7:46 PM, "Ted Dunning" <[EMAIL PROTECTED]>
>> >>wrote:
>> >>
>> >>> Don't worry about being prickly.  Camille and I can beat you any day
>> >>>on that
>> >>> account.
>> >>>
>> >>> Just be good hearted and serious about making things good while you
>>are
>> >>> being prickly.
>> >>>
>> >>> On Wed, Oct 12, 2011 at 1:18 AM, Jordan Zimmerman
>> >>><[EMAIL PROTECTED]>wrote:
>> >>>
>> >>>> Sorry. I didn't mean to come across as prickly.
>> >>>>
>> >>>> ===================>> >>>> Jordan Zimmerman
>> >>>>
>> >>>> On Oct 11, 2011, at 4:52 PM, "Camille Fournier"
>><[EMAIL PROTECTED]>
>> >>>> wrote:
>> >>>>
>> >>>>> You asked for feedback, I gave it.
>> >>>>
>> >>
>> >
>>
>>

Not to be too buzzword-compliant, but I've found the FSM module in Akka to
be really great for modeling, uhh, FSMs, and the use of ZooKeeper makes one
think a lot in terms of FSMs, so a Scala/Akka/FSM-based client it might be a
fruitful path.

.. Adam

On Tue, Oct 11, 2011 at 9:23 PM, Joe Stein
<[EMAIL PROTECTED]>wrote:

> I might have some time early next week but the Scala ZK lib I use is built
> up from https://github.com/twitter/scala-zookeeper-client and I have been
> meaning to take some of the additions I have made implementing that (mostly
> new tests and some more boiler plate too) but now maybe I can go through
> some of your recipes and come up with something useful for Scala folks
> (like
> I did recently for Cassandra users for Hector, in my humble opinion).
>
> Most likely/realistically I can steal an iteration away from my current
> workload the first week of November now that I am afforded time to work on
> open source.
>
> On Tue, Oct 11, 2011 at 10:55 PM, Jordan Zimmerman
> <[EMAIL PROTECTED]>wrote:
>
> > It might be possible to put a Scala wrapper around the recipe classes.
> I'll
> > see what I can do - or if you want to contribute that ;)
> >
> > ===================> > Jordan Zimmerman
> >
> > On Oct 11, 2011, at 4:40 PM, "Joe Stein" <[EMAIL PROTECTED]
> >
> > wrote:
> >
> > > Its not written in Scala :( otherwise the contribution is great in
> having
> > recipes for more quick start ready, try to get more new users, to build
> > right away but I like the lower level access at times (myself as a zk dev
> > user) I need to find time to release the twitter Scala client I have been
> > using and have grown from their wrapper.
> > >
> > > Nothing bad, some good stuff.  First post to this list for me.  Thanks
> to
> > all an everyone for making a kickass server I use everyday.  The client
> > Netflix released would have helped my initial learning curve but I would
> > have ended up where I am now the same.
> > >
> > > /*
> > > Joe Stein
> > > http://www.medialets.com
> > > Twitter: @allthingshadoop
> > > */
> > >
> > > On Oct 11, 2011, at 7:00 PM, Camille Fournier <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> My first reaction to the code, also at a high-level browse, was that
> it
> > is
> > >> way over-complicated. Might just be the builder style, which I also
> > don't
> > >> care for. I had to go several levels deep through interfaces and impls
> > to
> > >> see the code that does anything of meaning, which makes for a high
> > learning
> > >> curve and a debugging headache.
> > >>
> > >> So, it's not clear to me why anyone would use this over zkClient,
> except
> > >> that you have provided a pluggable retry style, which is nice. Any
> other
> > >> major benefits I'm missing from a scan?
> > >>
> > >> C
> > >> On Oct 11, 2011 4:53 PM, "Jordan Zimmerman" <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> The major raison d'être for Curator is to make adding "recipes" much
> > >>> easier. Using the Curator Framework APIs you can easily build new
> > >>> usages/recipes and not worry about connection management.
> > >>>
> > >>> -JZ
> > >>>
> > >>> On 10/11/11 1:46 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
> > >>>
> > >>>> What I prefer to see in this context is a method that takes a
> closure
> > that
> > >>>> does the desired mutation and which encapsulates the necessary read,
> > >>>> modify,
> > >>>> write, retry logic.
> > >>>
> > >>>
> > >
> >
>
>
>
> --
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop
> */
>

Will Johnson:
> Jordan,
>
> I wanted to chime in and say that we ended up writing a similar library for
> our application.  For example we have a BreakableLock that implements
> java.util.concurrent.Lock but requires you to provide a BrokenLockListener
> implementation. Naming aside it looks pretty similar to your implementaion
> with the ClientClosingListener.  We also have similar code for dealing with
> path creation, retry etc.   In order to deal with watches we defined a
> Reconnectable interface that users can implement.  These reconnectables
> are added to a client and called on session expiration or other
> catastrophic client failures.  This gives users a clean way to reestablish
> their watches and reset internal state while centralizing all of the error
> handling.
>
> When we get around to refactoring or expanding our code base we'll
> definitely take a look at using Curator.  If nothing else it makes us feel
> good that someone else solved a lot of the same problems in a similar
> fashion.
>
> - will
Hi Will,

could you give more information about your client library? I'd like to include
it in my list of work that people were forced to do to work around the
shortcommings of the current ZooKeeper client API. You'd be number four after
zkclient[1], cages[2] and netflix's Curator. (These projects also add
features, but all of them had to work around the same API.)
I've filled ZOOKEEPER-911 that would provide one class for each possible
request to the ZooKeeper server. An instance of such an operation class could
then be passed to an execution strategy.

[1] http://github.com/sgroschupf/zkclient
[2] http://code.google.com/p/cages/

Imagine the equivalent in SQL: This would be a database client library that
forces you to call methods select(), update(), delete(), insert() depending on
what you want to do and throwing exceptions to the calling code.
You don't do this of course but have an object (or SQL literal string)
representing your query and pass this to some execute() method.

We need to wait for 3.4 to come out before work on ZOOKEEPER-911 could
continue.

Regards,

Thomas Koch, http://www.koch.ro

I'm not sure specifically what info you 'd like to know but our code was similar to curator's in high level design for locks and retrys.  One modification we'd like to see is to codify the exception handling cases taken from the wiki into interfaces instead of as an exception handling exercise for developers.  Having a method that throws one of ~20 exception types all subclassing a top level exception leads to a lot of:

Try {
  zooClient.trySomething();
} catch (KeeperException ke) {
  // try to do something smart.  
}

The problem is that there are a number of different things you may want to do but the code/api doesn't make that obvious to the developer.  It's only after reading the wiki that you begin to understand what might happen.  If the client/methods took interfaces to handle each of the error cases and the interfaces made it really clear what just happened and what might happen next developing apps with zookeeper would be much more straightforward (or at least more transparent).  Your retry handler is one such example covering a specific use case but there are others that can be built up from lower level interfaces around each of the exception cases.  Per your design (as i loosely understand it) all of these exception handlers could be passed to the execute method.

- will

-----Original Message-----
From: Thomas Koch [mailto:[EMAIL PROTECTED]]
Sent: Friday, November 04, 2011 12:36 PM
To: [EMAIL PROTECTED]
Cc: Will Johnson
Subject: Re: ANN: Curator - Netflix's ZooKeeper library

Will Johnson:
> Jordan,
>
> I wanted to chime in and say that we ended up writing a similar
> library for our application.  For example we have a BreakableLock that
> implements java.util.concurrent.Lock but requires you to provide a
> BrokenLockListener implementation. Naming aside it looks pretty
> similar to your implementaion with the ClientClosingListener.  We also have similar code for dealing with
> path creation, retry etc.   In order to deal with watches we defined a
> Reconnectable interface that users can implement.  These
> reconnectables are added to a client and called on session expiration
> or other catastrophic client failures.  This gives users a clean way
> to reestablish their watches and reset internal state while
> centralizing all of the error handling.
>
> When we get around to refactoring or expanding our code base we'll
> definitely take a look at using Curator.  If nothing else it makes us
> feel good that someone else solved a lot of the same problems in a
> similar fashion.
>
> - will
Hi Will,

could you give more information about your client library? I'd like to include it in my list of work that people were forced to do to work around the shortcommings of the current ZooKeeper client API. You'd be number four after zkclient[1], cages[2] and netflix's Curator. (These projects also add features, but all of them had to work around the same API.) I've filled ZOOKEEPER-911 that would provide one class for each possible request to the ZooKeeper server. An instance of such an operation class could then be passed to an execution strategy.

[1] http://github.com/sgroschupf/zkclient
[2] http://code.google.com/p/cages/

Imagine the equivalent in SQL: This would be a database client library that forces you to call methods select(), update(), delete(), insert() depending on what you want to do and throwing exceptions to the calling code.
You don't do this of course but have an object (or SQL literal string) representing your query and pass this to some execute() method.

We need to wait for 3.4 to come out before work on ZOOKEEPER-911 could continue.

Regards,

Thomas Koch, http://www.koch.ro

>
>could you give more information about your client library? I'd like to
>include
>it in my list of work that people were forced to do to work around the
>shortcommings of the current ZooKeeper client API. You'd be number four
>after

A lot could be solved by adding a retry (injectable please) mechanism.
Retrying is the basis for most of the Zk wrappers I've seen (and written).
Secondly, high quality implementations of the common recipes.

-JZ