Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> The State of Python Zookeeper libraries and collaboration

Ben Bangert 2012-05-17, 00:50
Copy link to this message
Re: The State of Python Zookeeper libraries and collaboration
Are you planning on writing a pure-python client (does not call out to the
C bindings via zkpython) or are you planning on writing a solid wrapper
around the C bindings. Implementing a pure-python client would go a long
way towards making various green thread frameworks work without having to
jump through hoops.  I think we'd have to add support to Jute so that it
would generate python data classes kind of like it does now with Java and C.

Assuming you go with a wrapper around the C bindings, I would suggest you
take a look at something called "xthread.py", which was a thread
synchronization primitives library that a guy proposed to the eventlet
project a while back and provides Lock, Notify, etc etc which are safe to
use and notify between real threads and green threads.  It gives a safe way
to deal with sending data and doing proper locks without having to worry
about calling out to "green" things from within non greened contexts (such
as the zk callback functions).  It's eventlet specific, but the concepts
and probably fair amount of the code can probably be adapted and use.

Or pure-python.  That works too. :D


On Wed, May 16, 2012 at 5:50 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:

> It would seem that about 6 months or ago or so, there wasn't much out
> there in terms of higher level Python libs for Zookeeper. There was the
> Cloudera article on queues, and txzookeeper (which I'm sure many of us
> not using twisted immediately ignored).
> In the time since, several people including myself needed solutions
> involving Zookeeper with Python and seeing nothing out there all
> apparently began writing libraries (judging from the project timelines
> in most cases). I've been collaborating with the author of zc.zk (Jim
> Fulton) for awhile and we decided it'd make more sense to merge our
> efforts. In this spirit I began contacting all the other developers to
> gauge their interest and most have been interested.
> I created a python-zk organization on GitHub to be the home for this
> effort and moved over the zc.zk library (which people apparently had a
> hard time locating), along with the fairly widely used staticly compiled
> Python Zookeeper binding.
> https://github.com/python-zk
> Next up is to create the new merged core which I plan on basing mostly
> around the cleanest implementation I have seen so far (which also
> happens to be one of the only gevent compatible ones), kazoo. I've
> talked with the primary author of Kazoo, and the name may remain with
> the new merged package or it may get a new name if that doesn't work.
> I'm not terribly tied to names as much as I am to solid, well tested,
> well documented working code... but having catchy names does seem to help.
> I'm currently working on this full-time, so I expect it to be in a
> usable state in a week or so (hopefully not too optimistic). If you're
> interested in helping out, the more the better, please feel free to
> e-mail me directly or respond here.
> This stuff is complex, it needs many eyes on it and lots of code review.
> This hopefully explains why I'm so interested in having a single Python
> Zookeeper library along similar caliber to Netflix's Curator that has:
> - Very thorough unit/integration tests (100% coverage minimum)
> - Cleanly handles connection loss
> - Works under gevent or threaded/blocking
> - Very well documented (API docs and narrative)
> - Implements all the Zookeeper recipes
> - Service Discovery/Management
> - Higher level utility functions for common Zookeeper tasks
> In the mean-time, here is a summary of my research efforts and code
> review (if something isn't accurate, please feel free to correct).
> Please don't take this as a critique, I'm just trying to document what
> is out there for my own reference on merging and hopefully so other
> people coming along don't continue to replicate this. :)
> gevent-zookeeper
>    - https://github.com/jrydberg/gevent-zookeeper/
>    - Works under gevent
>    - No tests
Ben Bangert 2012-05-17, 21:41
Alan D. Cabrera 2012-05-18, 03:24
Mark Gius 2012-05-18, 17:11
Alan D. Cabrera 2012-06-03, 17:44
Patrick Hunt 2012-05-17, 22:46
Martin Kou 2012-05-20, 19:29
Ben Bangert 2012-05-21, 19:58
Mark Gius 2012-05-21, 20:23
Duncan Findlay 2012-05-22, 04:17