Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> The State of Python Zookeeper libraries and collaboration


Copy link to this message
-
Re: The State of Python Zookeeper libraries and collaboration
A pure Python implementation shouldn't be needed for gevent. It should be
sufficient to run the Zookeeper client in async mode in a background thread
(with mt or st.. personally I prefer st), and then use an eventfd() or a
pipe() to notify gevent.

In fact, if you look into gevent's thread pool library - they're using
libev's async signal which is also based on eventfd() and pipe(). So it's
not a new trick.

Best Regards,
Martin Kou

On Thu, May 17, 2012 at 9:24 AM, Mark Gius <[EMAIL PROTECTED]> wrote:

> Are you planning on writing a pure-python client (does not call out to the
> C bindings via zkpython) or are you planning on writing a solid wrapper
> around the C bindings. Implementing a pure-python client would go a long
> way towards making various green thread frameworks work without having to
> jump through hoops.  I think we'd have to add support to Jute so that it
> would generate python data classes kind of like it does now with Java and
> C.
>
> Assuming you go with a wrapper around the C bindings, I would suggest you
> take a look at something called "xthread.py", which was a thread
> synchronization primitives library that a guy proposed to the eventlet
> project a while back and provides Lock, Notify, etc etc which are safe to
> use and notify between real threads and green threads.  It gives a safe way
> to deal with sending data and doing proper locks without having to worry
> about calling out to "green" things from within non greened contexts (such
> as the zk callback functions).  It's eventlet specific, but the concepts
> and probably fair amount of the code can probably be adapted and use.
>
> Or pure-python.  That works too. :D
>
> Mark
>
> On Wed, May 16, 2012 at 5:50 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>
> > It would seem that about 6 months or ago or so, there wasn't much out
> > there in terms of higher level Python libs for Zookeeper. There was the
> > Cloudera article on queues, and txzookeeper (which I'm sure many of us
> > not using twisted immediately ignored).
> >
> > In the time since, several people including myself needed solutions
> > involving Zookeeper with Python and seeing nothing out there all
> > apparently began writing libraries (judging from the project timelines
> > in most cases). I've been collaborating with the author of zc.zk (Jim
> > Fulton) for awhile and we decided it'd make more sense to merge our
> > efforts. In this spirit I began contacting all the other developers to
> > gauge their interest and most have been interested.
> >
> > I created a python-zk organization on GitHub to be the home for this
> > effort and moved over the zc.zk library (which people apparently had a
> > hard time locating), along with the fairly widely used staticly compiled
> > Python Zookeeper binding.
> >
> > https://github.com/python-zk
> >
> > Next up is to create the new merged core which I plan on basing mostly
> > around the cleanest implementation I have seen so far (which also
> > happens to be one of the only gevent compatible ones), kazoo. I've
> > talked with the primary author of Kazoo, and the name may remain with
> > the new merged package or it may get a new name if that doesn't work.
> > I'm not terribly tied to names as much as I am to solid, well tested,
> > well documented working code... but having catchy names does seem to
> help.
> >
> > I'm currently working on this full-time, so I expect it to be in a
> > usable state in a week or so (hopefully not too optimistic). If you're
> > interested in helping out, the more the better, please feel free to
> > e-mail me directly or respond here.
> >
> > This stuff is complex, it needs many eyes on it and lots of code review.
> >
> > This hopefully explains why I'm so interested in having a single Python
> > Zookeeper library along similar caliber to Netflix's Curator that has:
> > - Very thorough unit/integration tests (100% coverage minimum)
> > - Cleanly handles connection loss
> > - Works under gevent or threaded/blocking
> > - Very well documented (API docs and narrative)