Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - The State of Python Zookeeper libraries and collaboration

Copy link to this message
Re: The State of Python Zookeeper libraries and collaboration
Mark Gius 2012-05-21, 20:23
On Mon, May 21, 2012 at 12:58 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:

> On 5/20/12 12:29 PM, Martin Kou wrote:
> > A pure Python implementation shouldn't be needed for gevent. It should be
> > sufficient to run the Zookeeper client in async mode in a background
> thread
> > (with mt or st.. personally I prefer st), and then use an eventfd() or a
> > pipe() to notify gevent.
> >
> > In fact, if you look into gevent's thread pool library - they're using
> > libev's async signal which is also based on eventfd() and pipe(). So it's
> > not a new trick.
> The current kazoo implementation uses something like this to pass off
> the events. Also, under a test script, it *seems* to work if the ZK
> thread plants a lambda onto a gevent Queue for a gevent greenlet worker
> to then execute.
> Here's an example script testing it (when the use_gevent is set to
> False, the other greenlet fails to print as its using a blocking Queue
> of course):
> http://paste.ofcode.org/37qxZAYzqYe43zbuFa3DqNf
> I haven't tried this out with the Python ZK lib, do you know offhand if
> this is going to run into any issue with gevent having a problem with
> items being added to its queue from the other thread?
> I'd like to avoid the extra complexity of the pipe's and such if this is
> going to work (which the test script seems to). I was also going with
> this approach to ensure sequential watch execution when used with gevent
> (and the watch func could spawn itself as a new greenlet if it needs to
> yield, etc.)
> --
> Ben Bangert
> (ben@ || http://) groovie.org
+1 on avoiding complexity, and wrangling python_zk so that it behaves with
eventlet/gevent is very complex.

The advantage of a pure-python client is that we don't have to play these
tricks to deal with the fact that the multithreaded library produces
Threads which are not Green safe, causing us to play games to ship code
back into a green thread.

The other trick is that there are more than one "green thread" libraries
out there.  gevent, eventlet, twisted, etc.  A pure python implementation
means that we don't have to special case for various libraries.  eventlet
just has to monkey patch the standard library, same with whatever gevent or
twisted do.

I'm happy to help with a client that interfaces with the C bindings, but I
think that a pure-python client would be cleaner and support a wider
variety of deployments.