Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> The State of Python Zookeeper libraries and collaboration


Copy link to this message
-
Re: The State of Python Zookeeper libraries and collaboration
On 5/20/12 12:29 PM, Martin Kou wrote:
> A pure Python implementation shouldn't be needed for gevent. It should be
> sufficient to run the Zookeeper client in async mode in a background thread
> (with mt or st.. personally I prefer st), and then use an eventfd() or a
> pipe() to notify gevent.
>
> In fact, if you look into gevent's thread pool library - they're using
> libev's async signal which is also based on eventfd() and pipe(). So it's
> not a new trick.

The current kazoo implementation uses something like this to pass off
the events. Also, under a test script, it *seems* to work if the ZK
thread plants a lambda onto a gevent Queue for a gevent greenlet worker
to then execute.

Here's an example script testing it (when the use_gevent is set to
False, the other greenlet fails to print as its using a blocking Queue
of course):
http://paste.ofcode.org/37qxZAYzqYe43zbuFa3DqNf

I haven't tried this out with the Python ZK lib, do you know offhand if
this is going to run into any issue with gevent having a problem with
items being added to its queue from the other thread?

I'd like to avoid the extra complexity of the pipe's and such if this is
going to work (which the test script seems to). I was also going with
this approach to ensure sequential watch execution when used with gevent
(and the watch func could spawn itself as a new greenlet if it needs to
yield, etc.)

--
Ben Bangert
(ben@ || http://) groovie.org