Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> The State of Python Zookeeper libraries and collaboration


Copy link to this message
-
Re: The State of Python Zookeeper libraries and collaboration
A pure Python implementation shouldn't be needed for gevent. It should be
sufficient to run the Zookeeper client in async mode in a background thread
(with mt or st.. personally I prefer st), and then use an eventfd() or a
pipe() to notify gevent.

In fact, if you look into gevent's thread pool library - they're using
libev's async signal which is also based on eventfd() and pipe(). So it's
not a new trick.

Best Regards,
Martin Kou

On Thu, May 17, 2012 at 9:24 AM, Mark Gius <[EMAIL PROTECTED]> wrote:

> Are you planning on writing a pure-python client (does not call out to the
> C bindings via zkpython) or are you planning on writing a solid wrapper
> around the C bindings. Implementing a pure-python client would go a long
> way towards making various green thread frameworks work without having to
> jump through hoops.  I think we'd have to add support to Jute so that it
> would generate python data classes kind of like it does now with Java and
> C.
>
> Assuming you go with a wrapper around the C bindings, I would suggest you
> take a look at something called "xthread.py", which was a thread
> synchronization primitives library that a guy proposed to the eventlet
> project a while back and provides Lock, Notify, etc etc which are safe to
> use and notify between real threads and green threads.  It gives a safe way
> to deal with sending data and doing proper locks without having to worry
> about calling out to "green" things from within non greened contexts (such
> as the zk callback functions).  It's eventlet specific, but the concepts
> and probably fair amount of the code can probably be adapted and use.
>
> Or pure-python.  That works too. :D
>
> Mark
>
> On Wed, May 16, 2012 at 5:50 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>
> > It would seem that about 6 months or ago or so, there wasn't much out
> > there in terms of higher level Python libs for Zookeeper. There was the
> > Cloudera article on queues, and txzookeeper (which I'm sure many of us
> > not using twisted immediately ignored).
> >
> > In the time since, several people including myself needed solutions
> > involving Zookeeper with Python and seeing nothing out there all
> > apparently began writing libraries (judging from the project timelines
> > in most cases). I've been collaborating with the author of zc.zk (Jim
> > Fulton) for awhile and we decided it'd make more sense to merge our
> > efforts. In this spirit I began contacting all the other developers to
> > gauge their interest and most have been interested.
> >
> > I created a python-zk organization on GitHub to be the home for this
> > effort and moved over the zc.zk library (which people apparently had a
> > hard time locating), along with the fairly widely used staticly compiled
> > Python Zookeeper binding.
> >
> > https://github.com/python-zk
> >
> > Next up is to create the new merged core which I plan on basing mostly
> > around the cleanest implementation I have seen so far (which also
> > happens to be one of the only gevent compatible ones), kazoo. I've
> > talked with the primary author of Kazoo, and the name may remain with
> > the new merged package or it may get a new name if that doesn't work.
> > I'm not terribly tied to names as much as I am to solid, well tested,
> > well documented working code... but having catchy names does seem to
> help.
> >
> > I'm currently working on this full-time, so I expect it to be in a
> > usable state in a week or so (hopefully not too optimistic). If you're
> > interested in helping out, the more the better, please feel free to
> > e-mail me directly or respond here.
> >
> > This stuff is complex, it needs many eyes on it and lots of code review.
> >
> > This hopefully explains why I'm so interested in having a single Python
> > Zookeeper library along similar caliber to Netflix's Curator that has:
> > - Very thorough unit/integration tests (100% coverage minimum)
> > - Cleanly handles connection loss
> > - Works under gevent or threaded/blocking
> > - Very well documented (API docs and narrative)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB