Ben Bangert
2012-08-30, 23:00
Ben Bangert
2012-08-30, 23:02
Michi Mutsuzaki
2012-08-30, 23:50
Ben Bangert
2012-08-31, 00:06
Henry Robinson
2012-08-31, 00:10
Ben Bangert
2012-08-31, 04:00
Ben Bangert
2012-08-31, 05:04
Henry Robinson
2012-08-31, 06:57
Ben Bangert
2012-08-31, 16:39
Michi Mutsuzaki
2012-08-31, 21:30
Ted Dunning
2012-08-31, 20:02
Ben Bangert
2012-08-30, 23:35
Michi Mutsuzaki
2012-08-30, 23:37
-
Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-30, 23:00
Based on the pookeeper code (https://github.com/maguro/pookeeper), I've implemented a version of kazoo that uses the wire protocol and is pure Python (https://github.com/python-zk/kazoo/tree/pure-python). Speaking to Zookeeper directly has been quite interesting. All the documentation assumes one uses the C/Java libs, so what actually happens doesn't seem to be cleanly or clearly defined anywhere beyond the implementations. Is there an actual spec somewhere?
So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
Why does Zookeeper drop the connection with a bad auth packet... but doesn't immediately terminate the session id? If the session ID is not supposed to be used, why not expire it? I wonder how much code out there is written with the assumption that once you see a AUTH_FAILED state, *your session is dead and your nodes are deleted*?
This isn't actually the case though, you get an AUTH_FAILED, the socket is dropped, and a reconnect with the same session id/password works just fine. Why bother dropping the connection to begin with?
If anyone is interested in why I decided to go with pure Python vs. the C lib/binding, I'd be happy to espouse on that in a separate thread.
Cheers,
Ben
So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
Why does Zookeeper drop the connection with a bad auth packet... but doesn't immediately terminate the session id? If the session ID is not supposed to be used, why not expire it? I wonder how much code out there is written with the assumption that once you see a AUTH_FAILED state, *your session is dead and your nodes are deleted*?
This isn't actually the case though, you get an AUTH_FAILED, the socket is dropped, and a reconnect with the same session id/password works just fine. Why bother dropping the connection to begin with?
If anyone is interested in why I decided to go with pure Python vs. the C lib/binding, I'd be happy to espouse on that in a separate thread.
Cheers,
Ben
+
Ben Bangert 2012-08-30, 23:00
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-30, 23:02
On Aug 30, 2012, at 4:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
I should note, this is the 3.3.6 release.
- Ben
> One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
I should note, this is the 3.3.6 release.
- Ben
+
Ben Bangert 2012-08-30, 23:02
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientMichi Mutsuzaki 2012-08-30, 23:50
Hi Ben,
> Based on the pookeeper code (https://github.com/maguro/pookeeper), I've implemented a version of kazoo that uses the wire protocol and is pure Python (https://github.com/python-zk/kazoo/tree/pure-python). Speaking to Zookeeper directly has been quite interesting. All the documentation assumes one uses the C/Java libs, so what actually happens doesn't seem to be cleanly or clearly defined anywhere beyond the implementations. Is there an actual spec somewhere?
I don't think there is an official spec beyond the zookeeper.jute
file. It would be very helpful if you can share what you have found
implementing your python client.
> One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
I think this is expected. ZooKeeper should not expire a session
because of authentication failure. That would make it easier for a
malicious client to expire random sessions. I don't know if there is a
technical reason for dropping the connection though. Maybe it was an
arbitrary decision?
--Michi
> Based on the pookeeper code (https://github.com/maguro/pookeeper), I've implemented a version of kazoo that uses the wire protocol and is pure Python (https://github.com/python-zk/kazoo/tree/pure-python). Speaking to Zookeeper directly has been quite interesting. All the documentation assumes one uses the C/Java libs, so what actually happens doesn't seem to be cleanly or clearly defined anywhere beyond the implementations. Is there an actual spec somewhere?
I don't think there is an official spec beyond the zookeeper.jute
file. It would be very helpful if you can share what you have found
implementing your python client.
> One of the oddities of passing a bad auth packet (ie, saying 'asdfjasdf' instead of 'digest'), is that Zookeeper returns an auth packet -4 with an error header, and then drops the connections. In the state diagram, the client is expected to 'end'. That's fine and all, my client happened to reconnect the first time with the prior session id/password.... which was still valid. Is this expected behavior?
I think this is expected. ZooKeeper should not expire a session
because of authentication failure. That would make it easier for a
malicious client to expire random sessions. I don't know if there is a
technical reason for dropping the connection though. Maybe it was an
arbitrary decision?
--Michi
+
Michi Mutsuzaki 2012-08-30, 23:50
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-31, 00:06
On Aug 30, 2012, at 4:50 PM, Michi Mutsuzaki <[EMAIL PROTECTED]> wrote:
> I don't think there is an official spec beyond the zookeeper.jute
> file. It would be very helpful if you can share what you have found
> implementing your python client.
The most interesting thing I found was that documented C/Java behavior isn't actually enforced beyond the client boundaries. For some reason I had expected the state transitions listed to be some limitation of the Zookeeper protocol itself. So it has had me curious about ways the client implementation itself could make using Zookeeper less error-prone, as Curator and Kazoo respectively try to make working with Zookeeper less error-prone in their languages. This is part of why I was baffled when an AUTH_FAILED resulted in connection termination, but not session expiration.
In the case of kazoo, its now substantially easier to debug what is actually happening since its always within the bounds of Python. Reinstalling debug builds of the C lib and Python C binding got hairy, and still obfuscated a lot. This made debugging the prior password mangling issue in the Python lib a major pain.
I've found that the C lib seems like a bit of a second-class citizen for Zookeeper... the read-only feature has had patches submitted since 2010, which still aren't applied. The Python C binding also has multiple patches... not applied. We applied several of these to fix memory leaks and the password mangling to our zc-zookeeper-static Python lib, but after looking through all the remaining C lib bugs/patches and Python C bugs/patches, its way more than we want to deal with. C isn't my forte, and I'd much much rather debug Java code or read the Java client if I'm curious about how something is done, rather than deal with 2 layers of C code in addition to TCP and the Java server.
Meanwhile, I can implement the new Zookeeper 3.4 methods after looking at the jute code in just minutes, and debugging a problem is trivial when its all Python code. Some people have mentioned that without the C os thread used by the C binding, its possible in heavy Python threading thrashes that a ping might not be sent... which is true, but the session timeout can be increased and thats a very small price to pay given the other things noted here with the C lib/binding.
And of course, for other runtimes (Pypy) that can't run C extensions, the pure Python kazoo will now work.
> I think this is expected. ZooKeeper should not expire a session
> because of authentication failure. That would make it easier for a
> malicious client to expire random sessions. I don't know if there is a
> technical reason for dropping the connection though. Maybe it was an
> arbitrary decision?
Yea, I wasn't terribly sure. The doc state diagram seems to indicate that an AUTH_FAILED should be treated the same as a SESSION_EXPIRATION or CLOSING event. However, your session is dead in the latter two cases, while the session is *not* dead in AUTH_FAILED, yet you end up in the same state. I would not be surprised if a substantial amount of code assumed that the session was dead when an AUTH_FAILED occurred, yet the session is not dead at all.
Cheers,
Ben
> I don't think there is an official spec beyond the zookeeper.jute
> file. It would be very helpful if you can share what you have found
> implementing your python client.
The most interesting thing I found was that documented C/Java behavior isn't actually enforced beyond the client boundaries. For some reason I had expected the state transitions listed to be some limitation of the Zookeeper protocol itself. So it has had me curious about ways the client implementation itself could make using Zookeeper less error-prone, as Curator and Kazoo respectively try to make working with Zookeeper less error-prone in their languages. This is part of why I was baffled when an AUTH_FAILED resulted in connection termination, but not session expiration.
In the case of kazoo, its now substantially easier to debug what is actually happening since its always within the bounds of Python. Reinstalling debug builds of the C lib and Python C binding got hairy, and still obfuscated a lot. This made debugging the prior password mangling issue in the Python lib a major pain.
I've found that the C lib seems like a bit of a second-class citizen for Zookeeper... the read-only feature has had patches submitted since 2010, which still aren't applied. The Python C binding also has multiple patches... not applied. We applied several of these to fix memory leaks and the password mangling to our zc-zookeeper-static Python lib, but after looking through all the remaining C lib bugs/patches and Python C bugs/patches, its way more than we want to deal with. C isn't my forte, and I'd much much rather debug Java code or read the Java client if I'm curious about how something is done, rather than deal with 2 layers of C code in addition to TCP and the Java server.
Meanwhile, I can implement the new Zookeeper 3.4 methods after looking at the jute code in just minutes, and debugging a problem is trivial when its all Python code. Some people have mentioned that without the C os thread used by the C binding, its possible in heavy Python threading thrashes that a ping might not be sent... which is true, but the session timeout can be increased and thats a very small price to pay given the other things noted here with the C lib/binding.
And of course, for other runtimes (Pypy) that can't run C extensions, the pure Python kazoo will now work.
> I think this is expected. ZooKeeper should not expire a session
> because of authentication failure. That would make it easier for a
> malicious client to expire random sessions. I don't know if there is a
> technical reason for dropping the connection though. Maybe it was an
> arbitrary decision?
Yea, I wasn't terribly sure. The doc state diagram seems to indicate that an AUTH_FAILED should be treated the same as a SESSION_EXPIRATION or CLOSING event. However, your session is dead in the latter two cases, while the session is *not* dead in AUTH_FAILED, yet you end up in the same state. I would not be surprised if a substantial amount of code assumed that the session was dead when an AUTH_FAILED occurred, yet the session is not dead at all.
Cheers,
Ben
+
Ben Bangert 2012-08-31, 00:06
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientHenry Robinson 2012-08-31, 00:10
On 30 August 2012 17:06, Ben Bangert <[EMAIL PROTECTED]> wrote:
> On Aug 30, 2012, at 4:50 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>
> wrote:
>
> > I don't think there is an official spec beyond the zookeeper.jute
> > file. It would be very helpful if you can share what you have found
> > implementing your python client.
>
> The most interesting thing I found was that documented C/Java behavior
> isn't actually enforced beyond the client boundaries. For some reason I had
> expected the state transitions listed to be some limitation of the
> Zookeeper protocol itself. So it has had me curious about ways the client
> implementation itself could make using Zookeeper less error-prone, as
> Curator and Kazoo respectively try to make working with Zookeeper less
> error-prone in their languages. This is part of why I was baffled when an
> AUTH_FAILED resulted in connection termination, but not session expiration.
>
> In the case of kazoo, its now substantially easier to debug what is
> actually happening since its always within the bounds of Python.
> Reinstalling debug builds of the C lib and Python C binding got hairy, and
> still obfuscated a lot. This made debugging the prior password mangling
> issue in the Python lib a major pain.
>
> I've found that the C lib seems like a bit of a second-class citizen for
> Zookeeper... the read-only feature has had patches submitted since 2010,
> which still aren't applied. The Python C binding also has multiple
> patches... not applied. We applied several of these to fix memory leaks and
> the password mangling to our zc-zookeeper-static Python lib, but after
> looking through all the remaining C lib bugs/patches and Python C
> bugs/patches, its way more than we want to deal with. C isn't my forte, and
> I'd much much rather debug Java code or read the Java client if I'm curious
> about how something is done, rather than deal with 2 layers of C code in
> addition to TCP and the Java server.
>
> Meanwhile, I can implement the new Zookeeper 3.4 methods after looking at
> the jute code in just minutes, and debugging a problem is trivial when its
> all Python code. Some people have mentioned that without the C os thread
> used by the C binding, its possible in heavy Python threading thrashes that
> a ping might not be sent... which is true, but the session timeout can be
> increased and thats a very small price to pay given the other things noted
> here with the C lib/binding.
>
> And of course, for other runtimes (Pypy) that can't run C extensions, the
> pure Python kazoo will now work.
>
FWIW, this is my only reservation about a pure Python client - there isn't
a spec, and three separate implementations that might have subtly different
behaviours can be a nightmare to maintain. Ben - if you're able to turn any
of your efforts towards documenting your observations about how the
protocol actually works, that would be awesome.
And as regards the unapplied Python patches - that's my bad, I should be
committing them much more often. Can you give me a list of those you've
found useful, and in return for your excellent work I'll get them committed
as soon as I can?
>
> > I think this is expected. ZooKeeper should not expire a session
> > because of authentication failure. That would make it easier for a
> > malicious client to expire random sessions. I don't know if there is a
> > technical reason for dropping the connection though. Maybe it was an
> > arbitrary decision?
>
> Yea, I wasn't terribly sure. The doc state diagram seems to indicate that
> an AUTH_FAILED should be treated the same as a SESSION_EXPIRATION or
> CLOSING event. However, your session is dead in the latter two cases, while
> the session is *not* dead in AUTH_FAILED, yet you end up in the same state.
> I would not be surprised if a substantial amount of code assumed that the
> session was dead when an AUTH_FAILED occurred, yet the session is not dead
> at all.
>
> Cheers,
> Ben
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
> On Aug 30, 2012, at 4:50 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>
> wrote:
>
> > I don't think there is an official spec beyond the zookeeper.jute
> > file. It would be very helpful if you can share what you have found
> > implementing your python client.
>
> The most interesting thing I found was that documented C/Java behavior
> isn't actually enforced beyond the client boundaries. For some reason I had
> expected the state transitions listed to be some limitation of the
> Zookeeper protocol itself. So it has had me curious about ways the client
> implementation itself could make using Zookeeper less error-prone, as
> Curator and Kazoo respectively try to make working with Zookeeper less
> error-prone in their languages. This is part of why I was baffled when an
> AUTH_FAILED resulted in connection termination, but not session expiration.
>
> In the case of kazoo, its now substantially easier to debug what is
> actually happening since its always within the bounds of Python.
> Reinstalling debug builds of the C lib and Python C binding got hairy, and
> still obfuscated a lot. This made debugging the prior password mangling
> issue in the Python lib a major pain.
>
> I've found that the C lib seems like a bit of a second-class citizen for
> Zookeeper... the read-only feature has had patches submitted since 2010,
> which still aren't applied. The Python C binding also has multiple
> patches... not applied. We applied several of these to fix memory leaks and
> the password mangling to our zc-zookeeper-static Python lib, but after
> looking through all the remaining C lib bugs/patches and Python C
> bugs/patches, its way more than we want to deal with. C isn't my forte, and
> I'd much much rather debug Java code or read the Java client if I'm curious
> about how something is done, rather than deal with 2 layers of C code in
> addition to TCP and the Java server.
>
> Meanwhile, I can implement the new Zookeeper 3.4 methods after looking at
> the jute code in just minutes, and debugging a problem is trivial when its
> all Python code. Some people have mentioned that without the C os thread
> used by the C binding, its possible in heavy Python threading thrashes that
> a ping might not be sent... which is true, but the session timeout can be
> increased and thats a very small price to pay given the other things noted
> here with the C lib/binding.
>
> And of course, for other runtimes (Pypy) that can't run C extensions, the
> pure Python kazoo will now work.
>
FWIW, this is my only reservation about a pure Python client - there isn't
a spec, and three separate implementations that might have subtly different
behaviours can be a nightmare to maintain. Ben - if you're able to turn any
of your efforts towards documenting your observations about how the
protocol actually works, that would be awesome.
And as regards the unapplied Python patches - that's my bad, I should be
committing them much more often. Can you give me a list of those you've
found useful, and in return for your excellent work I'll get them committed
as soon as I can?
>
> > I think this is expected. ZooKeeper should not expire a session
> > because of authentication failure. That would make it easier for a
> > malicious client to expire random sessions. I don't know if there is a
> > technical reason for dropping the connection though. Maybe it was an
> > arbitrary decision?
>
> Yea, I wasn't terribly sure. The doc state diagram seems to indicate that
> an AUTH_FAILED should be treated the same as a SESSION_EXPIRATION or
> CLOSING event. However, your session is dead in the latter two cases, while
> the session is *not* dead in AUTH_FAILED, yet you end up in the same state.
> I would not be surprised if a substantial amount of code assumed that the
> session was dead when an AUTH_FAILED occurred, yet the session is not dead
> at all.
>
> Cheers,
> Ben
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
+
Henry Robinson 2012-08-31, 00:10
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-31, 04:00
On Aug 30, 2012, at 5:10 PM, Henry Robinson <[EMAIL PROTECTED]> wrote:
> FWIW, this is my only reservation about a pure Python client - there isn't
> a spec, and three separate implementations that might have subtly different
> behaviours can be a nightmare to maintain. Ben - if you're able to turn any
> of your efforts towards documenting your observations about how the
> protocol actually works, that would be awesome.
Given that most ppl don't use the client directly, they program against 'helpers' such as Curator, I'm not sure thats such a big deal. The c style of completion callbacks already wasn't terribly useful with the fact that Python has its basic threading way, and many ppl use gevent for an async style (or twisted, also quite different).
The C binding falls apart badly with gevent, since gevent has issues talking across the C thread to the gevent hub. This necessitates a lot of very hacky Python code to try and bridge it, which isn't needed when its pure Python. And trying to finangle the zookeeper logging stream into proper Python logging was another set of hacks.
Right now, the code for the protocol handling looks very close to the Java client (Alan Cabrera did some great work on pookeeper which I refactored into kazoo). I don't see any reason to deviate too heavily, though of course there are some things that are nicer to implement for a more Pythonic feel. Keeping the 'feel' from a user perspective similar to the existing Zookeeper Programmers Guide is best just to avoid having to replicate docs.
On the happy note, Kazoo actually has a good amount of docs:
- http://kazoo.readthedocs.org/
Unfortunately the C lib only has doc strings, which make the Java API docs look like a documentation heaven in comparison. And you can't even get to the C API docs online... I finally got tired of digging them out of the source and using doxygen so I copied them up to my own server here:
http://groovie.org/zkdocs/
Some notes on the implementation...
Pure Python Zookeeper implementation:
- Approx. 280 lines of code for the socket response/request handling
- Approx. 250 lines of code for the request/response serialization/deserialization
- Anyone that knows Python reasonably well can trouble-shoot and contribute
- Can be used in gevent, Pypy, Jython (Jython doesn't even have a GIL!)
- Worst case... an exception bubbles up that you might have to catch
zkpython (Python binding to C lib):
- Approx. 1500 lines of C (Not including the C lib itself, which is another ~ 8000 lines of code)
- Anyone that knows and wants to read the C library *and* Python *and* the Python C binding tricks can trouble-shoot and contribute. And maybe the patches will actually be accepted and incorporated at some point...
- Only usable with CPython
- Worst case... Python segfaults
I really don't know many (hardly any) Python developers that know C well enough to debug it or dive into it. If their only Zookeeper experience is marred by bugs in the 'black box' of C, they'll move on to something else. Which saddens me cause I think Zookeeper is pretty awesome.
It took a week for us to figure out why our test suite failed on rare occasions. This wasn't helped by the fact that Zookeeper doesn't tell you if you supply a bad session id/password you don't get what you do in every app known to mankind (bad password or username).... it tells you SESSION_EXPIRED. Which is insanely confusing when you see your other client using that id/password happily connected still. We had to debug the Java server, use gdb and such to debug two C libs, etc. I really really don't want to ever repeat that experience, it was that bad. :)
On a side-note, why on Earth does Zookeeper not give you an AUTH_FAILED when you fail the auth for the session ID/password on connect?
I'd be happy to document and post more implementation details I've found about the actual protocol. I think it makes sense that powerful dynamic languages implement the protocol directly in a manner thats documented by the Zookeeper project rather than being crippled by using the C lib, and suffering segfaults as a result. Already for kazoo, I've been posting implementation details about how kazoo handles the C lib and bridging it to gevent/threads to help avoid common errors:
http://kazoo.readthedocs.org/en/latest/implementation.html
I can update that for protocol details, though it'd prolly be more useful to have a page on the Zookeeper site itself that discusses and documents the protocol and how it should be implemented for consistency.
Well, we've been maintaining a static zookeeper python library here:
https://github.com/python-zk/zc-zookeeper-static/
We've been adding critical patches to it as we've found them on Jira and in our own tests. Each Jira bug ticket is linked to on there. Several of those are patched in the custom ubuntu compiled distro of the python-zookeeper bindings as well.
But obviously at some point it becomes futile. We'd like to use the read-only feature, but there's no hope of that getting into the Python binding since its still not in the C lib: https://issues.apache.org/jira/browse/ZOOKEEPER-827
There's been patches for that since 2010... and still its not resolved. That's pretty discouraging, and given the lack of online generated C docs there's definitely a "we don't care much about the C lib" message being broadcast. It's very obvious the Java client is what gets the support. Searching Jira for 'zkpython' and seeing the various unresolved memory leaks and segfault issues is also sad. Kazoo is already getting use at several companies, and we all want this thing to be solid, to not seg-fault our Python, and to be able to easily trouble-shoot it without going through C gymnastics. :)
Cheers,
Ben
> FWIW, this is my only reservation about a pure Python client - there isn't
> a spec, and three separate implementations that might have subtly different
> behaviours can be a nightmare to maintain. Ben - if you're able to turn any
> of your efforts towards documenting your observations about how the
> protocol actually works, that would be awesome.
Given that most ppl don't use the client directly, they program against 'helpers' such as Curator, I'm not sure thats such a big deal. The c style of completion callbacks already wasn't terribly useful with the fact that Python has its basic threading way, and many ppl use gevent for an async style (or twisted, also quite different).
The C binding falls apart badly with gevent, since gevent has issues talking across the C thread to the gevent hub. This necessitates a lot of very hacky Python code to try and bridge it, which isn't needed when its pure Python. And trying to finangle the zookeeper logging stream into proper Python logging was another set of hacks.
Right now, the code for the protocol handling looks very close to the Java client (Alan Cabrera did some great work on pookeeper which I refactored into kazoo). I don't see any reason to deviate too heavily, though of course there are some things that are nicer to implement for a more Pythonic feel. Keeping the 'feel' from a user perspective similar to the existing Zookeeper Programmers Guide is best just to avoid having to replicate docs.
On the happy note, Kazoo actually has a good amount of docs:
- http://kazoo.readthedocs.org/
Unfortunately the C lib only has doc strings, which make the Java API docs look like a documentation heaven in comparison. And you can't even get to the C API docs online... I finally got tired of digging them out of the source and using doxygen so I copied them up to my own server here:
http://groovie.org/zkdocs/
Some notes on the implementation...
Pure Python Zookeeper implementation:
- Approx. 280 lines of code for the socket response/request handling
- Approx. 250 lines of code for the request/response serialization/deserialization
- Anyone that knows Python reasonably well can trouble-shoot and contribute
- Can be used in gevent, Pypy, Jython (Jython doesn't even have a GIL!)
- Worst case... an exception bubbles up that you might have to catch
zkpython (Python binding to C lib):
- Approx. 1500 lines of C (Not including the C lib itself, which is another ~ 8000 lines of code)
- Anyone that knows and wants to read the C library *and* Python *and* the Python C binding tricks can trouble-shoot and contribute. And maybe the patches will actually be accepted and incorporated at some point...
- Only usable with CPython
- Worst case... Python segfaults
I really don't know many (hardly any) Python developers that know C well enough to debug it or dive into it. If their only Zookeeper experience is marred by bugs in the 'black box' of C, they'll move on to something else. Which saddens me cause I think Zookeeper is pretty awesome.
It took a week for us to figure out why our test suite failed on rare occasions. This wasn't helped by the fact that Zookeeper doesn't tell you if you supply a bad session id/password you don't get what you do in every app known to mankind (bad password or username).... it tells you SESSION_EXPIRED. Which is insanely confusing when you see your other client using that id/password happily connected still. We had to debug the Java server, use gdb and such to debug two C libs, etc. I really really don't want to ever repeat that experience, it was that bad. :)
On a side-note, why on Earth does Zookeeper not give you an AUTH_FAILED when you fail the auth for the session ID/password on connect?
I'd be happy to document and post more implementation details I've found about the actual protocol. I think it makes sense that powerful dynamic languages implement the protocol directly in a manner thats documented by the Zookeeper project rather than being crippled by using the C lib, and suffering segfaults as a result. Already for kazoo, I've been posting implementation details about how kazoo handles the C lib and bridging it to gevent/threads to help avoid common errors:
http://kazoo.readthedocs.org/en/latest/implementation.html
I can update that for protocol details, though it'd prolly be more useful to have a page on the Zookeeper site itself that discusses and documents the protocol and how it should be implemented for consistency.
Well, we've been maintaining a static zookeeper python library here:
https://github.com/python-zk/zc-zookeeper-static/
We've been adding critical patches to it as we've found them on Jira and in our own tests. Each Jira bug ticket is linked to on there. Several of those are patched in the custom ubuntu compiled distro of the python-zookeeper bindings as well.
But obviously at some point it becomes futile. We'd like to use the read-only feature, but there's no hope of that getting into the Python binding since its still not in the C lib: https://issues.apache.org/jira/browse/ZOOKEEPER-827
There's been patches for that since 2010... and still its not resolved. That's pretty discouraging, and given the lack of online generated C docs there's definitely a "we don't care much about the C lib" message being broadcast. It's very obvious the Java client is what gets the support. Searching Jira for 'zkpython' and seeing the various unresolved memory leaks and segfault issues is also sad. Kazoo is already getting use at several companies, and we all want this thing to be solid, to not seg-fault our Python, and to be able to easily trouble-shoot it without going through C gymnastics. :)
Cheers,
Ben
+
Ben Bangert 2012-08-31, 04:00
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-31, 05:04
On Aug 30, 2012, at 9:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> zkpython (Python binding to C lib):
> - Approx. 1500 lines of C (Not including the C lib itself, which is another ~ 8000 lines of code)
> - Anyone that knows and wants to read the C library *and* Python *and* the Python C binding tricks can trouble-shoot and contribute. And maybe the patches will actually be accepted and incorporated at some point...
> - Only usable with CPython
> - Worst case... Python segfaults
>
> I really don't know many (hardly any) Python developers that know C well enough to debug it or dive into it. If their only Zookeeper experience is marred by bugs in the 'black box' of C, they'll move on to something else. Which saddens me cause I think Zookeeper is pretty awesome.
Wanted to update this cause I think this might have come off as harsh. I greatly appreciate the Python C binding work, without it I wouldn't have started using Zookeeper. I definitely think we can do better to make it more enjoyable to use Zookeeper from Python though.
- Ben
> zkpython (Python binding to C lib):
> - Approx. 1500 lines of C (Not including the C lib itself, which is another ~ 8000 lines of code)
> - Anyone that knows and wants to read the C library *and* Python *and* the Python C binding tricks can trouble-shoot and contribute. And maybe the patches will actually be accepted and incorporated at some point...
> - Only usable with CPython
> - Worst case... Python segfaults
>
> I really don't know many (hardly any) Python developers that know C well enough to debug it or dive into it. If their only Zookeeper experience is marred by bugs in the 'black box' of C, they'll move on to something else. Which saddens me cause I think Zookeeper is pretty awesome.
Wanted to update this cause I think this might have come off as harsh. I greatly appreciate the Python C binding work, without it I wouldn't have started using Zookeeper. I definitely think we can do better to make it more enjoyable to use Zookeeper from Python though.
- Ben
+
Ben Bangert 2012-08-31, 05:04
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientHenry Robinson 2012-08-31, 06:57
On 30 August 2012 22:04, Ben Bangert <[EMAIL PROTECTED]> wrote:
> On Aug 30, 2012, at 9:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>
> > zkpython (Python binding to C lib):
> > - Approx. 1500 lines of C (Not including the C lib itself, which is
> another ~ 8000 lines of code)
> > - Anyone that knows and wants to read the C library *and* Python *and*
> the Python C binding tricks can trouble-shoot and contribute. And maybe the
> patches will actually be accepted and incorporated at some point...
> > - Only usable with CPython
> > - Worst case... Python segfaults
> >
> > I really don't know many (hardly any) Python developers that know C well
> enough to debug it or dive into it. If their only Zookeeper experience is
> marred by bugs in the 'black box' of C, they'll move on to something else.
> Which saddens me cause I think Zookeeper is pretty awesome.
>
> Wanted to update this cause I think this might have come off as harsh. I
> greatly appreciate the Python C binding work, without it I wouldn't have
> started using Zookeeper. I definitely think we can do better to make it
> more enjoyable to use Zookeeper from Python though.
>
>
Ben -
I took a look, and kazoo seems really promising. Thanks for building it!
Speaking as the guy who wrote 95% of the current Python bindings, I'll be
delighted if they can be superseded by a better implementation.
I just want to reiterate that everyone that works on ZooKeeper is a
volunteer, and I believe that none of the active committers are paid to
work on ZooKeeper full-time. That's not to completely abrogate
responsibility - the zkpython JIRAs need a good clean-up and have done for
a long time - but to make the broader point that the very best way to see
the change that you want made is to engage with and become an active member
of the community. Apache is all about community.
That means filing JIRAs, submitting your changes as patches (generating C
documentation would be a great patch) and badgering community members to
review (and committers, eventually, to commit). It's not that 'we' don't
care about the C client, it's that no-one is championing it enough at the
moment. Anyone can become that champion, and they are strongly encouraged
to do so.
I'd love to see kazoo get contributed back to Apache. Is that something
you're interested in doing? I'm not sure we can wholesale replace the
extant Python bindings because I know we have users that depend upon them -
but that's a solvable problem, it's no great pain to include both in a
release.
cheers,
Henry
> - Ben
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
> On Aug 30, 2012, at 9:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>
> > zkpython (Python binding to C lib):
> > - Approx. 1500 lines of C (Not including the C lib itself, which is
> another ~ 8000 lines of code)
> > - Anyone that knows and wants to read the C library *and* Python *and*
> the Python C binding tricks can trouble-shoot and contribute. And maybe the
> patches will actually be accepted and incorporated at some point...
> > - Only usable with CPython
> > - Worst case... Python segfaults
> >
> > I really don't know many (hardly any) Python developers that know C well
> enough to debug it or dive into it. If their only Zookeeper experience is
> marred by bugs in the 'black box' of C, they'll move on to something else.
> Which saddens me cause I think Zookeeper is pretty awesome.
>
> Wanted to update this cause I think this might have come off as harsh. I
> greatly appreciate the Python C binding work, without it I wouldn't have
> started using Zookeeper. I definitely think we can do better to make it
> more enjoyable to use Zookeeper from Python though.
>
>
Ben -
I took a look, and kazoo seems really promising. Thanks for building it!
Speaking as the guy who wrote 95% of the current Python bindings, I'll be
delighted if they can be superseded by a better implementation.
I just want to reiterate that everyone that works on ZooKeeper is a
volunteer, and I believe that none of the active committers are paid to
work on ZooKeeper full-time. That's not to completely abrogate
responsibility - the zkpython JIRAs need a good clean-up and have done for
a long time - but to make the broader point that the very best way to see
the change that you want made is to engage with and become an active member
of the community. Apache is all about community.
That means filing JIRAs, submitting your changes as patches (generating C
documentation would be a great patch) and badgering community members to
review (and committers, eventually, to commit). It's not that 'we' don't
care about the C client, it's that no-one is championing it enough at the
moment. Anyone can become that champion, and they are strongly encouraged
to do so.
I'd love to see kazoo get contributed back to Apache. Is that something
you're interested in doing? I'm not sure we can wholesale replace the
extant Python bindings because I know we have users that depend upon them -
but that's a solvable problem, it's no great pain to include both in a
release.
cheers,
Henry
> - Ben
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
+
Henry Robinson 2012-08-31, 06:57
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-31, 16:39
On Aug 30, 2012, at 11:57 PM, Henry Robinson <[EMAIL PROTECTED]> wrote:
> I took a look, and kazoo seems really promising. Thanks for building it!
> Speaking as the guy who wrote 95% of the current Python bindings, I'll be
> delighted if they can be superseded by a better implementation.
It's not all mine, David LaBissoniere created the original version that I merged other Python zookeeper code-bases into for the recipes and other edge case handling.
> I just want to reiterate that everyone that works on ZooKeeper is a
> volunteer, and I believe that none of the active committers are paid to
> work on ZooKeeper full-time. That's not to completely abrogate
> responsibility - the zkpython JIRAs need a good clean-up and have done for
> a long time - but to make the broader point that the very best way to see
> the change that you want made is to engage with and become an active member
> of the community. Apache is all about community.
Yep, I totally understand that having run a few open-source projects myself. My time thus far on this has been thanks to my employer (Mozilla) for letting my coworker and I bring kazoo up to speed and ensure it'll meet our needs. We're pretty big on community at Mozilla, and unifying the Python clients was seen as a good thing to work on.
> That means filing JIRAs, submitting your changes as patches (generating C
> documentation would be a great patch) and badgering community members to
> review (and committers, eventually, to commit). It's not that 'we' don't
> care about the C client, it's that no-one is championing it enough at the
> moment. Anyone can become that champion, and they are strongly encouraged
> to do so.
Right, that's the problem. I'm not a C guy, and I don't want to be. :)
If I'm going to spend a ton of time getting up to speed on a language, I've been eyeing Rust and/or Go and would prefer those than C. Especially since anyone using the C lib will eventually end up with leaky C abstractions in their code. Such as the whole zhandle nonsense... its just not needed in a Python client, its a C artifact. And there's no shortage of bugs, errors, and sometimes even segfaults in higher level languages that bind back to the C lib as a result.
> I'd love to see kazoo get contributed back to Apache. Is that something
> you're interested in doing? I'm not sure we can wholesale replace the
> extant Python bindings because I know we have users that depend upon them -
> but that's a solvable problem, it's no great pain to include both in a
> release.
It probably depends on what that means. It's been incredibly convenient to be on github for the easy pull request merging and tracking other forks. It seems like the code in the zookeeper contrib/ tree has its release schedule tied to Zookeeper itself, which seems like a bad decision to me since it means that client releases are tied to Zookeeper releases rather than being able to quickly push out new client releases when important bug fixes have been applied. Perhaps that's not the case, it's not entirely clear to me.
Cheers,
Ben
> I took a look, and kazoo seems really promising. Thanks for building it!
> Speaking as the guy who wrote 95% of the current Python bindings, I'll be
> delighted if they can be superseded by a better implementation.
It's not all mine, David LaBissoniere created the original version that I merged other Python zookeeper code-bases into for the recipes and other edge case handling.
> I just want to reiterate that everyone that works on ZooKeeper is a
> volunteer, and I believe that none of the active committers are paid to
> work on ZooKeeper full-time. That's not to completely abrogate
> responsibility - the zkpython JIRAs need a good clean-up and have done for
> a long time - but to make the broader point that the very best way to see
> the change that you want made is to engage with and become an active member
> of the community. Apache is all about community.
Yep, I totally understand that having run a few open-source projects myself. My time thus far on this has been thanks to my employer (Mozilla) for letting my coworker and I bring kazoo up to speed and ensure it'll meet our needs. We're pretty big on community at Mozilla, and unifying the Python clients was seen as a good thing to work on.
> That means filing JIRAs, submitting your changes as patches (generating C
> documentation would be a great patch) and badgering community members to
> review (and committers, eventually, to commit). It's not that 'we' don't
> care about the C client, it's that no-one is championing it enough at the
> moment. Anyone can become that champion, and they are strongly encouraged
> to do so.
Right, that's the problem. I'm not a C guy, and I don't want to be. :)
If I'm going to spend a ton of time getting up to speed on a language, I've been eyeing Rust and/or Go and would prefer those than C. Especially since anyone using the C lib will eventually end up with leaky C abstractions in their code. Such as the whole zhandle nonsense... its just not needed in a Python client, its a C artifact. And there's no shortage of bugs, errors, and sometimes even segfaults in higher level languages that bind back to the C lib as a result.
> I'd love to see kazoo get contributed back to Apache. Is that something
> you're interested in doing? I'm not sure we can wholesale replace the
> extant Python bindings because I know we have users that depend upon them -
> but that's a solvable problem, it's no great pain to include both in a
> release.
It probably depends on what that means. It's been incredibly convenient to be on github for the easy pull request merging and tracking other forks. It seems like the code in the zookeeper contrib/ tree has its release schedule tied to Zookeeper itself, which seems like a bad decision to me since it means that client releases are tied to Zookeeper releases rather than being able to quickly push out new client releases when important bug fixes have been applied. Perhaps that's not the case, it's not entirely clear to me.
Cheers,
Ben
+
Ben Bangert 2012-08-31, 16:39
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientMichi Mutsuzaki 2012-08-31, 21:30
> Yep, I totally understand that having run a few open-source projects myself.
As a matter of fact, Ben helped me with an issue I was having with
Routes back in 2008.
http://markmail.org/message/5l5aebwinbtzewk2
--Michi
As a matter of fact, Ben helped me with an issue I was having with
Routes back in 2008.
http://markmail.org/message/5l5aebwinbtzewk2
--Michi
+
Michi Mutsuzaki 2012-08-31, 21:30
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientTed Dunning 2012-08-31, 20:02
Donating code to apache doesn't prevent you from having your own release
cycle on github. Your releases just won't be Apache releases. It also
doesn't prevent you from pushing diffs for kazoo for Apache trunk. That
way you can release whenever you like and Apache can bundle the current
stable version of kazoo with each release. It might be nice to push a new
stable just before each ZK release, but that isn't critical if the stable
release more or less keeps up with your bleeding edge.
On Fri, Aug 31, 2012 at 12:39 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> > I'd love to see kazoo get contributed back to Apache. Is that something
> > you're interested in doing? I'm not sure we can wholesale replace the
> > extant Python bindings because I know we have users that depend upon
> them -
> > but that's a solvable problem, it's no great pain to include both in a
> > release.
>
> It probably depends on what that means. It's been incredibly convenient to
> be on github for the easy pull request merging and tracking other forks. It
> seems like the code in the zookeeper contrib/ tree has its release schedule
> tied to Zookeeper itself, which seems like a bad decision to me since it
> means that client releases are tied to Zookeeper releases rather than being
> able to quickly push out new client releases when important bug fixes have
> been applied. Perhaps that's not the case, it's not entirely clear to me.
>
cycle on github. Your releases just won't be Apache releases. It also
doesn't prevent you from pushing diffs for kazoo for Apache trunk. That
way you can release whenever you like and Apache can bundle the current
stable version of kazoo with each release. It might be nice to push a new
stable just before each ZK release, but that isn't critical if the stable
release more or less keeps up with your bleeding edge.
On Fri, Aug 31, 2012 at 12:39 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> > I'd love to see kazoo get contributed back to Apache. Is that something
> > you're interested in doing? I'm not sure we can wholesale replace the
> > extant Python bindings because I know we have users that depend upon
> them -
> > but that's a solvable problem, it's no great pain to include both in a
> > release.
>
> It probably depends on what that means. It's been incredibly convenient to
> be on github for the easy pull request merging and tracking other forks. It
> seems like the code in the zookeeper contrib/ tree has its release schedule
> tied to Zookeeper itself, which seems like a bad decision to me since it
> means that client releases are tied to Zookeeper releases rather than being
> able to quickly push out new client releases when important bug fixes have
> been applied. Perhaps that's not the case, it's not entirely clear to me.
>
+
Ted Dunning 2012-08-31, 20:02
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientBen Bangert 2012-08-30, 23:35
On Aug 30, 2012, at 4:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
If you checkout the pure-python kazoo branch, this script will segfault Zookeeper immediately on my machine:
import logging
from kazoo.client import KazooClient
from kazoo.protocol.serialization import (
Auth,
write_buffer,
write_string
)
logging.basicConfig(level=logging.DEBUG)
class BadAuth(Auth):
type = 100
def serialize(self):
return (write_string(self.scheme) + write_buffer(self.auth))
k = KazooClient()
k.start()
k._queue.put((BadAuth(0, 'digest', 'user:password'), None))
It apparently really really doesn't like the fact that the auth_type is missing from the payload. A proper message length is provided though (for the admittedly malformed request), whatever Zookeeper is doing to read the buffer fails to account for the string being where it expected the int. Shouldn't this return a marshaling error?
- Ben
> So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
If you checkout the pure-python kazoo branch, this script will segfault Zookeeper immediately on my machine:
import logging
from kazoo.client import KazooClient
from kazoo.protocol.serialization import (
Auth,
write_buffer,
write_string
)
logging.basicConfig(level=logging.DEBUG)
class BadAuth(Auth):
type = 100
def serialize(self):
return (write_string(self.scheme) + write_buffer(self.auth))
k = KazooClient()
k.start()
k._queue.put((BadAuth(0, 'digest', 'user:password'), None))
It apparently really really doesn't like the fact that the auth_type is missing from the payload. A proper message length is provided though (for the admittedly malformed request), whatever Zookeeper is doing to read the buffer fails to account for the string being where it expected the int. Shouldn't this return a marshaling error?
- Ben
+
Ben Bangert 2012-08-30, 23:35
-
Re: Zookeeper protocol weirdness and pure Python kazoo clientMichi Mutsuzaki 2012-08-30, 23:37
It definitely looks like a bug to me. Please open a jira for this.
https://issues.apache.org/jira/browse/ZOOKEEPER
Thanks!
--Michi
On Thu, Aug 30, 2012 at 4:35 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> On Aug 30, 2012, at 4:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>> So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
>
> If you checkout the pure-python kazoo branch, this script will segfault Zookeeper immediately on my machine:
> import logging
>
> from kazoo.client import KazooClient
> from kazoo.protocol.serialization import (
> Auth,
> write_buffer,
> write_string
> )
>
> logging.basicConfig(level=logging.DEBUG)
>
>
> class BadAuth(Auth):
> type = 100
>
> def serialize(self):
> return (write_string(self.scheme) + write_buffer(self.auth))
>
> k = KazooClient()
> k.start()
> k._queue.put((BadAuth(0, 'digest', 'user:password'), None))
>
>
> It apparently really really doesn't like the fact that the auth_type is missing from the payload. A proper message length is provided though (for the admittedly malformed request), whatever Zookeeper is doing to read the buffer fails to account for the string being where it expected the int. Shouldn't this return a marshaling error?
>
> - Ben
https://issues.apache.org/jira/browse/ZOOKEEPER
Thanks!
--Michi
On Thu, Aug 30, 2012 at 4:35 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
> On Aug 30, 2012, at 4:00 PM, Ben Bangert <[EMAIL PROTECTED]> wrote:
>> So far, I've found that if I send malformed Auth packets that are missing the auth_type int, after a few times I can get the Zookeeper server to segfault. I'll attach some more log scripts and a test script to trigger it next.
>
> If you checkout the pure-python kazoo branch, this script will segfault Zookeeper immediately on my machine:
> import logging
>
> from kazoo.client import KazooClient
> from kazoo.protocol.serialization import (
> Auth,
> write_buffer,
> write_string
> )
>
> logging.basicConfig(level=logging.DEBUG)
>
>
> class BadAuth(Auth):
> type = 100
>
> def serialize(self):
> return (write_string(self.scheme) + write_buffer(self.auth))
>
> k = KazooClient()
> k.start()
> k._queue.put((BadAuth(0, 'digest', 'user:password'), None))
>
>
> It apparently really really doesn't like the fact that the auth_type is missing from the payload. A proper message length is provided though (for the admittedly malformed request), whatever Zookeeper is doing to read the buffer fails to account for the string being where it expected the int. Shouldn't this return a marshaling error?
>
> - Ben
+
Michi Mutsuzaki 2012-08-30, 23:37
