Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Make zookeeper more test friendly

Copy link to this message
Re: Make zookeeper more test friendly

I've created the first patch for this series - it adds support for injecting a session expiration:



On Jul 6, 2013, at 8:40 AM, Jordan Zimmerman <[EMAIL PROTECTED]> wrote:

> Is there a Jira to track this? I'd like to do some work on this.
> -Jordan
> On Jul 1, 2013, at 4:59 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>> It's come up a bunch of times before, would be great to find someone
>> to drive this!
>> http://markmail.org/message/vvk2ttrdhe6qqp2q
>> Patrick
>> On Fri, Jun 28, 2013 at 1:08 AM, kishore g <[EMAIL PROTECTED]> wrote:
>>> +1
>>> We had to do similar stuff internally at LinkedIn and most of the bugs we
>>> found were in the way session expiry/disconnect handling. We did a
>>> combination of iptables, SIGSTOP and having another client connect with
>>> same session id/password and close that connection. This is non trivial and
>>> requires some effort to wire up different pieces.
>>> However I would like to add that the even though our test cases worked we
>>> had weird issues during GC's and some times during long GC. GC on both
>>> server and client are problematic. For example clients would get a session
>>> expiry and then a syncconnected event but before syncconnected is processed
>>> there would be another session expiry. These scenarios are much harder to
>>> test for and reproduce.
>>> Thanks for taking this up.
>>> Thanks,
>>> Kishore G
>>> On Thu, Jun 27, 2013 at 10:38 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>>>> +1  this is a very big deal
>>>> On Thu, Jun 27, 2013 at 6:39 PM, Thawan Kooburat <[EMAIL PROTECTED]> wrote:
>>>>> Many recent issues that I saw internally is due to incorrect handling or
>>>>> no sufficient testing on ZooKeeper failure scenario in the custom wrapper
>>>>> API or in the applications.
>>>>> I am thinking that we might be able to expose a few more API calls that
>>>>> allow user write unit tests that cover various failure scenarios (similar
>>>>> to the TestableZookeer in zookeeper test) . This should also minimize the
>>>>> effort on setting the test framework.  Ideally, if we have a mock client
>>>>> that don't need a running the server that would be ideal, but I think it
>>>> is
>>>>> too much effort to write and maintain for all the languages. Our internal
>>>>> test facility is that we have a dedicated ensemble used by all unit
>>>> tests.
>>>>> This ensure application logic correctness but it is hard to test various
>>>>> failure scenarios.
>>>>> So my current thought is to expose the following functionalities.
>>>>> 1.  zookeeper_close() that don't actually send close request to the
>>>>> server:     This can be used to simulate a client crash without actually
>>>>> crashing the test program.
>>>>> 2.  Allow client to force triggering CONNECTION_LOSS or SESSSION_EXPIRE
>>>>> event:    This will allow the user to test their watchers and callback
>>>> (and
>>>>> possible race condition)
>>>>> Let me know if you have additional suggestions.
>>>>> --
>>>>> Thawan Kooburat