Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Make zookeeper more test friendly


Copy link to this message
-
Re: Make zookeeper more test friendly
FYI

I've created the first patch for this series - it adds support for injecting a session expiration:

https://issues.apache.org/jira/browse/ZOOKEEPER-1730

-JZ

On Jul 6, 2013, at 8:40 AM, Jordan Zimmerman <[EMAIL PROTECTED]> wrote:

> Is there a Jira to track this? I'd like to do some work on this.
>
> -Jordan
>
> On Jul 1, 2013, at 4:59 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>
>> It's come up a bunch of times before, would be great to find someone
>> to drive this!
>> http://markmail.org/message/vvk2ttrdhe6qqp2q
>>
>> Patrick
>>
>> On Fri, Jun 28, 2013 at 1:08 AM, kishore g <[EMAIL PROTECTED]> wrote:
>>> +1
>>>
>>> We had to do similar stuff internally at LinkedIn and most of the bugs we
>>> found were in the way session expiry/disconnect handling. We did a
>>> combination of iptables, SIGSTOP and having another client connect with
>>> same session id/password and close that connection. This is non trivial and
>>> requires some effort to wire up different pieces.
>>>
>>> However I would like to add that the even though our test cases worked we
>>> had weird issues during GC's and some times during long GC. GC on both
>>> server and client are problematic. For example clients would get a session
>>> expiry and then a syncconnected event but before syncconnected is processed
>>> there would be another session expiry. These scenarios are much harder to
>>> test for and reproduce.
>>>
>>> Thanks for taking this up.
>>>
>>> Thanks,
>>> Kishore G
>>>
>>>
>>> On Thu, Jun 27, 2013 at 10:38 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>>>
>>>> +1  this is a very big deal
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jun 27, 2013 at 6:39 PM, Thawan Kooburat <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Many recent issues that I saw internally is due to incorrect handling or
>>>>> no sufficient testing on ZooKeeper failure scenario in the custom wrapper
>>>>> API or in the applications.
>>>>>
>>>>> I am thinking that we might be able to expose a few more API calls that
>>>>> allow user write unit tests that cover various failure scenarios (similar
>>>>> to the TestableZookeer in zookeeper test) . This should also minimize the
>>>>> effort on setting the test framework.  Ideally, if we have a mock client
>>>>> that don't need a running the server that would be ideal, but I think it
>>>> is
>>>>> too much effort to write and maintain for all the languages. Our internal
>>>>> test facility is that we have a dedicated ensemble used by all unit
>>>> tests.
>>>>> This ensure application logic correctness but it is hard to test various
>>>>> failure scenarios.
>>>>>
>>>>> So my current thought is to expose the following functionalities.
>>>>>
>>>>> 1.  zookeeper_close() that don't actually send close request to the
>>>>> server:     This can be used to simulate a client crash without actually
>>>>> crashing the test program.
>>>>> 2.  Allow client to force triggering CONNECTION_LOSS or SESSSION_EXPIRE
>>>>> event:    This will allow the user to test their watchers and callback
>>>> (and
>>>>> possible race condition)
>>>>>
>>>>> Let me know if you have additional suggestions.
>>>>>
>>>>>
>>>>> --
>>>>> Thawan Kooburat
>>>>>
>>>>
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB