Zookeeper, mail # dev - RE: Question on test timeouts

Question on test timeouts

Vishal Kathuria 2011-08-18, 20:11

Hi,
I apologize if I this is mentioned somewhere in the wiki and I missed it.

For some tests, I see them pass and then fail because of a timeout. (This is on a clean checkout from the trunk)
junit.run:
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899 sec
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
   [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest FAILED (timeout)
Does anyone know what could be going on? The log doesn't say much about what happened.

Thanks!
Vishal

RE: Question on test timeouts

Fournier, Camille F. 2011-08-18, 20:21

The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.

-----Original Message-----
From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 18, 2011 4:11 PM
To: [EMAIL PROTECTED]
Subject: Question on test timeouts

Hi,
I apologize if I this is mentioned somewhere in the wiki and I missed it.

For some tests, I see them pass and then fail because of a timeout. (This is on a clean checkout from the trunk)
junit.run:
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899 sec
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
   [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest FAILED (timeout)
Does anyone know what could be going on? The log doesn't say much about what happened.

Thanks!
Vishal

RE: Question on test timeouts

Vishal Kathuria 2011-08-19, 18:02

Thanks Camille,
Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.

Vishal

-----Original Message-----
From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 18, 2011 1:21 PM
To: '[EMAIL PROTECTED]'
Subject: RE: Question on test timeouts

The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.

-----Original Message-----
From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 18, 2011 4:11 PM
To: [EMAIL PROTECTED]
Subject: Question on test timeouts

Hi,
I apologize if I this is mentioned somewhere in the wiki and I missed it.

For some tests, I see them pass and then fail because of a timeout. (This is on a clean checkout from the trunk)
junit.run:
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899 sec
   [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
   [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
   [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest FAILED (timeout)
Does anyone know what could be going on? The log doesn't say much about what happened.

Thanks!
Vishal

Re: Question on test timeouts

Mahadev Konar 2011-08-22, 05:10

Vishal,
Is https://issues.apache.org/jira/browse/ZOOKEEPER-1157 tracking all
the test failures you are seeing?

thanks
mahadev

On Fri, Aug 19, 2011 at 11:02 AM, Vishal Kathuria
<[EMAIL PROTECTED]> wrote:
> Thanks Camille,
> Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.
>
> Vishal
>
> -----Original Message-----
> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 1:21 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: Question on test timeouts
>
> The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.
>
> -----Original Message-----
> From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 4:11 PM
> To: [EMAIL PROTECTED]
> Subject: Question on test timeouts
>
> Hi,
> I apologize if I this is mentioned somewhere in the wiki and I missed it.
>
> For some tests, I see them pass and then fail because of a timeout. (This is on a clean checkout from the trunk)
> junit.run:
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899 sec
>
>
>
>
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest FAILED (timeout)
>
>
> Does anyone know what could be going on? The log doesn't say much about what happened.
>
> Thanks!
> Vishal
>

RE: Question on test timeouts

Vishal Kathuria 2011-08-22, 18:33

Hi Mahadev,
Yes I am using that Jira for tracking the issues I noticed with the tests. In my last run, the set of tests failing with timeout changed a bit so I am digging in to see what could be going on.

Thanks!
Vishal

-----Original Message-----
From: Mahadev Konar [mailto:[EMAIL PROTECTED]]
Sent: Sunday, August 21, 2011 10:11 PM
To: [EMAIL PROTECTED]
Subject: Re: Question on test timeouts

Vishal,
Is https://issues.apache.org/jira/browse/ZOOKEEPER-1157 tracking all the test failures you are seeing?

thanks
mahadev

On Fri, Aug 19, 2011 at 11:02 AM, Vishal Kathuria <[EMAIL PROTECTED]> wrote:
> Thanks Camille,
> Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.
>
> Vishal
>
> -----Original Message-----
> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 1:21 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: Question on test timeouts
>
> The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.
>
> -----Original Message-----
> From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 4:11 PM
> To: [EMAIL PROTECTED]
> Subject: Question on test timeouts
>
> Hi,
> I apologize if I this is mentioned somewhere in the wiki and I missed it.
>
> For some tests, I see them pass and then fail because of a timeout.
> (This is on a clean checkout from the trunk)
> junit.run:
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899
> sec
>
>
>
>
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest
> FAILED (timeout)
>
>
> Does anyone know what could be going on? The log doesn't say much about what happened.
>
> Thanks!
> Vishal
>

Re: Question on test timeouts

Benjamin Reed 2011-08-22, 18:41

i've looked into this a bit. it looks like a bunch of our tests are
using 2ms tick times instead of 2000ms tick times, which may make
things fail on slower machines. do you think that is what you are
seeing?

ben

On Mon, Aug 22, 2011 at 11:33 AM, Vishal Kathuria
<[EMAIL PROTECTED]> wrote:
> Hi Mahadev,
> Yes I am using that Jira for tracking the issues I noticed with the tests. In my last run, the set of tests failing with timeout changed a bit so I am digging in to see what could be going on.
>
> Thanks!
> Vishal
>
> -----Original Message-----
> From: Mahadev Konar [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, August 21, 2011 10:11 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Question on test timeouts
>
> Vishal,
> Is https://issues.apache.org/jira/browse/ZOOKEEPER-1157 tracking all the test failures you are seeing?
>
> thanks
> mahadev
>
> On Fri, Aug 19, 2011 at 11:02 AM, Vishal Kathuria <[EMAIL PROTECTED]> wrote:
>> Thanks Camille,
>> Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.
>>
>> Vishal
>>
>> -----Original Message-----
>> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, August 18, 2011 1:21 PM
>> To: '[EMAIL PROTECTED]'
>> Subject: RE: Question on test timeouts
>>
>> The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.
>>
>> -----Original Message-----
>> From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, August 18, 2011 4:11 PM
>> To: [EMAIL PROTECTED]
>> Subject: Question on test timeouts
>>
>> Hi,
>> I apologize if I this is mentioned somewhere in the wiki and I missed it.
>>
>> For some tests, I see them pass and then fail because of a timeout.
>> (This is on a clean checkout from the trunk)
>> junit.run:
>> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
>> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899
>> sec
>>
>>
>>
>>
>> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
>> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>> [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest
>> FAILED (timeout)
>>
>>
>> Does anyone know what could be going on? The log doesn't say much about what happened.
>>
>> Thanks!
>> Vishal
>>
>

Re: Question on test timeouts

Patrick Hunt 2011-08-22, 23:04

Can you both check your ulimits? I was seeing random failures when
"max user processes" was too low (1024, although this seems to be an
issue with server shutdown) Also if the "open files" is too low.

What does "ulimit -a" look like?

Patrick

On Fri, Aug 19, 2011 at 11:02 AM, Vishal Kathuria
<[EMAIL PROTECTED]> wrote:
> Thanks Camille,
> Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.
>
> Vishal
>
> -----Original Message-----
> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 1:21 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: Question on test timeouts
>
> The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.
>
> -----Original Message-----
> From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 18, 2011 4:11 PM
> To: [EMAIL PROTECTED]
> Subject: Question on test timeouts
>
> Hi,
> I apologize if I this is mentioned somewhere in the wiki and I missed it.
>
> For some tests, I see them pass and then fail because of a timeout. (This is on a clean checkout from the trunk)
> junit.run:
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899 sec
>
>
>
>
> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest FAILED (timeout)
>
>
> Does anyone know what could be going on? The log doesn't say much about what happened.
>
> Thanks!
> Vishal
>

Re: Question on test timeouts

Patrick Hunt 2011-08-22, 23:05

On Mon, Aug 22, 2011 at 11:41 AM, Benjamin Reed <[EMAIL PROTECTED]> wrote:
> i've looked into this a bit. it looks like a bunch of our tests are
> using 2ms tick times instead of 2000ms tick times, which may make
> things fail on slower machines. do you think that is what you are
> seeing?

Urg. Ben can you create a subtask off 1157 for this? List the tests
which are effected.

fwiw on my CI setup I'm not currently seeing random failures on tests
- although I have seen this when:

1) when running multiple slots on a Jenkins host and running both
hadoop (or hbase, etc...) unit tests at the same time as running ZK
unit tests
2) when running with insufficient ulimits

Patrick

> On Mon, Aug 22, 2011 at 11:33 AM, Vishal Kathuria
> <[EMAIL PROTECTED]> wrote:
>> Hi Mahadev,
>> Yes I am using that Jira for tracking the issues I noticed with the tests. In my last run, the set of tests failing with timeout changed a bit so I am digging in to see what could be going on.
>>
>> Thanks!
>> Vishal
>>
>> -----Original Message-----
>> From: Mahadev Konar [mailto:[EMAIL PROTECTED]]
>> Sent: Sunday, August 21, 2011 10:11 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Question on test timeouts
>>
>> Vishal,
>> Is https://issues.apache.org/jira/browse/ZOOKEEPER-1157 tracking all the test failures you are seeing?
>>
>> thanks
>> mahadev
>>
>> On Fri, Aug 19, 2011 at 11:02 AM, Vishal Kathuria <[EMAIL PROTECTED]> wrote:
>>> Thanks Camille,
>>> Comforting to know I am not the only one seeing this, so it is not a regression out of my changes.
>>>
>>> Vishal
>>>
>>> -----Original Message-----
>>> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
>>> Sent: Thursday, August 18, 2011 1:21 PM
>>> To: '[EMAIL PROTECTED]'
>>> Subject: RE: Question on test timeouts
>>>
>>> The hammer tests always seem to fail for me too. I've started ignoring them, which is probably not a good thing.
>>>
>>> -----Original Message-----
>>> From: Vishal Kathuria [mailto:[EMAIL PROTECTED]]
>>> Sent: Thursday, August 18, 2011 4:11 PM
>>> To: [EMAIL PROTECTED]
>>> Subject: Question on test timeouts
>>>
>>> Hi,
>>> I apologize if I this is mentioned somewhere in the wiki and I missed it.
>>>
>>> For some tests, I see them pass and then fail because of a timeout.
>>> (This is on a clean checkout from the trunk)
>>> junit.run:
>>> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
>>> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.899
>>> sec
>>>
>>>
>>>
>>>
>>> [junit] Running org.apache.zookeeper.test.ObserverQuorumHammerTest
>>> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>>> [junit] Test org.apache.zookeeper.test.ObserverQuorumHammerTest
>>> FAILED (timeout)
>>>
>>>
>>> Does anyone know what could be going on? The log doesn't say much about what happened.
>>>
>>> Thanks!
>>> Vishal
>>>
>>
>