-RE: Out of memory running ZK unit tests against trunk
Laxman 2011-07-29, 14:00
In QuorumPeer, when the peer is in LOOKING state we are starting
ReadOnlyZooKeeperServer in a separate thread. And we are shutting down this
server even before startup which has no effect. Also, as this is not a
blocking call QP keeps on spawning new servers.
1) ReadOnlyZooKeeperServer.startup() need not be called in separate a
2) ReadOnlyZooKeeperServer.startup() is not a blocking call. Need to
introduce a method like Leader.lead(), Follower.followLeader()
3) Shutdown should be called only after the a/m blocking call is returned.
From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 29, 2011 6:24 AM
To: [EMAIL PROTECTED]
Subject: Re: Out of memory running ZK unit tests against trunk
Near the end of this test (QuorumZxidSyncTest) there are tons of
threads running - 115 "ProcessThread" threads, similar numbers of
Also I see ~100 ReadOnlyRequestProcessor - why is this running as a
separate thread? (henry/flavio?)
Regardless, I'll enter a 3.4.0 blocker to clean this up - I suspect
that the server shutdown is not shutting down fully for some reason.
On Thu, Jul 28, 2011 at 5:28 PM, Mahadev Konar <[EMAIL PROTECTED]>
> Nice find Pat. I cant see a reason on why that should happen. Can we
> just do a stack dump and compare?
> On Thu, Jul 28, 2011 at 1:54 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>> I tracked this down to a low ulimit setting on the particular jenkins
>> host where this was failing (max processes).
>> Specifically the following test was failing on trunk, but not on
>> branch 3_3, which concerns me
>> there haven't been any real changes to this test between versions, any
>> insight into why the server is using more threads in trunk vs
>> On Fri, Jul 22, 2011 at 10:58 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>>> I've never seen this before, but in my CI environment (sun jdk
>>> 1.6.0_20) I'm seeing some intermittent failures such as the following.
>>> Has anyone added/modified tests for 3.4.0 that might be using more
>>> threads/memory than previously? Creating ZK clients but not closing
>>> them, etc...
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>> at java.lang.Thread.start0(Native Method)
>>> at java.lang.Thread.start(Thread.java:597)
>>> at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:103)
>>> at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:67)