Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Out of memory running ZK unit tests against trunk


Copy link to this message
-
RE: Out of memory running ZK unit tests against trunk
In QuorumPeer, when the peer is in LOOKING state we are starting
ReadOnlyZooKeeperServer in a separate thread. And we are shutting down this
server even before startup which has no effect. Also, as this is not a
blocking call QP keeps on spawning new servers.

1) ReadOnlyZooKeeperServer.startup() need not be called in separate a
thread.
2) ReadOnlyZooKeeperServer.startup() is not a blocking call. Need to
introduce a method like Leader.lead(), Follower.followLeader()
3) Shutdown should be called only after the a/m blocking call is returned.
-----Original Message-----
From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 29, 2011 6:24 AM
To: [EMAIL PROTECTED]
Subject: Re: Out of memory running ZK unit tests against trunk

Near the end of this test (QuorumZxidSyncTest) there are tons of
threads running - 115 "ProcessThread" threads, similar numbers of
SessionTracker.

Also I see ~100 ReadOnlyRequestProcessor - why is this running as a
separate thread? (henry/flavio?)

Regardless, I'll enter a 3.4.0 blocker to clean this up - I suspect
that the server shutdown is not shutting down fully for some reason.

Patrick

On Thu, Jul 28, 2011 at 5:28 PM, Mahadev Konar <[EMAIL PROTECTED]>
wrote:
> Nice find Pat. I cant see a reason on why that should happen. Can we
> just do a stack dump and compare?
>
> thanks
> mahadev
>
> On Thu, Jul 28, 2011 at 1:54 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>> I tracked this down to a low ulimit setting on the particular jenkins
>> host where this was failing (max processes).
>>
>> Specifically the following test was failing on trunk, but not on
>> branch 3_3, which concerns me
>>    ./src/java/test/org/apache/zookeeper/test/QuorumZxidSyncTest.java
>>
>> there haven't been any real changes to this test between versions, any
>> insight into why the server is using more threads in trunk vs
>> branch33?
>>
>> Patrick
>>
>> On Fri, Jul 22, 2011 at 10:58 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>>> I've never seen this before, but in my CI environment (sun jdk
>>> 1.6.0_20) I'm seeing some intermittent failures such as the following.
>>>
>>> Has anyone added/modified tests for 3.4.0 that might be using more
>>> threads/memory than previously? Creating ZK clients but not closing
>>> them, etc...
>>>
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>       at java.lang.Thread.start0(Native Method)
>>>       at java.lang.Thread.start(Thread.java:597)
>>>       at
org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.
java:114)
>>>       at
org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:406)
>>>       at
org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:186)
>>>       at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:103)
>>>       at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:67)
>>>       at
org.apache.zookeeper.test.QuorumZxidSyncTest.setUp(QuorumZxidSyncTest.java:3
9)
>>>
>>> Patrick
>>>
>>
>