Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Out of memory running ZK unit tests against trunk


Copy link to this message
-
RE: Out of memory running ZK unit tests against trunk
In QuorumPeer, when the peer is in LOOKING state we are starting
ReadOnlyZooKeeperServer in a separate thread. And we are shutting down this
server even before startup which has no effect. Also, as this is not a
blocking call QP keeps on spawning new servers.

1) ReadOnlyZooKeeperServer.startup() need not be called in separate a
thread.
2) ReadOnlyZooKeeperServer.startup() is not a blocking call. Need to
introduce a method like Leader.lead(), Follower.followLeader()
3) Shutdown should be called only after the a/m blocking call is returned.
-----Original Message-----
From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 29, 2011 6:24 AM
To: [EMAIL PROTECTED]
Subject: Re: Out of memory running ZK unit tests against trunk

Near the end of this test (QuorumZxidSyncTest) there are tons of
threads running - 115 "ProcessThread" threads, similar numbers of
SessionTracker.

Also I see ~100 ReadOnlyRequestProcessor - why is this running as a
separate thread? (henry/flavio?)

Regardless, I'll enter a 3.4.0 blocker to clean this up - I suspect
that the server shutdown is not shutting down fully for some reason.

Patrick

On Thu, Jul 28, 2011 at 5:28 PM, Mahadev Konar <[EMAIL PROTECTED]>
wrote:
> Nice find Pat. I cant see a reason on why that should happen. Can we
> just do a stack dump and compare?
>
> thanks
> mahadev
>
> On Thu, Jul 28, 2011 at 1:54 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>> I tracked this down to a low ulimit setting on the particular jenkins
>> host where this was failing (max processes).
>>
>> Specifically the following test was failing on trunk, but not on
>> branch 3_3, which concerns me
>>    ./src/java/test/org/apache/zookeeper/test/QuorumZxidSyncTest.java
>>
>> there haven't been any real changes to this test between versions, any
>> insight into why the server is using more threads in trunk vs
>> branch33?
>>
>> Patrick
>>
>> On Fri, Jul 22, 2011 at 10:58 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>>> I've never seen this before, but in my CI environment (sun jdk
>>> 1.6.0_20) I'm seeing some intermittent failures such as the following.
>>>
>>> Has anyone added/modified tests for 3.4.0 that might be using more
>>> threads/memory than previously? Creating ZK clients but not closing
>>> them, etc...
>>>
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>       at java.lang.Thread.start0(Native Method)
>>>       at java.lang.Thread.start(Thread.java:597)
>>>       at
org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.
java:114)
>>>       at
org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:406)
>>>       at
org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:186)
>>>       at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:103)
>>>       at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:67)
>>>       at
org.apache.zookeeper.test.QuorumZxidSyncTest.setUp(QuorumZxidSyncTest.java:3
9)
>>>
>>> Patrick
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB