Hi,

I have an issue with ZK clients waiting forever. The stack for the
waiting threads looks like the following.

> java.lang.Thread.State: WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
>  at java.lang.Object.wait(Object.java:485)
>  at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
>  - locked <0x00002aab19a019b0>
>    (a org.apache.zookeeper.ClientCnxn$Packet)
>  at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1241)
>  at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1271)
>  ...
Look at the stack further I noticed many more threads hung. All with a
similar call stack (but different client calls, though).

> java.lang.Thread.State: WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
>  at java.lang.Object.wait(Object.java:485)
>  at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
>  - locked <0x00002aab19a013a8>
>    (a org.apache.zookeeper.ClientCnxn$Packet)
>  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:804)
>  ...

Looking at the logs, it seems that all this started with a connection
loss during nights.

03:31:32.855 [Worker-76] WARN  ... KeeperErrorCode = ConnectionLoss ...
03:31:32.867 [Worker-65] WARN  ... KeeperErrorCode = ConnectionLoss ...

However, then I found this:

03:32:49.417 [ZooKeeper Gate Connect Thread-SendThread(zk-03:2181)]
ERROR org.apache.zookeeper.ClientCnxn - from ZooKeeper Gate Connect
Thread-SendThread(zk-03:2181)
java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.resize(HashMap.java:462) ~[na:1.6.0_24]
        at java.util.HashMap.addEntry(HashMap.java:755) ~[na:1.6.0_24]
        at java.util.HashMap.put(HashMap.java:385) ~[na:1.6.0_24]
        at java.util.HashSet.add(HashSet.java:200) ~[na:1.6.0_24]
        at
java.util.AbstractCollection.addAll(AbstractCollection.java:305)
~[na:1.6.0_24]
        at
org.apache.zookeeper.ZooKeeper$ZKWatchManager.materialize(ZooKeeper.java:165)
~[na:na]
        at
org.apache.zookeeper.ClientCnxn$EventThread.queueEvent(ClientCnxn.java:474)
~[na:na]
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1172)
~[na:na]
I was wondering if this may have caused any race condition in the ZK client?

-Gunnar

--
Gunnar Wagenknecht
[EMAIL PROTECTED]
http://wagenknecht.org/

Am 21.06.2011 09:14, schrieb Gunnar Wagenknecht:
> I was wondering if this may have caused any race condition in the ZK client?

Forgot to mention, this happens with ZK 3.3.3.

-Gunnar

--
Gunnar Wagenknecht
[EMAIL PROTECTED]
http://wagenknecht.org/

Am 21.06.2011 09:14, schrieb Gunnar Wagenknecht:
> I was wondering if this may have caused any race condition in the ZK client?

Oddly enough, the OOM might have killed the SendThread, i.e. in the JVM
thread dump there is no SendThread any more. Just the EventThread.

"ZooKeeper Gate Connect Thread-EventThread" daemon prio=10
tid=0x000000005a37d800 nid=0x4e03 waiting on condition [0x000000004357f000]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x00002aaab158f1d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
 at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

-Gunnar
--
Gunnar Wagenknecht
[EMAIL PROTECTED]
http://wagenknecht.org/

Am 21.06.2011 09:32, schrieb Gunnar Wagenknecht:
> Am 21.06.2011 09:14, schrieb Gunnar Wagenknecht:
>> I was wondering if this may have caused any race condition in the ZK client?
>
> Oddly enough, the OOM might have killed the SendThread, i.e. in the JVM
> thread dump there is no SendThread any more. Just the EventThread.

A look at ClientCnxn.java confirms this. I don't know if this is
expected/intended behavior. Thus, I opened
https://issues.apache.org/jira/browse/ZOOKEEPER-1100 to document and
discuss it. IMHO it should not just die silently. Someone should recognize.

-Gunnar

--
Gunnar Wagenknecht
[EMAIL PROTECTED]
http://wagenknecht.org/