Have you lot ever seen a hang on client close?  A user over in
hbaselandia is seeing such a thing reportedly (See below -- I'm trying
to get more info).  I thought I'd come ask the experts for their take.
 Should we be running close of zk connection inside a timer
interrupting it if it gets stuck?

Good stuff,
St.Ack
If I dump the JVM threads in a console, it looks like the region
server wants to close all zookeeper connections and blocks until this
is done.
HRegionServer.java:672 --> HConnectionManager.deleteConnection(conf, true);

"regionserver60020-EventThread" daemon prio=10 tid=0x000000005d0dc000
nid=0x7e1d waiting on condition [0x0000000042941000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781c9ce00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

"regionserver60020" prio=10 tid=0x00002aaab023e000 nid=0x7e1b in
Object.wait() [0x000000004273f000]
  java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00000007be9ad330> (a
org.apache.zookeeper.ClientCnxn$Packet)
       at java.lang.Object.wait(Object.java:485)
       at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
       - locked <0x00000007be9ad330> (a org.apache.zookeeper.ClientCnxn$Packet)
       at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1295)
       at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:531)
       - locked <0x0000000781fae170> (a org.apache.zookeeper.ZooKeeper)
       at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:399)
       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1050)
       at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:175)
       - locked <0x00000007801b6800> (a
org.apache.hadoop.hbase.client.HConnectionManager$1)
       at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
       at java.lang.Thread.run(Thread.java:662)

"main-EventThread" daemon prio=10 tid=0x00002aaab0540000 nid=0x7e10
waiting on condition [0x0000000042139000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781faf3e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

Unfortunately, some connections are never closed so the server does
not shut down.

What version of ZK is being used? See:
https://issues.apache.org/jira/browse/ZOOKEEPER-846

Patrick

On Mon, Apr 4, 2011 at 10:48 AM, Stack <[EMAIL PROTECTED]> wrote:
> Have you lot ever seen a hang on client close?  A user over in
> hbaselandia is seeing such a thing reportedly (See below -- I'm trying
> to get more info).  I thought I'd come ask the experts for their take.
>  Should we be running close of zk connection inside a timer
> interrupting it if it gets stuck?
>
> Good stuff,
> St.Ack
>
>
> If I dump the JVM threads in a console, it looks like the region
> server wants to close all zookeeper connections and blocks until this
> is done.
> HRegionServer.java:672 --> HConnectionManager.deleteConnection(conf, true);
>
> "regionserver60020-EventThread" daemon prio=10 tid=0x000000005d0dc000
> nid=0x7e1d waiting on condition [0x0000000042941000]
>  java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x0000000781c9ce00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
> "regionserver60020" prio=10 tid=0x00002aaab023e000 nid=0x7e1b in
> Object.wait() [0x000000004273f000]
>  java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007be9ad330> (a
> org.apache.zookeeper.ClientCnxn$Packet)
>       at java.lang.Object.wait(Object.java:485)
>       at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
>       - locked <0x00000007be9ad330> (a org.apache.zookeeper.ClientCnxn$Packet)
>       at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1295)
>       at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:531)
>       - locked <0x0000000781fae170> (a org.apache.zookeeper.ZooKeeper)
>       at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:399)
>       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1050)
>       at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:175)
>       - locked <0x00000007801b6800> (a
> org.apache.hadoop.hbase.client.HConnectionManager$1)
>       at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
>       at java.lang.Thread.run(Thread.java:662)
>
> "main-EventThread" daemon prio=10 tid=0x00002aaab0540000 nid=0x7e10
> waiting on condition [0x0000000042139000]
>  java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x0000000781faf3e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
> Unfortunately, some connections are never closed so the server does
> not shut down.
>