Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # user - Hung up zookeeper client close?


Copy link to this message
-
Hung up zookeeper client close?
Stack 2011-04-04, 17:48
Have you lot ever seen a hang on client close?  A user over in
hbaselandia is seeing such a thing reportedly (See below -- I'm trying
to get more info).  I thought I'd come ask the experts for their take.
 Should we be running close of zk connection inside a timer
interrupting it if it gets stuck?

Good stuff,
St.Ack
If I dump the JVM threads in a console, it looks like the region
server wants to close all zookeeper connections and blocks until this
is done.
HRegionServer.java:672 --> HConnectionManager.deleteConnection(conf, true);

"regionserver60020-EventThread" daemon prio=10 tid=0x000000005d0dc000
nid=0x7e1d waiting on condition [0x0000000042941000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781c9ce00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

"regionserver60020" prio=10 tid=0x00002aaab023e000 nid=0x7e1b in
Object.wait() [0x000000004273f000]
  java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00000007be9ad330> (a
org.apache.zookeeper.ClientCnxn$Packet)
       at java.lang.Object.wait(Object.java:485)
       at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
       - locked <0x00000007be9ad330> (a org.apache.zookeeper.ClientCnxn$Packet)
       at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1295)
       at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:531)
       - locked <0x0000000781fae170> (a org.apache.zookeeper.ZooKeeper)
       at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:399)
       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1050)
       at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:175)
       - locked <0x00000007801b6800> (a
org.apache.hadoop.hbase.client.HConnectionManager$1)
       at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
       at java.lang.Thread.run(Thread.java:662)

"main-EventThread" daemon prio=10 tid=0x00002aaab0540000 nid=0x7e10
waiting on condition [0x0000000042139000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781faf3e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

Unfortunately, some connections are never closed so the server does
not shut down.
+
Patrick Hunt 2011-04-04, 19:16