Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Hung up zookeeper client close?


Copy link to this message
-
Hung up zookeeper client close?
Have you lot ever seen a hang on client close?  A user over in
hbaselandia is seeing such a thing reportedly (See below -- I'm trying
to get more info).  I thought I'd come ask the experts for their take.
 Should we be running close of zk connection inside a timer
interrupting it if it gets stuck?

Good stuff,
St.Ack
If I dump the JVM threads in a console, it looks like the region
server wants to close all zookeeper connections and blocks until this
is done.
HRegionServer.java:672 --> HConnectionManager.deleteConnection(conf, true);

"regionserver60020-EventThread" daemon prio=10 tid=0x000000005d0dc000
nid=0x7e1d waiting on condition [0x0000000042941000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781c9ce00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

"regionserver60020" prio=10 tid=0x00002aaab023e000 nid=0x7e1b in
Object.wait() [0x000000004273f000]
  java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00000007be9ad330> (a
org.apache.zookeeper.ClientCnxn$Packet)
       at java.lang.Object.wait(Object.java:485)
       at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
       - locked <0x00000007be9ad330> (a org.apache.zookeeper.ClientCnxn$Packet)
       at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1295)
       at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:531)
       - locked <0x0000000781fae170> (a org.apache.zookeeper.ZooKeeper)
       at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:399)
       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1050)
       at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:175)
       - locked <0x00000007801b6800> (a
org.apache.hadoop.hbase.client.HConnectionManager$1)
       at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
       at java.lang.Thread.run(Thread.java:662)

"main-EventThread" daemon prio=10 tid=0x00002aaab0540000 nid=0x7e10
waiting on condition [0x0000000042139000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x0000000781faf3e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

Unfortunately, some connections are never closed so the server does
not shut down.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB