Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - identify the cause of a large quantity of zookeeper threads


Copy link to this message
-
Re: identify the cause of a large quantity of zookeeper threads
Camille Fournier 2012-10-03, 19:24
Are you properly reusing your ZooKeeper clients? You should (probably)
create one client and share it throughout your process to send requests to
the server, not create a new client per usage. Seems like you're not
creating and reusing clients appropriately.

C

On Wed, Oct 3, 2012 at 2:21 PM, Reynold Xin <[EMAIL PROTECTED]> wrote:

> I am trying to debug a problem in which when I run jstack, I see a very
> large number of zookeeper threads, and as the program continues running,
> more threads like the following are created. I suspect somewhere in the
> program it is not releasing some resource properly, but I am not sure where
> it is.
>
> What would be the cause of these threads waiting? Are they created by
> launching new instances of zookeeper, or created by read/write operations?
>
> Thanks.
>
>
> "main-EventThread" daemon prio=5 tid=7fbf2f71d800 nid=0x11c4e4000 waiting
> on condition [11c4e3000]
>    java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <77e5838b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>  at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
>
> "main-SendThread(localhost:21819)" daemon prio=5 tid=7fbf2f651800
> nid=0x11c3e1000 runnable [11c3e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>  at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <77e582700> (a sun.nio.ch.Util$2)
>  - locked <77e5826f0> (a java.util.Collections$UnmodifiableSet)
> - locked <77e5824e8> (a sun.nio.ch.KQueueSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>
> "main-EventThread" daemon prio=5 tid=7fbf2f64f800 nid=0x11c0a7000 waiting
> on condition [11c0a6000]
>    java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <77e39be90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>  at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
>
> "main-SendThread(localhost:21819)" daemon prio=5 tid=7fbf2f0b6000
> nid=0x11b7d2000 runnable [11b7d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>  at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <77e39acd8> (a sun.nio.ch.Util$2)
>  - locked <77e39acc8> (a java.util.Collections$UnmodifiableSet)
> - locked <77e39aac0> (a sun.nio.ch.KQueueSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>
> "main-EventThread" daemon prio=5 tid=7fbf2f0b5000 nid=0x117c9e000 waiting
> on condition [117c9d000]
>    java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <77dfd9540> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)