Hello Jun,

Please see my comments inline.

On 09-Jul-2013, at 8:32 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

No, we kept dedicated consumer listening to the topic in question.
We did this because this queue processes huge amounts of data.

Yes. We can see all the threads in the thread dump.
We have ensured that the threads do not die due to an Exception.

Please look at the stack trace below. We see all the threads waiting like this:

"event_queue@150" prio=10 tid=0x00007eff28e41800 nid=0x31f9 waiting on condition [0x00007efedae6d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000640248618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
        at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
        at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
        at com.spr.messageprocessor.KafkaStreamRunnable.run(KafkaStreamRunnable.java:49)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB