Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue


Copy link to this message
-
Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue
The high-level consumer will block if a) there are no more messages
available, b) the next message available is larger than the maximum fetch size you
have specified, or c) your client code simply stops pulling messages from the
iterator (the blocking queue will fill up).

Not sure if this applies to what you are seeing, but you should be aware of
these conditions.

Philip

On Thu, Aug 22, 2013 at 12:10:54AM +0800, Frank Yao wrote:
> Hi,
>
> My java consumer client will suddenly stop consuming the stream. While I
> enabled the 'TRACE' log, I found that this stop always happened in the end
> of a queuechunk consumption - after several times' chunk actually. What
> weird is that in log there are not any useful information. It occurred
> suddenly.
>
> I use jstack to figure out what happened and I found that perhaps the
> thread is locked:
>   java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fa399970> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51
>
> After google, I found some body has met this problem before, but no
> solution there.
> http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+[EMAIL PROTECTED]%3E
>
> If any one can offer any help, I'll really appreciate it.
>
> -----------------
>
> Frank Yao
> @VIPSHOP, Shanghai, China

--
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, CA.
www.loggly.com

Come join us!
http://loggly.com/company/careers/