Hello -- we're using 0.72. We're looking at the source, but want to be sure. :-)
We create a single ConsumerConnector, call createMessageStreams, and hand the streams off to individual threads. If one of those threads calls next() on a stream, gets some messages, and then *blocks* in some subsequent operation (and blocks for minutes), can it potentially cause all other threads (calling next() on other streams) to block too? Does something inside the ConsumerConnector block all other stream processing? This would explain some behaviour we're seeing.
Actually, you are right. This can happen on a single topic too, if you have more than one consumer thread. Each consumer thread pulls data from a blocking queue, one or more fetchers are putting data into the queue. Say, you have two consumer threads and two partitions from the same broker. There is a single fetcher that fetches both partitions and it will put one partition's data into a separate queue. So, if one thread stops consuming data, it's queue will be full at some point. This will block the fetcher from putting the data into the other queue.
Jun On Wed, Jun 12, 2013 at 9:10 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
On Jun 12, 2013, at 9:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext