Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Why does the high level consumer block, or rather where does it?

Copy link to this message
Why does the high level consumer block, or rather where does it?
S Ahmed 2014-01-06, 01:54
I'm trying to trace through the codebase and figure out where exactly the
block occurs in the high level consumer?

public void run() {
     ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
     while (it.hasNext())
         System.out.println("Thread " + m_threadNumber + ": " + new
     System.out.println("Shutting down Thread: " + m_threadNumber);

So from what I understand, the while.it.hasNext() will block if there are
no new messages for this particular topic/partion correct?

Just to understand, can someone clarify where in the kafka source this
block occurs, i.e. the broker that this consumer is connected to will keep
a socket connection open to this consumer and block until a new message
that is owned by this consumer thread arrives and then pushes it to the
consumer to process.

Is it at the iterator level somewhere?