Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Consumer Multi-Fetch


Copy link to this message
-
Re: Consumer Multi-Fetch
Joel Koshy 2014-03-06, 02:46
On Thu, Mar 06, 2014 at 02:27:34AM +0000, Sandon Jacobs wrote:

The high-level consumer actually uses multi-fetch. You will need to
have some idea of the max message size and set your fetch size
accordingly. Unfortunately if you are consuming a very large number of
topics this can increase the memory requirements of the consumer.  We
intend to address this in the consumer re-write - there is a separate
design review thread on that.
The consumer essentially pre-fetches and queues the chunks in memory
and the offsets are not incremented/check-pointed until the
application thread actually iterates over the messsages.
The high-level consumer does queuing as described above, but you can
reduce the number of queued chunks.

Joel