Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer Multi-Fetch

Copy link to this message
Re: Consumer Multi-Fetch
On Thu, Mar 06, 2014 at 02:27:34AM +0000, Sandon Jacobs wrote:

The high-level consumer actually uses multi-fetch. You will need to
have some idea of the max message size and set your fetch size
accordingly. Unfortunately if you are consuming a very large number of
topics this can increase the memory requirements of the consumer.  We
intend to address this in the consumer re-write - there is a separate
design review thread on that.
The consumer essentially pre-fetches and queues the chunks in memory
and the offsets are not incremented/check-pointed until the
application thread actually iterates over the messsages.
The high-level consumer does queuing as described above, but you can
reduce the number of queued chunks.