On Thu, Mar 06, 2014 at 02:27:34AM +0000, Sandon Jacobs wrote:
The high-level consumer actually uses multi-fetch. You will need to
have some idea of the max message size and set your fetch size
accordingly. Unfortunately if you are consuming a very large number of
topics this can increase the memory requirements of the consumer. We
intend to address this in the consumer re-write - there is a separate
design review thread on that.
The consumer essentially pre-fetches and queues the chunks in memory
and the offsets are not incremented/check-pointed until the
application thread actually iterates over the messsages.
The high-level consumer does queuing as described above, but you can
reduce the number of queued chunks.