Kafka, mail # user - Re: Consumer Multi-Fetch - 2014-03-06, 02:46
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Consumer Multi-Fetch
On Thu, Mar 06, 2014 at 02:27:34AM +0000, Sandon Jacobs wrote:

The high-level consumer actually uses multi-fetch. You will need to
have some idea of the max message size and set your fetch size
accordingly. Unfortunately if you are consuming a very large number of
topics this can increase the memory requirements of the consumer.  We
intend to address this in the consumer re-write - there is a separate
design review thread on that.
The consumer essentially pre-fetches and queues the chunks in memory
and the offsets are not incremented/check-pointed until the
application thread actually iterates over the messsages.
The high-level consumer does queuing as described above, but you can
reduce the number of queued chunks.

Joel
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB