Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> are commitOffsets botched to zookeeper?

Copy link to this message
Re: are commitOffsets botched to zookeeper?
Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch write
api. So if you commit after every message at a high rate, it will be slow
and inefficient. Besides it will cause zookeeper performance to degrade.

On May 16, 2013 6:54 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:

> We are calling commitOffsets after every message consumption.  It looks to
> be ~60% slower, with 29 partitions.  If a single KafkaStream thread is from
> a connector, and there are 29 partitions, then commitOffsets sends 29
> offset updates, correct?  Are these offset updates batched in one send to
> zookeeper?
> thanks,
> rob