Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> are commitOffsets botched to zookeeper?


+
Rob Withers 2013-05-17, 01:54
Copy link to this message
-
Re: are commitOffsets botched to zookeeper?
Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch write
api. So if you commit after every message at a high rate, it will be slow
and inefficient. Besides it will cause zookeeper performance to degrade.

Thanks,
Neha
On May 16, 2013 6:54 PM, "Rob Withers" <[EMAIL PROTECTED]> wrote:

> We are calling commitOffsets after every message consumption.  It looks to
> be ~60% slower, with 29 partitions.  If a single KafkaStream thread is from
> a connector, and there are 29 partitions, then commitOffsets sends 29
> offset updates, correct?  Are these offset updates batched in one send to
> zookeeper?
>
> thanks,
> rob

 
+
Rob Withers 2013-05-17, 05:32
+
Neha Narkhede 2013-05-17, 13:18
+
Neha Narkhede 2013-05-17, 13:35
+
Neha Narkhede 2013-05-20, 16:40