Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Max message size & fetch size related?


Copy link to this message
-
Max message size & fetch size related?
Hi all,

We are using kafka 0.7.1

 I am seeing this error in the producer ->

kafka.common.MessageSizeTooLargeException
at
kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMessageSize$1.apply(SyncProducer.scala:141)
at
kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMessageSize$1.apply(SyncProducer.scala:139)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at kafka.message.MessageSet.foreach(MessageSet.scala:87)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$verifyMessageSize(SyncProducer.scala:139)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at
kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.scala:116)
at kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
at kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
at kafka.producer.ProducerPool.send(ProducerPool.scala:102)
at kafka.producer.Producer.zkSend(Producer.scala:143)
at kafka.producer.Producer.send(Producer.scala:105)
at kafka.javaapi.producer.Producer.send(Producer.scala:104)

We are using the max message size as 1 <1000000>000000 <1000000> bytes.

I am planning to bump this up to say 5 <5000000>000000 <5000000> bytes.

I am just wondering do I need to change any other properties in
producer/broker/consumer that may get impacted by this bump up?

Also, I came across this thread, which relates to fetch size relative to
max message size -> https://issues.apache.org/jira/browse/KAFKA-247

Could anyone of you advise me on places I may get impacted and change
accordingly?
Thanks,
Puneet Mehta