Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Max message size & fetch size related?


Copy link to this message
-
Max message size & fetch size related?
Hi all,

We are using kafka 0.7.1

 I am seeing this error in the producer ->

kafka.common.MessageSizeTooLargeException
at
kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMessageSize$1.apply(SyncProducer.scala:141)
at
kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMessageSize$1.apply(SyncProducer.scala:139)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at kafka.message.MessageSet.foreach(MessageSet.scala:87)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$verifyMessageSize(SyncProducer.scala:139)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at
kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.scala:116)
at kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
at kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
at kafka.producer.ProducerPool.send(ProducerPool.scala:102)
at kafka.producer.Producer.zkSend(Producer.scala:143)
at kafka.producer.Producer.send(Producer.scala:105)
at kafka.javaapi.producer.Producer.send(Producer.scala:104)

We are using the max message size as 1 <1000000>000000 <1000000> bytes.

I am planning to bump this up to say 5 <5000000>000000 <5000000> bytes.

I am just wondering do I need to change any other properties in
producer/broker/consumer that may get impacted by this bump up?

Also, I came across this thread, which relates to fetch size relative to
max message size -> https://issues.apache.org/jira/browse/KAFKA-247

Could anyone of you advise me on places I may get impacted and change
accordingly?
Thanks,
Puneet Mehta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB