Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Max message size & fetch size related?


+
Puneet Mehta 2012-09-24, 22:51
Copy link to this message
-
Re: Max message size & fetch size related?
Swapnil Ghike 2012-09-24, 23:06
Hi Puneet,

Yes you will need to bump up the maxMessageSize in server.KafkaConfig and
fetchSize in consumer.ConsumerConfig.

server.KafkaConfig.maxMessageSize can be same as
producer.ProducerConfig.maxMessageSize.
You can set consumer.ConsumerConfig.fetchsize to a value greater than or
equal to producer.ProducerConfig.maxMessageSize.

Thanks,
Swapnil

On 9/24/12 3:51 PM, "Puneet Mehta" <[EMAIL PROTECTED]> wrote:

>Hi all,
>
>We are using kafka 0.7.1
>
> I am seeing this error in the producer ->
>
>kafka.common.MessageSizeTooLargeException
>at
>kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMe
>ssageSize$1.apply(SyncProducer.scala:141)
>at
>kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyMe
>ssageSize$1.apply(SyncProducer.scala:139)
>at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>at kafka.message.MessageSet.foreach(MessageSet.scala:87)
>at
>kafka.producer.SyncProducer.kafka$producer$SyncProducer$$verifyMessageSize
>(SyncProducer.scala:139)
>at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
>at
>kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.sca
>la:116)
>at
>kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
>at
>kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
>at kafka.producer.ProducerPool.send(ProducerPool.scala:102)
>at kafka.producer.Producer.zkSend(Producer.scala:143)
>at kafka.producer.Producer.send(Producer.scala:105)
>at kafka.javaapi.producer.Producer.send(Producer.scala:104)
>
>We are using the max message size as 1 <1000000>000000 <1000000> bytes.
>
>I am planning to bump this up to say 5 <5000000>000000 <5000000> bytes.
>
>I am just wondering do I need to change any other properties in
>producer/broker/consumer that may get impacted by this bump up?
>
>Also, I came across this thread, which relates to fetch size relative to
>max message size -> https://issues.apache.org/jira/browse/KAFKA-247
>
>Could anyone of you advise me on places I may get impacted and change
>accordingly?
>
>
>Thanks,
>Puneet Mehta
+
Puneet Mehta 2012-09-24, 23:22
+
Swapnil Ghike 2012-09-25, 00:18