Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> InvalidMessageSizeException


Copy link to this message
-
InvalidMessageSizeException
One of our consumers keeps getting an invalid message size exception. I'm
pretty sure that we don't have a message size this big (1.7G). We have two
other consumer groups consuming messages from the same Kafka instance
happily over the last few days.

Since we keep the logs around for a fixed interval and this consumer group
has fallen pretty far behind, is it possible that somehow log truncation is
causing this? We are on kafka 0.6 BTW.

At this point, I'm inclined to wipe out the "/consumers/<group>/offsets"
node in zookeeper to get this system going again. Would that be the
preferred way of getting out of this bad state?

Let me know if there is any other trouble shooting/diagnostics I can run on
 the system before I reboot!

Manish

[$DATE] ERROR k.c.FetcherRunnable [] - error in FetcherRunnable
kafka.common.InvalidMessageSizeException: invalid message size:1852339316
only received bytes:307196 at 0 possible causes (1) a single message larger
than the fetch size; (2) log corruption
at
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:75)
~[rookery-vacuum.jar:na]
at
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:61)
~[rookery-vacuum.jar:na]
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:58)
~[rookery-vacuum.jar:na]
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:50)
~[rookery-vacuum.jar:na]
at
kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
~[rookery-vacuum.jar:na]
at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:70)
~[rookery-vacuum.jar:na]
at
kafka.consumer.FetcherRunnable$$anonfun$run$4.apply(FetcherRunnable.scala:80)
~[rookery-vacuum.jar:na]
at kafka.con...

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB