Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Closed] (KAFKA-767) Message Size check should be done after assigning the offsets


Copy link to this message
-
[jira] [Closed] (KAFKA-767) Message Size check should be done after assigning the offsets

     [ https://issues.apache.org/jira/browse/KAFKA-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede closed KAFKA-767.
-------------------------------

    
> Message Size check should be done after assigning the offsets
> -------------------------------------------------------------
>
>                 Key: KAFKA-767
>                 URL: https://issues.apache.org/jira/browse/KAFKA-767
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>    Affects Versions: 0.8
>            Reporter: Sriram Subramanian
>            Assignee: Sriram Subramanian
>            Priority: Blocker
>              Labels: p1
>             Fix For: 0.8
>
>         Attachments: KAFKA-767-v1.patch, KAFKA-767-v2.patch
>
>
> Replica fetcher thread fails with messageSizeTooLarge exception. One theory is that this check is happening before decompress - assign offsets - compress phase. Hence the final compressed size can be different from that obtained from the produce request. This causes replica fetcher thread to be permanently down and prevents the broker from being in sync.
> 2013/02/20 02:19:25.447 ERROR [ReplicaFetcherThread] [ReplicaFetcherThread-0-274] [kafka] []  [ReplicaFetcherThread-0-274], Error due to
> kafka.common.MessageSizeTooLargeException: Message size is 1000028 bytes which exceeds the maximum configured message size of 1000000.
>         at kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:353)
>         at kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:339)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
>         at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:339)
>         at kafka.log.Log.append(Log.scala:262)
>         at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:52)
>         at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:130)
>         at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
>         at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:125)
>         at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
>         at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
>         at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
>         at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB