Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> message stuck, possible problem setting fetch.message.max.bytes


Copy link to this message
-
Re: message stuck, possible problem setting fetch.message.max.bytes
thanks for the help.  For others who happen upon this thread, the problem
was indeed on the consumer side. Spark (0.9.1) needs a bit of help setting
the Kafka properties for big messages.

    // setup Kafka with manual parameters to allow big messaging
    //see
spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
    val kafkaParams = Map[String, String](
      "zookeeper.connect" -> zkQuorum, "group.id" -> group,
      "zookeeper.connection.timeout.ms" -> "10000",
      "fetch.message.max.bytes" -> "10485760",    // 10MB
      "fetch.size" -> "10485760")    // not needed?
    val lines = kafka.KafkaUtils.createStream[String, String,
StringDecoder, StringDecoder](
                ssc, kafkaParams, topicpMap,
StorageLevel.MEMORY_AND_DISK_SER_2).map(_._2)

 sorry about all the messages on this topic for those of you who aren't
getting digests
On Fri, Jun 27, 2014 at 10:43 AM, Louis Clark <[EMAIL PROTECTED]> wrote: