Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - [jira] [Commented] (KAFKA-682) java.lang.OutOfMemoryError: Java heap space


+
Neha Narkhede 2013-01-07, 21:32
+
Joel Koshy 2013-01-08, 19:12
Copy link to this message
-
[jira] [Commented] (KAFKA-682) java.lang.OutOfMemoryError: Java heap space
"Joel Koshy 2013-01-04, 19:14

    [ https://issues.apache.org/jira/browse/KAFKA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544119#comment-13544119 ]

Joel Koshy commented on KAFKA-682:
----------------------------------

You might need to increase your heap size. What do you have it set to right now? Would you be able to run the broker with -XX:+HeapDumpOnOutOfMemoryError to get a heap-dump?

In case you are overriding defaults - what's the replication factor for the topic, num-required-acks for the producer requests, and producer request timeout? Are any requests going through or are the produce requests expiring?

                
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
>                 Key: KAFKA-682
>                 URL: https://issues.apache.org/jira/browse/KAFKA-682
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8
>         Environment: $ uname -a
> Linux rngadam-think 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:32:08 UTC 2012 i686 i686 i686 GNU/Linux
> $ java -version
> java version "1.7.0_09"
> OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
> OpenJDK Server VM (build 23.2-b09, mixed mode)
>            Reporter: Ricky Ng-Adam
>
> git pull (commit 32dae955d5e2e2dd45bddb628cb07c874241d856)
> ...build...
> ./sbt update
> ./sbt package
> ...run...
> bin/zookeeper-server-start.sh config/zookeeper.properties
> bin/kafka-server-start.sh config/server.properties
> ...then configured fluentd with kafka plugin...
> gem install fluentd --no-ri --no-rdoc
> gem install fluent-plugin-kafka
> fluentd -c ./fluent/fluent.conf -vv
> ...then flood fluentd with messages inputted from syslog and outputted to kafka.
> results in (after about 10000 messages of 1K each in 3s):
> [2013-01-05 02:00:52,087] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>     at kafka.api.ProducerRequest$$anonfun$1$$anonfun$apply$1.apply(ProducerRequest.scala:45)
>     at kafka.api.ProducerRequest$$anonfun$1$$anonfun$apply$1.apply(ProducerRequest.scala:42)
>     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>     at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:282)
>     at scala.collection.immutable.Range$$anon$1.foreach(Range.scala:274)
>     at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>     at scala.collection.immutable.Range.map(Range.scala:39)
>     at kafka.api.ProducerRequest$$anonfun$1.apply(ProducerRequest.scala:42)
>     at kafka.api.ProducerRequest$$anonfun$1.apply(ProducerRequest.scala:38)
>     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:227)
>     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:227)
>     at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:282)
>     at scala.collection.immutable.Range$$anon$1.foreach(Range.scala:274)
>     at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:227)
>     at scala.collection.immutable.Range.flatMap(Range.scala:39)
>     at kafka.api.ProducerRequest$.readFrom(ProducerRequest.scala:38)
>     at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:32)
>     at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:32)
>     at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:47)
>     at kafka.network.Processor.read(SocketServer.scala:298)
>     at kafka.network.Processor.run(SocketServer.scala:209)
>     at java.lang.Thread.run(Thread.java:722)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
+
Jun Rao 2013-01-04, 23:06
+
Joel Koshy 2013-01-05, 01:36
+
Jay Kreps 2013-01-18, 05:26