Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-1024) possible memory leak in 0.8 beta1 producer with G1GC


+
Jason Toffaletti 2013-08-26, 21:10
Copy link to this message
-
[jira] [Commented] (KAFKA-1024) possible memory leak in 0.8 beta1 producer with G1GC

    [ https://issues.apache.org/jira/browse/KAFKA-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750681#comment-13750681 ]

Jay Kreps commented on KAFKA-1024:
----------------------------------

Yes.
                
> possible memory leak in 0.8 beta1 producer with G1GC
> ----------------------------------------------------
>
>                 Key: KAFKA-1024
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1024
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>         Environment: java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>            Reporter: Jason Toffaletti
>
> I have this in my pom.xml
>         <dependency>
>             <groupId>org.apache.kafka</groupId>
>             <artifactId>kafka_2.9.2</artifactId>
>             <version>0.8.0-beta1</version>
>         </dependency>
>         <dependency>
>             <groupId>org.scala-lang</groupId>
>             <artifactId>scala-library</artifactId>
>             <version>2.9.2</version>
>         </dependency>
>         <dependency>
>             <groupId>org.xerial.snappy</groupId>
>             <artifactId>snappy-java</artifactId>
>             <version>1.0.4.1</version>
>         </dependency>
> I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.
> What else can I provide to help debug this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
+
Jason Toffaletti 2013-08-27, 21:17
+
Jason Toffaletti 2013-08-27, 22:57
+
Jason Toffaletti 2013-08-27, 18:57
+
Jay Kreps 2013-08-28, 17:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB