Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> large amount of disk space freed on restart


+
Jason Rosenberg 2013-05-23, 04:46
Copy link to this message
-
Re: large amount of disk space freed on restart
It isn't uncommon if a process has an open file handle on a file that is
deleted, the space is not freed until the handle is closed. So restarting
the process that has a handle on the file would cause the space to be freed
also.

You can troubleshoot that with lsof.
Normally, I see 2-4 log segments deleted every hour in my brokers.  I see
log lines like this:

2013-05-23 04:40:06,857  INFO [kafka-logcleaner-0] log.LogManager -
Deleting log segment 00000000035434043157.kafka from <redacted topic>

However, it seems like if I restart the broker, a massive amount of disk
space is freed (without a corresponding flood of these log segment deleted
messages).  Is there an explanation for this?  Does kafka keep reference to
file segments around, and reuse them as needed or something?  And then or
restart, the references to those free segment files are dropped?

Thoughts?

This is with 0.7.2.

Jason

 
+
Jason Rosenberg 2013-05-23, 05:17
+
Jonathan Creasy 2013-05-23, 05:25
+
Jason Rosenberg 2013-05-23, 05:49
+
Jun Rao 2013-05-23, 14:15
+
Jason Rosenberg 2013-05-23, 18:06
+
Jun Rao 2013-05-24, 03:56
+
Jason Rosenberg 2013-07-14, 08:37
+
Jay Kreps 2013-07-14, 16:45
+
Jason Rosenberg 2013-07-16, 18:23
+
Jay Kreps 2013-07-16, 20:32
+
Jason Rosenberg 2013-07-26, 07:43
+
Jay Kreps 2013-07-26, 16:46
+
Jason Rosenberg 2013-07-26, 21:00
+
Jay Kreps 2013-07-26, 21:03
+
Mike Heffner 2013-09-09, 15:18
+
Jay Kreps 2013-09-09, 17:48
+
Jason Rosenberg 2013-09-09, 18:41
+
Mike Heffner 2013-09-09, 21:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB