Kafka, mail # user - Re: large amount of disk space freed on restart - 2013-07-26, 16:46
 Search Hadoop and all its subprojects:
+
Jason Rosenberg 2013-07-26, 07:43
Copy link to this message
-
Re: large amount of disk space freed on restart
Interesting.

Yes, Kafka keeps all log files open indefinitely. There is no inherent
reason this needs to be the case, though, it would be possible to LRU out
old file descriptors and close them if they are not accessed for a few
hours and then reopen on the first access. We just haven't implemented
anything like that.

It would be good to understand this a little better. Does xfs pre-allocate
space for all open files? Perhaps just closing the file on log role and
opening it read-only would solve the issue? Is this at all related to the
use of sparse files for the indexes (i.e. RandomAccessFile.setLength(10MB)
when we create the index)? Does this effect other filesystems or just xfs?

-Jay
On Fri, Jul 26, 2013 at 12:42 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
 
+
Jason Rosenberg 2013-07-26, 21:00
+
Jay Kreps 2013-07-26, 21:03
+
Jason Rosenberg 2013-07-14, 08:37
+
Jay Kreps 2013-07-14, 16:45
+
Jason Rosenberg 2013-07-16, 18:23
+
Jay Kreps 2013-07-16, 20:32
+
Mike Heffner 2013-09-09, 15:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB