Kafka, mail # user - Re: large amount of disk space freed on restart - 2013-07-16, 18:23
Solr & Elasticsearch trainings in New York & San Fransisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: large amount of disk space freed on restart
Ok,

An update on this.  It seems we are using XFS, which is available in newer
versions of Centos.  It definitely does pre-allocate space as a file grows,
see:
http://serverfault.com/questions/406069/why-are-my-xfs-filesystems-suddenly-consuming-more-space-and-full-of-sparse-file

Apparently it's not hard-allocated space, and would be released under
resource pressure....seems we may need to update how we monitor disk space
usage, etc....

But, it seems that the default log file size of 1.1Gb, causes it to jump to
preallocate an extra Gb.  So, in theory, if  I set a strategic log file
size to be just under the threshold that forces it to exponentially double
the size from 1Gb to 2Gb, I should be able to mostly solve this issue.
 E.g. use 950Mb instead of 1.1Gb max log file size.

If I change the max log file size for a broker, and restart it, will it
respect the new size going forward?

Jason
On Sun, Jul 14, 2013 at 9:44 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB