We are also seeing this problem with version 0.7.1 and logs on an XFS partition. At our largest scale we can frequently free over 600GB of disk usage by simply restarting Kafka. We've examined the `lsof` output from the Kafka process and while it does appear to have FDs open for all log files on disk (even those long past read from), it does not have any files open that were previously deleted from disk.
Du output agrees that the seen size is much larger than apparent-size size:
root@kafkanode-1:/raid0/kafka-logs/measures-0# du -h 00000000242666442619.kafka 1.1G 00000000242666442619.kafka root@kafkanode-1:/raid0/kafka-logs/measures-0# du -h --apparent-size 00000000242666442619.kafka 513M 00000000242666442619.kafka Our log size/retention policy is:
We tried dropping the caches from the Stack Overflow suggestion (sync; echo 3 > /proc/sys/vm/drop_caches) but that didn't seem to clear up the extra space. Haven't had the chance to try remounting with the allocsize option.
In summary, it would be great if Kafka would close FD's to log files that hadn't been read from for some period of time if it addresses this issue.
On Fri, Jul 26, 2013 at 5:03 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
This could certainly be done. It would be slightly involved since you would need to implement some kind of file-handle cache for both indexes and log files and re-open them on demand when a read occurs. If someone wants to take a shot at this, the first step would be to get a design wiki in place on how this would work. This is potentially nice to reduce the open file count (though open files are pretty cheap).
That said this issue only impacts xfs and it seems to be fixed by that setting jonathan found. I wonder if you could give that a try and see if it works for you too? I feel dealing with closed files does add a lot of complexity so if there is an easy fix I would probably rather avoid it.
-Jay On Mon, Sep 9, 2013 at 8:17 AM, Mike Heffner <[EMAIL PROTECTED]> wrote:
Mike On Mon, Sep 9, 2013 at 2:41 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Mike Heffner <[EMAIL PROTECTED]> Librato, Inc.
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext