Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - large amount of disk space freed on restart


Copy link to this message
-
Re: large amount of disk space freed on restart
Jay Kreps 2013-07-26, 16:46
Interesting.

Yes, Kafka keeps all log files open indefinitely. There is no inherent
reason this needs to be the case, though, it would be possible to LRU out
old file descriptors and close them if they are not accessed for a few
hours and then reopen on the first access. We just haven't implemented
anything like that.

It would be good to understand this a little better. Does xfs pre-allocate
space for all open files? Perhaps just closing the file on log role and
opening it read-only would solve the issue? Is this at all related to the
use of sparse files for the indexes (i.e. RandomAccessFile.setLength(10MB)
when we create the index)? Does this effect other filesystems or just xfs?

-Jay
On Fri, Jul 26, 2013 at 12:42 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> It looks like xfs will reclaim the preallocated space for a file, after it
> is closed.
>
> Does kafka close a file after it has reached it's max size and started
> writing to the next log file in sequence?  Or does it keep all open until
> they are deleted, or the server quits (that's what it seems like).
>
> I could imagine that it might need to keep log files open, in order to
> allow consumers access to them.  But does it keep them open indefinitely,
> after there is no longer any data to be written to them, and no consumers
> are currently attempting to read from them?
>
> Jason
>
>
> On Tue, Jul 16, 2013 at 4:32 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Interesting. Yes it will respect whatever setting it is given for new
> > segments created from that point on.
> >
> > -Jay
> >
> >
> > On Tue, Jul 16, 2013 at 11:23 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Ok,
> > >
> > > An update on this.  It seems we are using XFS, which is available in
> > newer
> > > versions of Centos.  It definitely does pre-allocate space as a file
> > grows,
> > > see:
> > >
> > >
> >
> http://serverfault.com/questions/406069/why-are-my-xfs-filesystems-suddenly-consuming-more-space-and-full-of-sparse-file
> > >
> > > Apparently it's not hard-allocated space, and would be released under
> > > resource pressure....seems we may need to update how we monitor disk
> > space
> > > usage, etc....
> > >
> > > But, it seems that the default log file size of 1.1Gb, causes it to
> jump
> > to
> > > preallocate an extra Gb.  So, in theory, if  I set a strategic log file
> > > size to be just under the threshold that forces it to exponentially
> > double
> > > the size from 1Gb to 2Gb, I should be able to mostly solve this issue.
> > >  E.g. use 950Mb instead of 1.1Gb max log file size.
> > >
> > > If I change the max log file size for a broker, and restart it, will it
> > > respect the new size going forward?
> > >
> > > Jason
> > >
> > >
> > > On Sun, Jul 14, 2013 at 9:44 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > As Jun says, we haven't seen that issue and no one else has reported
> > that
> > > > but it sounds like a bug of some kind.
> > > >
> > > > In 0.7 we don't do any preallocation of anything. The only time files
> > > > shrink is during recovery--we re-checksum all messages that may not
> > have
> > > > been flushed and if any invalid messages are found we truncate them
> > off.
> > > > This would only occur after an unclean shutdown (kill -9) and you
> would
> > > get
> > > > a bunch of errors in log warning you this was happening.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > > On Sun, Jul 14, 2013 at 1:36 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > An update on this.  It appears that the phenomenon I'm seeing is
> that
> > > > disk
> > > > > space is freed on restart, but it's not due files getting deleted
> on
> > > > > restart, but instead files are getting truncated on restart.  It
> > > appears
> > > > > that log files get pre-allocated to a larger size than is used
> right
> > > > away.
> > > > >  Upon restart, they get truncated to the size of the file that
> > actually
> > > > > contains data.  Does this make sense?