Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> large amount of disk space freed on restart


Copy link to this message
-
Re: large amount of disk space freed on restart
Ok,

An update on this.  It seems we are using XFS, which is available in newer
versions of Centos.  It definitely does pre-allocate space as a file grows,
see:
http://serverfault.com/questions/406069/why-are-my-xfs-filesystems-suddenly-consuming-more-space-and-full-of-sparse-file

Apparently it's not hard-allocated space, and would be released under
resource pressure....seems we may need to update how we monitor disk space
usage, etc....

But, it seems that the default log file size of 1.1Gb, causes it to jump to
preallocate an extra Gb.  So, in theory, if  I set a strategic log file
size to be just under the threshold that forces it to exponentially double
the size from 1Gb to 2Gb, I should be able to mostly solve this issue.
 E.g. use 950Mb instead of 1.1Gb max log file size.

If I change the max log file size for a broker, and restart it, will it
respect the new size going forward?

Jason
On Sun, Jul 14, 2013 at 9:44 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hey Jason,
>
> As Jun says, we haven't seen that issue and no one else has reported that
> but it sounds like a bug of some kind.
>
> In 0.7 we don't do any preallocation of anything. The only time files
> shrink is during recovery--we re-checksum all messages that may not have
> been flushed and if any invalid messages are found we truncate them off.
> This would only occur after an unclean shutdown (kill -9) and you would get
> a bunch of errors in log warning you this was happening.
>
> -Jay
>
>
>
> On Sun, Jul 14, 2013 at 1:36 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > An update on this.  It appears that the phenomenon I'm seeing is that
> disk
> > space is freed on restart, but it's not due files getting deleted on
> > restart, but instead files are getting truncated on restart.  It appears
> > that log files get pre-allocated to a larger size than is used right
> away.
> >  Upon restart, they get truncated to the size of the file that actually
> > contains data.  Does this make sense?
> >
> > Before restart, I see a large number of log files size 2.1Gb.  Upon
> > restart, the disk space reclaimed drops to almost half that, on average.
> >
> > Thoughts?
> >
> > Jason
> >
> >
> > On Thu, May 23, 2013 at 8:55 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > I haven't seen this issue before. We do have ~1K topics in one of the
> > Kafka
> > > clusters at LinkedIn.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, May 23, 2013 at 11:05 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Yeah, that's what it looks like to me (looking at the code).  So, I'm
> > > > guessing it's some os level caching, resource recycling.  Have you
> ever
> > > > heard of this happening?  One thing that might be different in my
> usage
> > > > from the norm is a relatively large number of topics (e.g. ~2K
> topics).
> > > >
> > > > Jason
> > > >
> > > >
> > > > On Thu, May 23, 2013 at 7:14 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Jason,
> > > > >
> > > > > Kafka closes the handler of all delete files. Otherwise, the broker
> > > will
> > > > > run out of file handler quickly.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, May 22, 2013 at 10:17 PM, Jason Rosenberg <
> [EMAIL PROTECTED]>
> > > > > wrote:
> > > > >
> > > > > > So, does this indicate kafka (or the jvm itself) is not
> > aggressively
> > > > > > closing file handles of deleted files?  Is there a fix for this?
> >  Or
> > > is
> > > > > > there not likely anything to be done?  What happens if the disk
> > fills
> > > > up
> > > > > > with file handles for phantom deleted files?
> > > > > >
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > > On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <[EMAIL PROTECTED]>
> > wrote:
> > > > > >
> > > > > > > It isn't uncommon if a process has an open file handle on a
> file
> > > that
> > > > > is
> > > > > > > deleted, the space is not freed until the handle is closed. So
> > > > > restarting
> > >
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB