Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> large amount of disk space freed on restart


Copy link to this message
-
Re: large amount of disk space freed on restart
Well, it sounds like files were deleted while Kafka still had them open. Or
something else opened them while Kafka deleted them. I haven't noticed this
on our systems but we haven't looked for it either.

Is anything outside of Kafka deleting  or reading those files?
On May 23, 2013 1:17 AM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:

> So, does this indicate kafka (or the jvm itself) is not aggressively
> closing file handles of deleted files?  Is there a fix for this?  Or is
> there not likely anything to be done?  What happens if the disk fills up
> with file handles for phantom deleted files?
>
> Jason
>
>
> On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <[EMAIL PROTECTED]> wrote:
>
>> It isn't uncommon if a process has an open file handle on a file that is
>> deleted, the space is not freed until the handle is closed. So restarting
>> the process that has a handle on the file would cause the space to be
>> freed
>> also.
>>
>> You can troubleshoot that with lsof.
>> Normally, I see 2-4 log segments deleted every hour in my brokers.  I see
>> log lines like this:
>>
>> 2013-05-23 04:40:06,857  INFO [kafka-logcleaner-0] log.LogManager -
>> Deleting log segment 00000000035434043157.kafka from <redacted topic>
>>
>> However, it seems like if I restart the broker, a massive amount of disk
>> space is freed (without a corresponding flood of these log segment deleted
>> messages).  Is there an explanation for this?  Does kafka keep reference
>> to
>> file segments around, and reuse them as needed or something?  And then or
>> restart, the references to those free segment files are dropped?
>>
>> Thoughts?
>>
>> This is with 0.7.2.
>>
>> Jason
>>
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB