Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - large amount of disk space freed on restart


Copy link to this message
-
Re: large amount of disk space freed on restart
Jonathan Creasy 2013-05-23, 05:25
Well, it sounds like files were deleted while Kafka still had them open. Or
something else opened them while Kafka deleted them. I haven't noticed this
on our systems but we haven't looked for it either.

Is anything outside of Kafka deleting  or reading those files?
On May 23, 2013 1:17 AM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:

> So, does this indicate kafka (or the jvm itself) is not aggressively
> closing file handles of deleted files?  Is there a fix for this?  Or is
> there not likely anything to be done?  What happens if the disk fills up
> with file handles for phantom deleted files?
>
> Jason
>
>
> On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <[EMAIL PROTECTED]> wrote:
>
>> It isn't uncommon if a process has an open file handle on a file that is
>> deleted, the space is not freed until the handle is closed. So restarting
>> the process that has a handle on the file would cause the space to be
>> freed
>> also.
>>
>> You can troubleshoot that with lsof.
>> Normally, I see 2-4 log segments deleted every hour in my brokers.  I see
>> log lines like this:
>>
>> 2013-05-23 04:40:06,857  INFO [kafka-logcleaner-0] log.LogManager -
>> Deleting log segment 00000000035434043157.kafka from <redacted topic>
>>
>> However, it seems like if I restart the broker, a massive amount of disk
>> space is freed (without a corresponding flood of these log segment deleted
>> messages).  Is there an explanation for this?  Does kafka keep reference
>> to
>> file segments around, and reuse them as needed or something?  And then or
>> restart, the references to those free segment files are dropped?
>>
>> Thoughts?
>>
>> This is with 0.7.2.
>>
>> Jason
>>
>
>