Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - large amount of disk space freed on restart


+
Jason Rosenberg 2013-05-23, 04:46
+
Jonathan Creasy 2013-05-23, 04:51
+
Jason Rosenberg 2013-05-23, 05:17
+
Jonathan Creasy 2013-05-23, 05:25
Copy link to this message
-
Re: large amount of disk space freed on restart
Jason Rosenberg 2013-05-23, 05:49
No, nothing outside of kafka would look at those files....

I'm wondering if it's an os level thing too....
On Wed, May 22, 2013 at 10:25 PM, Jonathan Creasy <[EMAIL PROTECTED]> wrote:

> Well, it sounds like files were deleted while Kafka still had them open.
> Or something else opened them while Kafka deleted them. I haven't noticed
> this on our systems but we haven't looked for it either.
>
> Is anything outside of Kafka deleting  or reading those files?
> On May 23, 2013 1:17 AM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
>> So, does this indicate kafka (or the jvm itself) is not aggressively
>> closing file handles of deleted files?  Is there a fix for this?  Or is
>> there not likely anything to be done?  What happens if the disk fills up
>> with file handles for phantom deleted files?
>>
>> Jason
>>
>>
>> On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <[EMAIL PROTECTED]> wrote:
>>
>>> It isn't uncommon if a process has an open file handle on a file that is
>>> deleted, the space is not freed until the handle is closed. So restarting
>>> the process that has a handle on the file would cause the space to be
>>> freed
>>> also.
>>>
>>> You can troubleshoot that with lsof.
>>> Normally, I see 2-4 log segments deleted every hour in my brokers.  I see
>>> log lines like this:
>>>
>>> 2013-05-23 04:40:06,857  INFO [kafka-logcleaner-0] log.LogManager -
>>> Deleting log segment 00000000035434043157.kafka from <redacted topic>
>>>
>>> However, it seems like if I restart the broker, a massive amount of disk
>>> space is freed (without a corresponding flood of these log segment
>>> deleted
>>> messages).  Is there an explanation for this?  Does kafka keep reference
>>> to
>>> file segments around, and reuse them as needed or something?  And then or
>>> restart, the references to those free segment files are dropped?
>>>
>>> Thoughts?
>>>
>>> This is with 0.7.2.
>>>
>>> Jason
>>>
>>
>>

 
+
Jun Rao 2013-05-23, 14:15
+
Jason Rosenberg 2013-05-23, 18:06
+
Jun Rao 2013-05-24, 03:56
+
Jason Rosenberg 2013-07-14, 08:37
+
Jay Kreps 2013-07-14, 16:45
+
Jason Rosenberg 2013-07-16, 18:23
+
Jay Kreps 2013-07-16, 20:32
+
Jason Rosenberg 2013-07-26, 07:43
+
Jay Kreps 2013-07-26, 16:46
+
Jason Rosenberg 2013-07-26, 21:00
+
Jay Kreps 2013-07-26, 21:03
+
Mike Heffner 2013-09-09, 15:18
+
Jay Kreps 2013-09-09, 17:48
+
Jason Rosenberg 2013-09-09, 18:41
+
Mike Heffner 2013-09-09, 21:07