Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - log file flush?


Copy link to this message
-
Re: log file flush?
Jay Kreps 2013-02-19, 18:00
To be clear: to lose data in the filesystem you need to hard kill the
machine. A hard kill of the process will not cause that.

-Jay

On Tue, Feb 19, 2013 at 8:25 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Jason,
>
> Although messages are always written to the log segment file, they
> initially are only in the file system's pagecache. As Swapnil mentioned
> earlier, messages are flushed to disk periodically. If you do a clean
> shutdown (kill -15), we close all log file, which should flush all dirty
> data to disk. If you do a hard kill or your machine just crashed, the
> unflushed data may be lost. The data that you saw in the .log file can be
> just in the pagecache.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 19, 2013 at 4:05 AM, Jason Huang <[EMAIL PROTECTED]> wrote:
>
>> Thanks for response.
>>
>> My confusion is that - once I see the message content in the .log
>> file, doesn't that mean the message has already been flushed to the
>> hard drive? Why would those messages still get lost if someone
>> manually kill the process (or if the server crashes unexpectedly)?
>>
>> Jason
>>
>> On Tue, Feb 19, 2013 at 6:53 AM, Swapnil Ghike <[EMAIL PROTECTED]>
>> wrote:
>> > Correction - The flush happens based on *number of messages* and time
>> > limits, whichever is hit first.
>> >
>> >
>> >
>> > On 2/19/13 3:50 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
>> >
>> >>The flush happens based on size and time limits,
>> >>whichever is hit first.
>> >
>>