Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> log file flush?


+
Jason Huang 2013-02-19, 11:28
+
Swapnil Ghike 2013-02-19, 11:51
+
Swapnil Ghike 2013-02-19, 11:54
+
Jason Huang 2013-02-19, 12:06
+
Jun Rao 2013-02-19, 16:26
Copy link to this message
-
Re: log file flush?
To be clear: to lose data in the filesystem you need to hard kill the
machine. A hard kill of the process will not cause that.

-Jay

On Tue, Feb 19, 2013 at 8:25 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Jason,
>
> Although messages are always written to the log segment file, they
> initially are only in the file system's pagecache. As Swapnil mentioned
> earlier, messages are flushed to disk periodically. If you do a clean
> shutdown (kill -15), we close all log file, which should flush all dirty
> data to disk. If you do a hard kill or your machine just crashed, the
> unflushed data may be lost. The data that you saw in the .log file can be
> just in the pagecache.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 19, 2013 at 4:05 AM, Jason Huang <[EMAIL PROTECTED]> wrote:
>
>> Thanks for response.
>>
>> My confusion is that - once I see the message content in the .log
>> file, doesn't that mean the message has already been flushed to the
>> hard drive? Why would those messages still get lost if someone
>> manually kill the process (or if the server crashes unexpectedly)?
>>
>> Jason
>>
>> On Tue, Feb 19, 2013 at 6:53 AM, Swapnil Ghike <[EMAIL PROTECTED]>
>> wrote:
>> > Correction - The flush happens based on *number of messages* and time
>> > limits, whichever is hit first.
>> >
>> >
>> >
>> > On 2/19/13 3:50 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
>> >
>> >>The flush happens based on size and time limits,
>> >>whichever is hit first.
>> >
>>

 
+
Jason Huang 2013-02-20, 04:03
+
Jay Kreps 2013-02-20, 04:28
+
Jason Huang 2013-02-20, 07:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB