Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Solution for blocking fsync in 0.8

Copy link to this message
Re: Solution for blocking fsync in 0.8
Thanks Jay.  This is a very helpful investigation!

On 05/24/2012 01:40 PM, Jay Kreps wrote:
> Unfortunately *any* call to fsync will block appends even in a background
> thread so how can we give control over physical disk persistence without
> introducing high latency for the producer? The answer is that the linux
> pdflush daemon actually does a very similar thing to our flush parameters.
> pdflush is a daemon running on every linux machine that controls the
> writing of buffered/cached data back to disk. It allows you to control the
> percentage of memory filled with dirty pages by giving it either a
> percentage of memory, a time out for any dirty page to be written, or a
> fixed number of dirty bytes.
This would however by necessity by a global setting right?  (Assuming
there is no /proc trickery to change per-pid pdflush behaviour)