Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> kafka file persistance blocks, ramifications on payload sizes and flush timings


Copy link to this message
-
kafka file persistance blocks, ramifications on payload sizes and flush timings
On the dev list I was curious as to how kafka persists the in-memory to
disk, and Jay responded with:

"filechannel.force() always fully syncs the file to disk. This is done
irrespective of message boundaries. The file is locked during this
time so other appends are blocked."

So doing a little match, if my payload sizes are 20KB, and I flush once
there is 10K items, that means:

10 000 x 20 480 bytes = 195.3125 megabytes

What I am curious is, how long does this flush to disk take, and are there
any built it metrics/logging that I can measure the average time it takes
to write the in-memory to disk?  Or what about the time a producer is
blocked during a flush to disk?

*
*
+
Edward Smith 2012-05-09, 15:32
+
Jun Rao 2012-05-09, 17:11