Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> kafka file persistance blocks, ramifications on payload sizes and flush timings


Copy link to this message
-
kafka file persistance blocks, ramifications on payload sizes and flush timings
On the dev list I was curious as to how kafka persists the in-memory to
disk, and Jay responded with:

"filechannel.force() always fully syncs the file to disk. This is done
irrespective of message boundaries. The file is locked during this
time so other appends are blocked."

So doing a little match, if my payload sizes are 20KB, and I flush once
there is 10K items, that means:

10 000 x 20 480 bytes = 195.3125 megabytes

What I am curious is, how long does this flush to disk take, and are there
any built it metrics/logging that I can measure the average time it takes
to write the in-memory to disk?  Or what about the time a producer is
blocked during a flush to disk?

*
*
+
Edward Smith 2012-05-09, 15:32
+
Jun Rao 2012-05-09, 17:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB