Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> hsync is too slower than hflush


+
haosdent 2013-08-25, 05:11
Copy link to this message
-
Re: hsync is too slower than hflush
50ms is believable. hsync makes each DN call fsync and wait for acks, so
you'd expect at least a disk seek time (~10ms) with some extra time
depending on how much unsync'd data is being written.

So, just as some back of the envelope math, assuming a disk that can write
at 100MB/s:

50ms - 10ms seek = 40ms writing time
100 MB/s * 40ms = 4MB

If you're hsync'ing every 4MB, 50ms would be exactly what I'd expect.

Best,
Andrew
On Sat, Aug 24, 2013 at 10:11 PM, haosdent <[EMAIL PROTECTED]> wrote:

> Hi, all. Hadoop support hsync which would call fsync of system after
> 2.0.2. I have tested the performance of hsync() and hflush() again and
> again, but I found that the hsync call() everytime would spent nearly 50ms
> while the hflush call() just spent 2ms. In this slide(
> http://www.slideshare.net/enissoz/hbase-and-hdfs-understanding-filesystem-usagePage 18), the author mentions that hsync() is 2x slower than hflush(). So,
> is anything wrong? Thank you very much and looking forward to your help.
>
> --
> Best Regards,
> Haosong Huang
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
+
haosdent 2013-08-26, 02:44
+
Andrew Wang 2013-08-26, 03:18
+
haosdent 2013-08-26, 03:21
+
lei liu 2013-08-26, 14:30
+
Andrew Wang 2013-08-26, 17:44
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB