Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> hsync is too slower than hflush


Copy link to this message
-
Re: hsync is too slower than hflush
In fact, I just write 4k in every hsync. Datenode would write checksum file and data file when I hsync data to datanode. Each of them would spent nearly 25ms, so a hsync call would spent nearly 50ms. But hflush is very fast, which spent both 1ms in write checksum and data. If a hsync would spent 50ms, what meanings we use it? Or my test way is wrong?

--
Best Regards,
Haosong Huang
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, August 26, 2013 at 7:07 AM, Andrew Wang wrote:

> 50ms is believable. hsync makes each DN call fsync and wait for acks, so
> you'd expect at least a disk seek time (~10ms) with some extra time
> depending on how much unsync'd data is being written.
>
> So, just as some back of the envelope math, assuming a disk that can write
> at 100MB/s:
>
> 50ms - 10ms seek = 40ms writing time
> 100 MB/s * 40ms = 4MB
>
> If you're hsync'ing every 4MB, 50ms would be exactly what I'd expect.
>
> Best,
> Andrew
>
>
> On Sat, Aug 24, 2013 at 10:11 PM, haosdent <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > Hi, all. Hadoop support hsync which would call fsync of system after
> > 2.0.2. I have tested the performance of hsync() and hflush() again and
> > again, but I found that the hsync call() everytime would spent nearly 50ms
> > while the hflush call() just spent 2ms. In this slide(
> > http://www.slideshare.net/enissoz/hbase-and-hdfs-understanding-filesystem-usagePage 18), the author mentions that hsync() is 2x slower than hflush(). So,
> > is anything wrong? Thank you very much and looking forward to your help.
> >
> > --
> > Best Regards,
> > Haosong Huang
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB