Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - DFSOutputStream.sync() method latency time

lei liu 2013-03-28, 08:15
Copy link to this message
Re: DFSOutputStream.sync() method latency time
Yanbo Liang 2013-03-28, 15:40
1st when client wants to write data to HDFS, it should be create
Then the client write data to this output stream and this stream will
transfer data to all DataNodes with the constructed pipeline by the means
of Packet whose size is 64KB.
These two operations is concurrent, so the write latency is not simple

2nd the sync method only flush the last packet ( at most 64KB ) data to the

Because of the cocurrent processing of all these operations, so the latency
is smaller than the superposition of each operation.
It's parallel computing rather than serial computing in a sense.
2013/3/28 lei liu <[EMAIL PROTECTED]>

> When client  write data, if there are three replicates,  the sync method
> latency time formula should be:
> sync method  latency time = first datanode receive data time + sencond
> datanode receive data  time +  third datanode receive data time.
> if the three datanode receive data time all are 2 millisecond, so the sync
> method  latency time should is 6 millisecond,  but according to our our
> monitor, the the sync method  latency time is 2 millisecond.
> How to calculate sync method  latency time?
> Thanks,
> LiuLei
lei liu 2013-03-29, 02:50
Yanbo Liang 2013-03-29, 03:29