Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> transfer -> CreateSocketForPipeline : hardcoded length of 2?

Copy link to this message
transfer -> CreateSocketForPipeline : hardcoded length of 2?
Hi guys: I noticed that in the call to createSocketForPipeline, there is a
hardcoded length of "2".

sock = createSocketForPipeline(src, 2, dfsClient);

This cascades down to the "getDataNodeReadTimeout" method, resulting in a
multiplier of 2.

//from DFSClient.java
int getDatanodeReadTimeout(int numNodes) {
    return dfsClientConf.socketTimeout > 0 ?
        (HdfsServerConstants.READ_TIMEOUT_EXTENSION * numNodes +
            dfsClientConf.socketTimeout) : 0;

I wonder why the pipeline length is "2" as opposed to "1" ?  It seems that
transferring a single block should have a pipeline length of 1?

///for example: in the createBlockOutputStream method, we have
   s = createSocketForPipeline(nodes[0], nodes.length, dfsClient);

Is the "2", then, just used to add some cushion to the timeout? Or is
something expected to be happening during a block transfer which makes the
pipeline a 2 node, rather than 1 node one?

Maybe I'm misunderstanding something about the way the pipeline works so
thanks for helping and apologies if this question is a little silly.

Jay Vyas
Andrew Wang 2013-04-29, 18:05