Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> transfer -> CreateSocketForPipeline : hardcoded length of 2?


Copy link to this message
-
transfer -> CreateSocketForPipeline : hardcoded length of 2?
Hi guys: I noticed that in the call to createSocketForPipeline, there is a
hardcoded length of "2".

//from
sock = createSocketForPipeline(src, 2, dfsClient);

This cascades down to the "getDataNodeReadTimeout" method, resulting in a
multiplier of 2.

//from DFSClient.java
int getDatanodeReadTimeout(int numNodes) {
    return dfsClientConf.socketTimeout > 0 ?
        (HdfsServerConstants.READ_TIMEOUT_EXTENSION * numNodes +
            dfsClientConf.socketTimeout) : 0;
  }

I wonder why the pipeline length is "2" as opposed to "1" ?  It seems that
transferring a single block should have a pipeline length of 1?

///for example: in the createBlockOutputStream method, we have
   s = createSocketForPipeline(nodes[0], nodes.length, dfsClient);

Is the "2", then, just used to add some cushion to the timeout? Or is
something expected to be happening during a block transfer which makes the
pipeline a 2 node, rather than 1 node one?

Maybe I'm misunderstanding something about the way the pipeline works so
thanks for helping and apologies if this question is a little silly.

--
Jay Vyas
http://jayunit100.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB